Third-party agent observability with Microsoft Agent 365 (Frontier)

Important

This feature is part of Frontier preview program. Frontier connects you directly with Microsoft’s latest AI innovations. Frontier previews are subject to the existing preview terms of your customer agreements. As these features are still in development, their availability and capabilities may change over time.

Organizations increasingly deploy AI agents across multiple platforms, including Microsoft Copilot Studio, Microsoft Foundry, and third-party platforms such as Amazon Bedrock, Google Vertex AI, Anthropic Claude, and Salesforce Agentforce. As these environments grow, operational visibility becomes fragmented. Administrators, security teams, and compliance teams need a consistent way to understand how agents behave across the organization.

Third-party agent observability in Microsoft Agent 365 extends the Agent 365 observability experience to supported connected platforms. After you connect a platform and synchronize agents into the Agent 365 registry, Agent 365 can collect and normalize supported observability telemetry into a common operating model.

By using this capability, you can:

  • View third-party agents alongside Microsoft agents.
  • Monitor activity and operational health.
  • Investigate individual agent executions.
  • Analyze usage trends and runtime behavior.
  • Extend observability signals into security and governance experiences.

How third-party observability works

Third-party observability builds on the Connected platforms framework in Microsoft Agent 365. The workflow consists of four stages.

1 - Connect the external platform

Create a connection to a supported platform, such as Amazon Bedrock, Google Vertex AI, Anthropic Claude, Salesforce Agentforce, or Databricks Genie. For the platforms that currently support observability, see Supported observability platforms.

Agent 365 authenticates to the external environment by using platform-specific credentials and synchronizes selected agent metadata into the Agent 365 registry.

Some platforms require more permissions or configuration for observability than they require to create the connection. Before you enable collection, review Amazon Bedrock permissions, Google Vertex AI prerequisites, or Anthropic Claude prerequisites for the platform you're connecting.

2 - Discover and register agents

During synchronization, Agent 365 imports selected third-party agent information into your organization's centralized agent inventory.

3 - Collect observability telemetry

Agent 365 collects agent activity from supported connected platforms and transforms it into a common observability model based on OpenTelemetry concepts. Agent 365 normalizes runs, sessions, tool invocations, exceptions, and execution traces into a consistent schema regardless of the source platform.

4 - Surface insights across Microsoft experiences

After Agent 365 processes the telemetry, operational insights can appear in Agent 365 experiences, including Agent Activity, Agent Overview, the Microsoft 365 admin center, and security and governance integrations.

You can analyze third-party agent observability data through the same framework as Microsoft-hosted agents, without learning a different telemetry model.

Supported observability platforms

Current observability support includes the following platforms:

Platform Supported environment
Amazon Bedrock Classic
Amazon Bedrock AgentCore
Google Vertex AI Google Vertex AI
Anthropic Claude Managed Agents

Additional platform integrations are expected over time.

Amazon Bedrock

Microsoft Agent 365 supports observability for agents running in Amazon Bedrock Classic and Amazon Bedrock AgentCore environments. Agent 365 can collect agent activity and transform it into observability records.

With the available telemetry, you can:

  • Review agent invocations.
  • Inspect execution traces.
  • Analyze tool usage.
  • Examine processing steps.
  • Investigate runtime exceptions.
  • Review resource utilization.
  • Analyze response generation activity.

Use this telemetry to investigate how Amazon Bedrock agents behave in the operational experience available for Microsoft-hosted agents.

Google Vertex AI

Microsoft Agent 365 supports observability for agents synchronized from Google Vertex AI.

Vertex AI observability data can be transformed into the Agent 365 observability model so that you can:

  • View agent executions.
  • Track sessions.
  • Analyze tool executions.
  • Monitor runtime behavior.
  • Review operational reports.

You can monitor Google Vertex AI agents alongside Microsoft agents and agents from other connected platforms in a unified Agent 365 experience.

Anthropic Claude

Microsoft Agent 365 supports observability for agents synchronized from Anthropic Claude Managed Agents. Agent 365 reads sessions and session events from the connected Anthropic workspace and transforms them into the common observability model.

With the available telemetry, you can:

  • Review agent invocations and sessions.
  • Inspect execution traces.
  • Analyze tool usage.
  • Investigate runtime exceptions.
  • View sessions for the agent in Agent Map.

After you interact with an agent in an Anthropic Claude session, allow 5 to 10 minutes for its activity to appear in Agent 365.

Amazon Bedrock permissions

You connect to Amazon Bedrock by supplying an AWS access key. The IAM user or role behind that key needs the permissions described in this section. Grant only the permissions for the environments you use.

Agent 365 reads your agents and their activity. Collecting observability data never invokes your agents, so it doesn't incur inference charges.

Amazon Bedrock Classic

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "bedrock:ListAgents",
        "bedrock:GetAgent",
        "ec2:DescribeRegions"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:CreateBucket"
      ],
      "Resource": "arn:aws:s3:::a365-observability-store-<account-id>-<region>-an"
    },
    {
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::a365-observability-store-<account-id>-<region>-an/*"
    }
  ]
}
Permission Purpose
bedrock:ListAgents Finds agents in each connected region.
bedrock:GetAgent Reads each agent's name, description, instructions, model, and status. Without it, agents are discovered with limited detail.
ec2:DescribeRegions Validates the regions you select. Returns region metadata only.
s3:ListBucket Confirms the activity bucket exists and finds each agent's activity.
s3:GetObject Reads each activity record.
s3:CreateBucket Optional. Lets Agent 365 create the bucket on first use. Omit it if you create the bucket yourself.

Activity storage

Amazon Bedrock Classic agents record activity to Amazon S3. Three things must be in place:

  1. Enable trace capture on each agent in the Amazon Bedrock console, and select an S3 bucket. Agent 365 can't enable this for you.
  2. Name the bucket a365-observability-store-<account-id>-<region>-an, unless you specify a bucket on the connection. The -an suffix places the bucket in your account regional namespace, so the name is reserved to your account and can't be taken by another AWS customer.
  3. Store activity under observability-store/<agent-id>/, where <agent-id> is the Bedrock agent ID, such as SPGCARJMQW.

Important

An S3 bucket exists in exactly one region. If your connection covers several regions, create one bucket per region, using the full region code such as eu-west-1.

If the bucket uses a customer managed key, also grant kms:Decrypt on that key.

Amazon Bedrock AgentCore

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "bedrock-agentcore:ListHarnesses",
        "bedrock-agentcore:GetHarness",
        "bedrock-agentcore:ListAgentRuntimes",
        "bedrock-agentcore:GetAgentRuntime",
        "ec2:DescribeRegions",
        "logs:DescribeLogGroups"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:StartQuery",
        "logs:GetQueryResults"
      ],
      "Resource": [
        "arn:aws:logs:*:<account-id>:log-group:/aws/bedrock-agentcore/runtimes/*",
        "arn:aws:logs:*:<account-id>:log-group:aws/spans"
      ]
    }
  ]
}
Permission Purpose
bedrock-agentcore:ListHarnesses, GetHarness Finds harness-based agents and reads their model, memory, and tool configuration.
bedrock-agentcore:ListAgentRuntimes, GetAgentRuntime Finds runtimes not managed by a harness. Runtimes that belong to a harness are skipped, so an agent is never listed twice.
ec2:DescribeRegions Validates the regions you select.
logs:StartQuery, logs:GetQueryResults Retrieves agent activity from CloudWatch Logs.
logs:DescribeLogGroups Recommended. Finds all of an agent's log groups, including custom endpoints. Without it, only the default endpoint is read. This action can't be scoped to individual log groups.

Activity storage

AgentCore records activity to Amazon CloudWatch Logs, in one of two locations depending on when the agent was created and the region it runs in:

Log group Used by
/aws/bedrock-agentcore/runtimes/<runtime-id>-<endpoint> Agents using unified observability
aws/spans Other agents, and gateway and built-in tool activity

Agent 365 reads both and combines the results.

Important

AgentCore activity becomes queryable only after Transaction Search is enabled in the Amazon CloudWatch console. This is a one-time setting per AWS account. If agents are discovered but no activity appears, check this first.

Optional permissions

Capability Permissions
Show tools, collaborators, and knowledge sources for Classic agents bedrock:ListAgentActionGroups, GetAgentActionGroup, ListAgentCollaborators, ListAgentKnowledgeBases, ListDataSources, GetDataSource
Show tools, MCP servers, and connected agents for AgentCore agents bedrock-agentcore:GetGateway, ListGatewayTargets, GetGatewayTarget
Delete the agent in AWS when you remove it from Agent 365 bedrock:DeleteAgent, bedrock-agentcore:DeleteHarness, DeleteAgentRuntime
Connect every account in an AWS organization organizations:ListAccounts, sts:AssumeRole

Without the detail permissions, agents are still discovered and their activity is still collected. Only the extra detail is missing.

Warning

Deleting an agent in AWS is permanent. Grant the delete permissions only if you intend Agent 365 to remove agents from your AWS account. They're never used during synchronization, only when an administrator explicitly removes an agent.

To connect an AWS organization, grant the permissions in the management account, then create a role with the same name in each member account. The role must trust the management account and carry the permissions for the agent types used in that account. Suspended accounts are skipped.

Restrict the policy further

AWS doesn't support resource-level permissions on list operations, so bedrock:ListAgents, bedrock-agentcore:ListHarnesses, bedrock-agentcore:ListAgentRuntimes, ec2:DescribeRegions, organizations:ListAccounts, and logs:DescribeLogGroups must use "Resource": "*".

You can scope the remaining operations to your account and regions:

Resource ARN format
Bedrock agent arn:aws:bedrock:<region>:<account-id>:agent/<agent-id>
AgentCore harness arn:aws:bedrock-agentcore:<region>:<account-id>:harness/<harness-id>
AgentCore runtime arn:aws:bedrock-agentcore:<region>:<account-id>:runtime/<runtime-id>

Caution

Avoid scoping to individual agent IDs. Agent 365 discovers agents it hasn't seen before, so naming specific agents means any agent created afterward isn't discovered until you update the policy.

Permissions Agent 365 doesn't request

Not requested What this means
bedrock:InvokeAgent, bedrock:InvokeModel Observability never runs your agents, so collecting telemetry doesn't incur inference charges.
Write access to agent configuration Nothing in Amazon Bedrock is created or modified, apart from the optional activity bucket.
s3:PutObject, s3:DeleteObject Activity records are read only. They're never written, changed, or deleted.
Broad Amazon S3 access The Amazon S3 permissions apply to the single activity bucket, not to other buckets in your account.

Agent 365 also calls sts:GetCallerIdentity to determine which AWS account the credentials belong to. This operation requires no permission.

Google Vertex AI prerequisites

Agent 365 reads Google Vertex AI observability data from BigQuery and Cloud Monitoring, not from the Vertex AI API. The permissions that allow Agent 365 to discover and manage your agents don't grant access to this telemetry, so you must complete the steps in this section separately.

Important

A connection that verifies successfully and imports agents doesn't confirm that observability is configured. Credential verification and agent synchronization use the Vertex AI API only. Agents can import correctly while telemetry remains unavailable because roles, APIs, or trace export are missing. Complete every step in this section, then confirm that activity appears on the agent's Activity tab.

Required roles

Grant these roles to the Google Cloud service account whose JSON key you supply to the connection. They're required in addition to the Vertex AI permissions the connection already needs for agent discovery and management, which are described in Connected platforms.

Required role Role ID Purpose
BigQuery Data Viewer roles/bigquery.dataViewer Read exported trace tables.
BigQuery Job User roles/bigquery.jobUser Execute queries to retrieve spans; requires bigquery.jobs.create.
Monitoring Viewer roles/monitoring.viewer Read request-count and latency metrics.
Cloud Trace User roles/cloudtrace.user Read Cloud Trace spans.

Note

Both BigQuery roles are required. Data Viewer alone permits reading the tables but not running the queries that retrieve spans, and Job User alone permits running queries but not reading the results.

Enable the required APIs

Enable all four APIs on the project where the agent is deployed:

  • Vertex AI API
  • BigQuery API
  • Cloud Trace API
  • Cloud Monitoring API

Enable telemetry and trace export

  1. Enable telemetry collection on the deployed Vertex AI agent. Agent 365 can't enable this feature for you, and no telemetry is produced until you do.
  2. Enable Trace Analytics BigQuery export so that traces are written to BigQuery.
  3. Confirm that the trace_analytics_link dataset and its _AllSpans view exist in the project. Agent 365 queries this view directly. If either is missing, trace export isn't configured.

Configure the connection

Configure the connection for the project and region where the agent is deployed. A connection scoped to a different project or region imports agents but returns no telemetry.

Anthropic Claude prerequisites

Anthropic Claude observability uses the same workspace ID and workspace-scoped API key that you use to create the connected-platform connection. For connection details, see Anthropic Claude Managed Agents.

Before you connect the platform:

  1. Create an Anthropic Claude workspace. Don't use the default workspace, because it doesn't expose the workspace ID required for the connection.
  2. Create an environment and a Managed Agent in that workspace.
  3. Create a session that uses the agent and environment, and then interact with the agent to generate session activity.
  4. Create a workspace-scoped API key. You need at least the Developer role in the Anthropic workspace to create the key.

Use the workspace ID and API key when you create the connection in the Microsoft 365 admin center. After you verify and save the connection, wait for the agent synchronization to finish before you check the agent's Activity tab.

Important

Anthropic Claude Managed Agents APIs are in beta, and the Anthropic Claude connected-platform integration is a preview feature.

Configure observability data collection

Agent observability data collection is enabled by default for all new connections. If you don't want to collect agent observability data, edit the connection and clear Collect agent observability data.

For connections created before this feature was introduced, agent observability data collection is disabled by default. To enable it, edit the connection and select Collect agent observability data.

Note

Enabling collection doesn't itself make telemetry available. The source platform must also be configured to produce and expose it. See Amazon Bedrock permissions, Google Vertex AI prerequisites, and Anthropic Claude prerequisites.

View third-party agent activity

After observability data is available, open a third-party agent and select the Activity tab to review operational metrics and trends.

Activity metrics

The Activity tab provides key observability measurements for each imported agent. You can view usage within an hour from when users interact with agents.

Metric Description
Sessions The number of agent sessions for the reporting period. A session refers to either conversational or autonomous activity. A conversational session is a period of continuous interaction between a user and an agent. A new session begins after 30 minutes of inactivity. An autonomous session is initiated by a trigger such as a schedule or another agent.
Exceptions The number of sessions in the selected date range that had an error.
Runtime Total time, in hours, that agents spend running during the reporting period, measured from the start of each run to its completion.
Users This metric isn't available.

These metrics help you understand adoption, usage patterns, reliability, and operational health.

Understand third-party metrics

Because third-party platforms use different identity systems, some metrics might differ from those for Microsoft-hosted agents. For example:

  • Active user information might not be available when the originating platform doesn't provide Microsoft Entra user identities.
  • Session, invocation, runtime, exception, and tool-call metrics remain available.
  • Missing identity-based metrics are displayed as unavailable instead of generating inaccurate results.

Expect some platform-specific variations based on the telemetry available from the source environment.

Troubleshoot third-party connections

Amazon Bedrock

Symptom Likely cause
The connection fails validation Missing ec2:DescribeRegions, or the access key is invalid or disabled.
Agents appear without a description, instructions, or model Missing bedrock:GetAgent.
Agents appear without tools or knowledge sources Missing optional detail permissions.
Classic agents appear, but no activity Trace capture isn't enabled on the agent, the bucket doesn't exist or has a different name, or missing s3:ListBucket or s3:GetObject.
AgentCore agents appear, but no activity CloudWatch Transaction Search isn't enabled, or missing logs:StartQuery or logs:GetQueryResults.
AgentCore activity is incomplete Missing logs:DescribeLogGroups and the agent uses a custom endpoint.
Standalone runtimes are missing Missing bedrock-agentcore:ListAgentRuntimes or GetAgentRuntime.
Some organization accounts are missing The role name doesn't exist in those accounts, its trust policy doesn't allow the management account, or the account is suspended.

Google Vertex AI

Symptom Likely cause
The connection verifies and agents import, but no activity appears Expected if you don't configure observability. Connection setup uses only the Vertex AI API. Work through Google Vertex AI prerequisites.
No spans, but metrics appear Missing roles/bigquery.dataViewer or roles/bigquery.jobUser, or trace export isn't enabled.
Queries fail although the service account can read the tables Missing roles/bigquery.jobUser. Reading tables and running queries are separate permissions.
No request-count or latency metrics Missing roles/monitoring.viewer, or the Cloud Monitoring API isn't enabled.
The trace_analytics_link dataset or _AllSpans view doesn't exist Trace Analytics BigQuery export isn't enabled on the project.
Some agents report activity and others don't Telemetry collection isn't enabled on the agents that report nothing. Set it per agent.
Agents import but telemetry is consistently empty The connection is scoped to a different project or region than the one where the agent is deployed.

Anthropic Claude

Symptom Likely cause
The connection fails validation The workspace ID or workspace-scoped API key is incorrect, the key belongs to a different workspace, or the key has expired.
The connection saves, but an agent doesn't appear The agent is in a different workspace, or agent synchronization hasn't finished.
The agent appears, but no activity is available No session activity has been generated, observability collection is disabled for the connection, or the 5 to 10 minute processing interval hasn't elapsed. Create a session, interact with the agent, and then check the Activity tab again.
A session doesn't appear in Agent Map Activity for the session hasn't been processed yet. Confirm that it appears on the agent's Activity tab, and then check Agent Map again.
  • Last updated on