Extend monitoring with Application Insights

  • 9 minutes

Some monitoring questions, such as which tools fail most often or respond most slowly, call for data that you can query and alert on yourself. In this unit, you learn how to request environment-level telemetry in Application Insights, validate that it arrives, and query it for your own measures.

Identify what environment-level telemetry adds

Application Insights is a feature of Azure Monitor, a service for monitoring live applications. It requires a Microsoft Azure subscription. Copilot Studio can send agent telemetry to Application Insights at two scopes:

  • Agent-level telemetry: Configured separately for each agent and available only with the standard harness. This option doesn't apply to agents powered by the GitHub Copilot harness.
  • Environment-level telemetry (preview): Configured once for a Copilot Studio environment and available with both the standard harness and the GitHub Copilot harness. It covers the agents in that environment.

The Monitor tab gives you a summary-level view of one agent. Environment-level telemetry writes agent activity to an Application Insights resource as spans. A span is a record of one operation, such as an agent invocation or a tool call. The spans use an OpenTelemetry-aligned format, which follows an emerging industry standard for observing agentic systems.

With the spans in Application Insights, you can:

  • Investigate agent runs, tool execution, and outputs across the agents in the environment.
  • Query the data with Kusto Query Language (KQL) for measures you define.
  • Create alerts and build dashboards on the results.

Request telemetry for your environment

Environment-level telemetry is typically owned by administrators, platform teams, or Center of Excellence teams. Setup requires the Power Platform administrator or Dynamics 365 administrator role at the tenant level. It also requires the environment administrator or system administrator role in the Dataverse environment. An administrator with these roles creates an export package of type Copilot Studio in the Power Platform admin center. Your part is to confirm that your agent qualifies, make the request, and then validate and read the data.

Before you make the request, check that your agent meets these requirements:

  • Managed environment: The feature is available only for managed environments.
  • Public cloud: The feature is available only in Microsoft public cloud environments.
  • Agent type: Only agents built in Copilot Studio are logged. Declarative agents are excluded.
  • Authentication and tenancy: Unauthenticated and multi-tenant agent configurations aren't logged.

Include these points for the administrator in your request:

  • Dedicated resource: Use an Application Insights resource that receives data from this environment only. Built-in Application Insights reports don't work correctly when one resource holds data from several environments. Also avoid sending agent-level and environment-level telemetry to the same resource.
  • Local authentication: The Local authentication property must be turned on in the target Application Insights resource.
  • Read access: You need access to the Application Insights resource to validate and query the data.

Learn more in Export data to Application Insights.

Validate the spans from a test conversation

Environment-level telemetry writes every span to the dependencies table in Application Insights. Each exported event is one row, and three event types appear:

  • InvokeAgent: The agent invocation for one turn. It carries the user prompt in gen_ai.input.messages.
  • ExecuteTool: One tool call. It carries the tool name in gen_ai.tool.name and the tool type in gen_ai.tool.type. It also carries the call arguments in gen_ai.tool.call.arguments and the result in gen_ai.tool.call.result.
  • OutputMessages: The agent's reply, in gen_ai.output.messages.

The gen_ai.* keys are stored in the customDimensions column. The table's native columns record each span's outcome, including success, duration, and resultCode, with values such as OK or ERROR.

Spans connect into traces. Each turn of a conversation is its own trace, and all spans in that turn share an operation_Id. The InvokeAgent span is the root of the trace. Each ExecuteTool and OutputMessages span points to the root through operation_ParentId. A conversation spans several turns, so group or filter by gen_ai.conversation.id to put its turns back together.

Environment-level telemetry also records conversations that you run while testing in Copilot Studio. To validate a new configuration:

  1. Run a test conversation with the agent, using a request that invokes at least one tool.
  2. In the Application Insights resource in Azure, open Logs.
  3. Confirm that agent spans appear in the dependencies table.
  4. Confirm that each turn's InvokeAgent, ExecuteTool, and OutputMessages spans share an operation_Id.

Telemetry delivery can take up to 24 hours on a new configuration. If no data appears after 24 hours, ask the administrator to confirm that the person who set up the export had the required roles. Insufficient permissions during setup are a common cause of missing telemetry.

Expect these behaviors when you read the spans:

  • Standalone replies: An OutputMessages span can arrive without an InvokeAgent parent and appear as a single-span trace.
  • Connected agents: In certain connected agent scenarios, the parent-child relationship between traces isn't mapped correctly.
  • Truncation: Large content, including agent responses and tool arguments, might be truncated.
  • Data loss: Export isn't transactional, so small amounts of data can be lost during transient service events.

Query telemetry for your own measures

Application Insights offers two ways to read the spans. The Agents (preview) views visualize the telemetry without KQL:

  • Agent Runs: Agent invocations from the InvokeAgent spans, with duration, success, and the conversation each run belongs to.
  • Tools: The tools that agents call, how often they call them, and how the calls perform, aggregated from the ExecuteTool spans.
  • Models: The models invoked across runs and their call patterns.

For your own measures, open Logs and write a KQL query. The following query summarizes one agent's tool calls from the past seven days. Replace <Agent name> with your agent's name.

dependencies
| where timestamp > ago(7d)
| where name == "ExecuteTool"
| where tostring(customDimensions["gen_ai.agent.name"]) == "<Agent name>"
| extend ToolName = tostring(customDimensions["gen_ai.tool.name"])
| summarize Calls = count(), AvgDuration = avg(duration) by ToolName, resultCode
| order by ToolName asc, resultCode asc

The query keeps only the ExecuteTool spans for the named agent. It reads the tool name from customDimensions. For each tool and result code, it counts the calls and averages their duration. Rows with an ERROR result code show which tools fail and how often. A high average duration points to slow tool calls.

A query like this one becomes a measure you can reuse. Azure Monitor log search alert rules run a log query at regular intervals and check its results. A query that counts failed tool calls can therefore drive an alert. You can also pin query results to an Azure dashboard or a workbook.

The schema can change over time. Before you build an alert or a dashboard on a column or key, confirm that it's still present. To list the current columns and customDimensions keys, run the schema discovery queries in Environment-level telemetry with Application Insights.

Treat access to the Application Insights resource as access to conversation content. Spans include user prompts, agent replies, tool arguments and results, and user identifiers such as user.email and user.name. Anyone who can query the resource can read that content. Make sure the configuration and its access meet your organization's data residency, privacy, and role-based access control (RBAC) requirements.

Learn more in Log queries in Azure Monitor.

Reflect: For an agent you manage or plan to publish, which tool behavior would you want an alert on? Which span fields would your query use to measure it?