Share via

Capture telemetry with Application Insights

  • Article

This article discusses how you can capture telemetry data from your Copilot Studio agent for use in Azure Application Insights.

In addition to the native analytics features within Copilot Studio, you can send telemetry data to Application Insights. Telemetry offers insights into your agent by tracking:

  • Logged messages and events sent to and from your agent
  • Topics to be triggered during user conversations
  • Custom telemetry events that can be sent from your topics

Important

Application Insights is a feature of Azure Monitor, an extensible Application Performance Management (APM) tool that allows you to monitor your live applications. It requires a subscription to Microsoft Azure.

Connect your Copilot Studio agent to Application Insights

To connect your agent to Application Insights, you first need to add your instrumentation key to your agent's configuration.

  1. Go to the Settings page for your agent, and select Advanced.

  2. Within the Application Insights section, populate the Connection string setting. See the Azure Monitor documentation to find out how to locate your connection string.

  3. Optionally, you can choose to enable one of the following settings.

    • Log activities: If enabled, details of incoming/outgoing messages and events are logged.

    • Log sensitive Activity properties: If enabled, the values of certain properties that could be considered sensitive on incoming/outgoing messages and events are included in logs. The properties that are considered potentially sensitive are userid, name, text and speak (text and speak properties only apply to messages).

Analyze bot telemetry with Application Insights

After you've connected your bot to Application Insights, telemetry data is logged when users interact with the bot, including testing within Copilot Studio. To see the logged telemetry data, navigate to the Logs section of your Application Insights resource in Azure.

From here, you can use Kusto queries to query and analyze your data. See example queries.

Example queries

A query can be as simple as specifying a single table, such as customEvents, which shows all custom telemetry events logged from Copilot Studio. But you can also use Kusto queries to narrow down your results further, including;

  • Adding a time interval
  • Extending your results using custom dimensions. Custom dimensions are custom properties that are logged, along with the prebuilt fields, such as timestamp or the event name
  • A where clause to limit the data returned based on a condition
  • Using more built-in Kusto functions to determine what and how information is shown

The example query below results in a line chart that shows how many distinct users communicated with your bot per day for the past 14 days.

Kusto 
let queryStartDate = ago(14d);
let queryEndDate = now();
let groupByInterval = 1d;
customEvents
| where timestamp > queryStartDate
| where timestamp < queryEndDate
| summarize uc=dcount(user_Id) by bin(timestamp, groupByInterval)
| render timechart

Important

The data within some fields varies and is more or less applicable, depending on the channel that is being used. For example, you'll only get a correct count of unique users in the query if they're authenticated users and their user ids are consistent across conversations. In anonymous scenarios where a random user id is generated per conversation, the user id field is less useful.

Excluding telemetry from test conversations from your queries

Your bot logs telemetry for all conversations, including those that happen within Copilot Studio during testing. If you want to exclude telemetry gathered during testing, you can extend your query with the designMode custom dimension that is captured on all events, and use a where clause in your query.

The following example shows all custom events, excluding those captured through the test canvas.

Kusto 
customEvents
| extend isDesignMode = customDimensions['designMode']
| where isDesignMode == "False"

Custom Dimensions

Much of the specific activity data received from Copilot Studio is stored in the customDimensions field. You can see a custom dimension field being used in a query to exclude telemetry from test conversations.

Field Description Sample Values
type Type of activity message, conversationUpdate, event, invoke
channelId Channel identifier emulator, directline, msteams, webchat
fromId From Identifier <id>
fromName Username from client John Bonham, Keith Moon, Steve Smith, Steve Gadd
locale Client origin locale en-us, zh-cn, en-GB, de-de, zh-CN
recipientId Recipient identifier <id>
recipientName Recipient name John Bonham, Keith Moon, Steve Smith, Steve Gadd
text Text in message find a coffee shop
designMode Conversation happened within the test canvas True / False