Share via

How to obtain detailed logs of agents created in Copilot Studio

Hasegawa 20 Reputation points
2026-06-04T01:08:52.2833333+00:00

I would like to obtain detailed logs of an agent created in Copilot Studio, specifically including:

Intermediate search queries generated during the response creation process

Information about accessed knowledge sources that were ultimately not used in the final response

Is it possible to retrieve such information using Application Insights?

If not, are there any alternative methods to obtain this level of detailed execution or reasoning data?

Microsoft Copilot | Microsoft 365 Copilot | Development
0 comments

Answer accepted by question author

Sayali-MSFT 6,021 Reputation points Microsoft External Staff Moderator
2026-06-04T09:16:32.0366667+00:00

Hello Hasegawa,

Application Insights in Copilot Studio can capture telemetry such as user messages, bot responses, topic triggers, events, and custom logs. However, it does not currently expose full internal reasoning details like intermediate search queries, discarded knowledge sources, semantic ranking scores, or hidden orchestration logic.

For deeper debugging, the best alternatives are:

  • Conversation snapshot/transcript diagnostics (dialog.json) from the test pane, which can reveal routing, tool calls, and orchestration behaviors.
  • Custom telemetry logging, where you explicitly log generated queries, retrieved documents, and filtering decisions into Application Insights.
  • Using custom RAG/Azure AI Foundry architectures if full retrieval and reasoning observability is required.

Useful Reference:
1.https://learn.microsoft.com/en-us/microsoft-copilot-studio/advanced-bot-framework-composer-capture-telemetry
2.https://github.com/MicrosoftDocs/dynamics365-guidance/blob/main/guidance/resources/copilot-studio-appinsights.md
3.https://microsoft.github.io/mcscatblog/posts/open-the-hood-copilot-studio-transcripts/

Was this answer helpful?

2 people found this answer helpful.

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest

  1. AI answer

    2026-06-04T01:09:08.43+00:00

    Application Insights can capture rich telemetry from Copilot Studio agents, but it does not provide full “reasoning traces” such as all intermediate search queries or a complete list of knowledge sources that were considered but not used in the final answer out of the box.

    What is available by default

    1. Application Insights integration
      • Copilot Studio can be connected to Azure Application Insights via a connection string in the agent’s Settings → Advanced → Application Insights.
      • Once connected, telemetry includes:
        • Logged messages and events sent to and from the agent.
        • Topics triggered during user conversations.
        • Custom telemetry events that authors explicitly log from topics.
      • Administrators can choose to log:
        • Incoming and outgoing messages/events.
        • Sensitive activity properties (userid, name, text, speak).
        • Node actions (in unified authoring, via the default integration).
      • Telemetry is then queryable in Application Insights using Kusto queries (for example, via the customEvents table) and can be visualized (time charts, filters, etc.).
    2. What this means for “detailed logs”
      • Out of the box, telemetry focuses on:
        • Conversation flow (which topics/nodes ran).
        • Messages exchanged.
        • Custom events that the author chooses to emit.
      • It does not automatically log:
        • All intermediate search queries generated by the orchestration layer.
        • A full list of all knowledge sources that were probed but not ultimately used in the final answer.

    How to get closer to the desired level of detail

    1. Custom telemetry events
      • Within topics, authors can use the Log custom telemetry event node to send arbitrary key–value data to Application Insights.
      • If the topic logic has access to details such as:
        • The query being sent to a knowledge source.
        • Which knowledge source or action is being invoked.
      • Then the topic can log a custom event (for example, KnowledgeSearchAttempt) with properties like queryText, sourceName, resultCount, or selectedForAnswer = true/false.
      • These events appear in customEvents and can be analyzed with Kusto queries.
    2. Activity logging options
      • Enabling Log activities and Log sensitive Activity properties increases the granularity of what is captured about messages and events, which can help correlate user inputs, topic execution, and any custom telemetry events that represent intermediate steps.
    3. Purview audit logs for agent usage
      • For tenant-wide governance and usage visibility, Microsoft Purview audit logs capture:
        • User interactions with Copilot agents (CopilotInteraction events).
        • Metadata such as user, time, organization, transcript thread ID.
        • References to accessed resources (files, documents, emails, etc.) in the DSPM for AI solution, including AccessedResources with Action, Type, Name, SensitivityLabelId, and whether an XPIA was detected.
      • These logs show which resources Copilot accessed to answer a prompt, but they are not a full reasoning trace and do not enumerate every internal search query.

    What is not supported

    • There is no documented capability to:
      • Automatically capture every intermediate search query generated by the agent’s internal orchestration.
      • Automatically log all knowledge sources that were evaluated but not used in the final response.
      • Expose the full internal “chain-of-thought” or reasoning trace of the model.

    Recommended approach

    1. Use Application Insights for:
      • Conversation-level telemetry (messages, topics, node actions).
      • Custom telemetry events that explicitly log the intermediate steps that the topic logic can see.
    2. Use Microsoft Purview for:
      • Compliance and governance visibility into which resources were accessed by Copilot and AI applications (including Copilot Studio agents), via audit logs and the DSPM for AI solution.
    3. Design topics with observability in mind:
      • Where possible, surface key intermediate decisions (which knowledge source was chosen, what query was sent, whether results were used) into topic logic and log them via custom telemetry events.

    This combination provides detailed operational and governance telemetry, but not a full, automatic reasoning trace of all internal search and selection steps.

    References:

    AI-generated content may be incorrect. Read our transparency notes for more information.

    Was this answer helpful?

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.