Best approach to extract and centralize analytics for multiple Copilot Studio agents for Power BI reporting

1. Best architecture to centralize analytics across multiple agents

A layered architecture using Dataverse as the system of record, extended with Microsoft analytics and reporting tools, is recommended:

1. Dataverse as core data store
   • Copilot Studio already writes conversation transcripts and custom analytics data to Dataverse tables:
     • conversationtranscript for detailed conversation data across all agents.
     • bot for agent metadata.
     • botcomponent for topics, entities, and dialogs.
   • These tables naturally support cross‑agent analytics because they store data for all agents in the environment.
2. Long‑term storage and analytics layer
   • Default retention for conversation data is 30 days in both Copilot Studio’s internal analytics service and Dataverse. For enterprise‑scale, enable longer‑term storage by:
     • Extending Dataverse retention for conversation transcripts where appropriate.
     • Exporting Dataverse data (especially conversationtranscript) to a cost‑effective store such as Azure Data Lake Storage Gen2 using Azure Synapse Link for Dataverse, as described in the compliance and auditing guidance.
3. Reporting and insights layer
   • Use a combination of:
     • Power BI over Dataverse and/or exported data in Azure Data Lake.
     • Copilot Studio Kit for prebuilt KPIs and sample Power BI reports.
     • Viva Insights Copilot Studio agents report for tenant‑wide, cross‑agent rollups and business value metrics.
4. Optional telemetry and security layer
   • For operational telemetry and incident response, send custom events to Azure Application Insights and surface them via Microsoft Sentinel and Microsoft Purview audit logs.

This architecture centralizes analytics across all agents, supports long‑term trend analysis, and separates operational telemetry from business KPIs.

2. Dataverse vs. other sources (Application Insights, Log Analytics, Purview)

Each source serves a distinct purpose; Dataverse is the primary source for conversation analytics:

• Dataverse (primary analytics source)
  • Stores conversation transcripts and custom analytics data.
  • Tables used for analytics:
    • bot (Copilot table) – agent details.
    • botcomponent – topics, entities, dialogs.
    • conversationtranscript – detailed conversation data for all agents.
  • Best suited for:
    • Usage metrics.
    • Conversation performance.
    • Transcript‑level analysis.
    • Long‑term KPI storage when combined with Copilot Studio Kit and/or export to Data Lake.
• Copilot Studio Kit (Dataverse‑based KPIs)
  • Stores aggregated metrics and KPIs in Dataverse.
  • Benefits:
    • Aggregated metrics for agents in Dataverse.
    • Long‑term KPI retention.
    • Ability to track variables such as NPS.
    • Full conversation transcripts saved alongside KPI records.
    • Sample Power BI reports that can be extended.
• Azure Application Insights
  • Receives custom telemetry events from agents (via Bot Framework Composer telemetry integration).
  • Best suited for:
    • Logged messages and events to/from agents.
    • Topic triggering behavior.
    • Custom telemetry events for deep operational diagnostics.
• Microsoft Sentinel + Microsoft Purview
  • Purview: audit Copilot Studio activities, data modifications, and user actions.
  • Sentinel: ingest Purview audit logs and create custom detection rules.
  • Best suited for:
    • Compliance and auditing.
    • Threat detection and incident response.
• Viva Insights
  • Receives aggregated Copilot Studio agent data via Microsoft 365 services.
  • Provides tenant‑wide, cross‑agent analytics and business impact views.

For enterprise analytics, Dataverse (plus Synapse Link to Data Lake) is the recommended primary source for conversation and KPI data, complemented by Application Insights for telemetry and Purview/Sentinel for governance.

3. Best way to export or stream data continuously for Power BI

Use a combination of Dataverse retention, export, and Viva Insights integration:

1. Dataverse retention and export
   • Conversation transcripts are stored in the Dataverse conversationtranscripts table with a default 30‑day retention.
   • Extend the retention period for conversation transcripts in Dataverse when needed.
   • For long‑term, scalable storage and analytics:
     • Export raw conversation transcripts to Azure Data Lake Storage Gen2 using Azure Synapse Link for Dataverse.
     • This enables continuous, near‑real‑time replication of Dataverse tables to Data Lake for Power BI to consume.
2. Copilot Studio Kit KPIs
   • Use Copilot Studio Kit to:
     • Store aggregated KPIs and conversation metrics in Dataverse.
     • Retain KPI data for long‑term tracking.
   • Connect Power BI directly to these Dataverse KPI tables or to their exported copies in Data Lake.
3. Viva Insights integration
   • Enable data flow from Copilot Studio to Viva Insights via Microsoft 365 services.
   • Configure:
     • Microsoft 365 services setting for Copilots and generative AI features.
     • “Sharing Copilot Studio agent data with Viva Insights” in Power Platform admin center.
     • Cross‑geo boundary sharing if Viva Insights and Copilot Studio environments are in different regions.
   • Use the Copilot Studio agents report Power BI template from the Viva Insights analyst experience for tenant‑wide analytics.
4. Recommended Power BI templates and best practices
5. Copilot Studio Kit Power BI reports
   • Copilot Studio Kit includes sample Power BI reports for conversation KPIs.
   • These can be extended to:
     • Add custom KPIs (e.g., NPS).
     • Combine with other Dataverse tables or external data sources.
6. Copilot Studio agents report (Viva Insights)
   • A Power BI template that:
     • Combines all agents in the tenant into a single set of dashboards.
     • Provides pages such as Agent Overview, Top agents used, Conversational Agent Impact, Autonomous Agent Impact.
     • Includes Agent Assisted Hours and Agent Assisted Value, with a configurable productivity rate.
     • Offers cross‑agent rollups (compare multiple agents at once).
     • Provides trend charts on engagement, satisfaction, and resolution beyond the native 30‑day Copilot Studio window.
     • Shows side‑by‑side conversational and autonomous metrics.
   • Requires at least 50 Copilot licenses and one published agent in a production environment.
7. Power BI over exported transcripts and KPIs
   • Build custom models that:
     • Join conversationtranscript data with KPI tables from Copilot Studio Kit.
     • Aggregate by agent, topic, channel, and time for trend analysis.
     • Incorporate sentiment or NPS variables defined in KPIs.
8. Power BI tips and troubleshooting
   • Use the Power BI FAQ and troubleshooting guidance associated with Viva Insights templates for sharing and performance best practices.
9. Additional data sources for complete analysis (operational + governance + usage)

To achieve a comprehensive view, combine the following:

1. Operational and performance data
   • Azure Application Insights telemetry:
     • Logged messages and events.
     • Topic triggers.
     • Custom telemetry events.
   • Use this for performance monitoring, error analysis, and topic‑level behavior.
2. Usage and conversation analytics
   • Dataverse:
     • conversationtranscript for raw conversations.
     • bot and botcomponent for agent and topic metadata.
   • Copilot Studio Kit:
     • Aggregated KPIs, NPS, and long‑term metrics.
3. Business value and cross‑agent impact
   • Viva Insights Copilot Studio agents report:
     • Agent Assisted Hours and Value.
     • Cross‑agent rollups and trend charts.
   • Copilot business impact report (in Viva Insights):
     • Joins Copilot usage with business outcome metrics (e.g., deals closed, cases resolved, on‑time delivery, claims processed).
4. Governance, compliance, and security
   • Microsoft Purview:
     • Audit Copilot Studio activities, data modifications, and user actions.
   • Microsoft Sentinel:
     • Ingest Purview audit logs and create custom detection rules for Copilot Studio events.
   • Power Platform admin center:
     • Monitor message usage, capacity, and dependencies (Power Automate, prompt builder) to prevent overages and throttling.
     • Use the Copilot area for recommendations, alerts, and content moderation (blocked queries).
5. Conversation transcript analysis with AI
   • Use the reference architecture for extracting and analyzing agent conversation transcripts with AI to:
     • Uncover key insights.
     • Improve accuracy.
     • Enhance decision‑making.
   • The Copilot Studio Kit conversation analyzer can help analyze transcripts from custom agents.

By combining Dataverse (and Synapse Link), Copilot Studio Kit, Viva Insights templates, Application Insights, and Purview/Sentinel, it is possible to build scalable, centralized Power BI analytics that cover usage, performance, business value, and governance across all Copilot Studio agents.

References:

• Develop a custom analytics strategy
• Analytics considerations for Copilot Studio vs Dynamics 365 Omnichannel
• Copilot Studio agents report
• Measure and improve agent performance with KPIs and analytics
• View aggregated analytics in Viva Insights
• Monitor operations, compliance, and capacity
• Communicate the value story to stakeholders