Edit

Share via

Create an Agent Framework dashboard

In this guide, you learn how to create and customize an Agent Framework dashboard in Azure Managed Grafana. After adding instrumentation to send data to Application Insights, you can use this prebuilt dashboard to visualize and monitor the performance of your Agent Framework applications. This dashboard helps you track the health, performance, and cost of individual AI agents built with the Microsoft Agent Framework.

The Microsoft Agent Framework is an open-source development kit for building AI agents and multi-agent workflows in .NET and Python. The Agent Framework dashboard provides comprehensive monitoring of individual agent operations, including token usage, costs, errors, response times, and detailed trace analysis.

Note

For monitoring multi-agent workflows and graph-based execution flows, see Create an Agent Framework Workflow dashboard.

What you can monitor

The Agent Framework dashboard provides real-time insights into individual AI agent operations:

  • Performance monitoring: Track response times, success rates, and throughput to assess agent health and identify performance issues
  • Token usage and costs: Monitor token consumption per agent and estimate costs based on model pricing
  • Error analysis: Identify and debug agent-level errors with detailed trace information
  • Agent activity: Compare performance across different agents
  • Trace analysis: View detailed execution traces for individual agent operations with timing, dependencies, and span relationships

Prerequisites

Before you begin, ensure you have:

Import the prebuilt Agent Framework dashboard

Import the prebuilt Agent Framework dashboard into your Grafana workspace.

  1. In the Azure portal, open your Azure Managed Grafana workspace and select the Endpoint URL to open the Grafana portal.

  2. In the Grafana portal, go to Dashboards > New > Import.

  3. Under Find and import dashboards, enter the dashboard ID 24156.

  4. Select Load.

  5. Configure the import settings:

    • Name: Optionally customize the dashboard name.
    • Folder: Select a folder to organize your dashboard.
    • Unique identifier (UID): Leave as default or customize.
    • Azure Monitor: Select your Azure Monitor data source from the dropdown.

    Note

    Ensure your Azure Managed Grafana workspace has the Monitoring Reader role on the Application Insights resource. If not, assign the role to the workspace's managed identity.

  6. Select Import.

  7. After importing the dashboard, use the dropdown selectors at the top of the dashboard to filter by agent name, operation type, and time range.

Tip

You can also access this dashboard directly from the Azure portal. Go to Monitor > Dashboards with Grafana (preview), and select Agent Framework, or use the direct link: Agent Framework dashboard.

Screenshot of Grafana showing Agent Framework performance metrics.

Screenshot of Grafana showing Agent Framework traces panels.

Screenshot of Grafana showing Agent Framework detailed metrics panels.

Customize the dashboard

Customize the dashboard to your specific monitoring needs.

To add a new panel:

  1. Select Edit > Add > Visualization at the top of the dashboard.

  2. Configure the query:

    • Data source: Select Azure Monitor.
    • Query type: Select Traces or Logs depending on the panel.
    • Resource: Choose your Application Insights resource.
    • Query: Write a KQL query to retrieve agent telemetry. For example: 
      traces
| where customDimensions.AgentName != ""
| summarize avg(duration) by bin(timestamp, 5m), tostring(customDimensions.AgentName)

  3. Configure visualization options:

    • Panel title: Enter a descriptive title.
    • Visualization type: Choose from Time series, Stat, Gauge, Bar chart, Table, or other types.
    • Unit: Set the appropriate unit (milliseconds, count, currency, etc.).
    • Thresholds: Define warning and critical thresholds for visual alerts.

  4. Select Apply to add the panel to your dashboard.

To customize cost estimation:

  1. Locate the "Daily Cost Estimation" panel and select Edit.

  2. Update the cost calculation in the KQL query to match your model pricing:

    // Example: Adjust these values to your model's pricing
input_cost = input_tokens * 0.00000025,   // Your input token rate
output_cost = output_tokens * 0.000002    // Your output token rate

  3. Select Apply and save the dashboard.

Technical details

The dashboard queries Azure Application Insights using the following setup:

  • Data source: Application Insights with OpenTelemetry instrumentation
  • Key metrics tracked:
    • Agent operations (count, duration, success rate)
    • Token usage (input tokens, output tokens)
    • Cost estimation (based on configurable token pricing)
    • Error rates and error details
    • Trace visualization with span relationships
  • Grouping: Metrics split by agent name, operation type, and status code
  • Aggregations: Count for throughput, average/percentiles for latency, sum for tokens and costs
  • Instrumentation: Requires OpenTelemetry SDK configuration in your Agent Framework application

For detailed instrumentation instructions, see:

Related content

  • Last updated on