Guidance Required on Production Observability Best Practices for Custom AI Teammate in Agent 365

Question

Hi Team,

We are implementing a custom agent using Microsoft Agent 365 as an AI Teammate. Users create agent identities from a blueprint and use those agents within Microsoft Teams.

As part of our implementation, we have integrated the Agent 365 Observability APIs and currently capture the following metrics:

• Agent Activity
  • Session count
    • Active users
      • Exception count
        • Agent runtime
• Agent Map
  • Total tool invocations
  • Failed tool executions

We would like to understand the recommended observability practices for a production deployment. Specifically, we have the following questions:

1. Tool Exception Details
   • Should we also capture and expose the reason for tool execution failures?
     • If yes, where can these details be viewed in the Microsoft 365 Admin Center or other observability dashboards?
     1. Session Failure Details
        • If an agent session fails, should we report the session failure along with the failure reason?
          • If yes, where is this information expected to appear in the Microsoft observability experience?
          1. Additional Recommendations
             • Besides the metrics listed above, are there any additional telemetry events, metrics, or best practices that Microsoft recommends for production-grade observability of custom AI Teammates?

Our goal is to align our implementation with Microsoft's recommended production standards for Agent 365 observability.

Thank you for your guidance.

Answer 1

Hi @Akshay Babar

Capturing the reason for tool execution failures in addition to the failed execution count is recommended. Recording details such as the tool name, error category, and correlation identifiers provides much better operational visibility and simplifies troubleshooting. Agent 365 observability is built on OpenTelemetry and is intended to capture telemetry for agent invocations, tool calls, and exceptions as part of the overall execution trace.

Similarly, if an agent session fails, it is recommended to capture the failure reason along with relevant context, such as the failed tool, failed step, or dependency involved.

Telemetry collected through Agent 365 observability flows into Microsoft observability experiences, including Microsoft 365 Admin Center, Microsoft Defender, and Microsoft Purview. Detailed diagnostics are most valuable when emitted as structured OpenTelemetry trace and span data, allowing failures and exceptions to be correlated with specific sessions, runs, and tool invocations.

In addition to the metrics you are already capturing, I recommend tracking success and failure rates, end-to-end and tool-level latency, dependency health, timeout rates, and correlation attributes such as agentId, conversationId, and traceId.

Ref: https://learn.microsoft.com/en-us/microsoft-agent-365/developer/observability-concepts

https://learn.microsoft.com/en-us/microsoft-agent-365/developer/observability?tabs=python

I hope this information helps.