Edit

Deep investigation examples in the Azure Copilot Observability Agent

A deep investigation is a troubleshooting workflow run by the Azure Copilot Observability Agent. The following examples show how a deep investigation can connect application, infrastructure, dependency, and database signals to explain why failures occur and what to do next.

Virtual machine performance regression

Here’s an example of a latency alert that fires on an Application Insights resource backed by a virtual machine:

  1. The agent uses the alert’s resource and time range to scope the problem.
  2. It expands scope from the application to the host virtual machine and the dependencies the application calls.
  3. It correlates signals across layers: a CPU spike on the virtual machine (based on metric anomaly), increased latency and error rates in application requests (based on logs analysis), and a deployment to the virtual machine just before the spike (Activity Log).
  4. The agent concludes that the application latency spike is likely caused by CPU pressure, which in turn was probably caused by the recent deployment.

Suggested next steps: scale out the virtual machine or roll back the deployment.

VM investigation flow from latency symptoms to CPU anomaly root cause and scaling recommendations.

AKS pods stuck in Pending after a rollout

Here’s an example of an AKS workload where pods from a recent rollout sit in Pending and never reach Running:

  1. The agent scopes to the workload and its hosting cluster.
  2. It analyzes signals at three levels: the workload (pod state and Kubernetes events), the node (CPU and memory near capacity), and the cluster (resource trend over recent hours).
  3. It correlates the workload’s Pending state with node-pool saturation (based on node CPU and memory metrics) and surfaces the matching scheduler events.
  4. The agent concludes that the pods can’t be scheduled because the node pool is at capacity.

Suggested next steps: scale out the node pool or right-size the workload’s requests.

AKS investigation flow from pending pods to node-pool saturation root cause and scaling recommendations.

Agent failures caused by backend SQL saturation

This pattern is common across Foundry agents, Foundry GenAI agents, and other application-hosted agents when the backend database is briefly saturated.

  1. The agent scopes to the affected agent or application, its dependencies and tool calls, and the SQL backend.
  2. It analyzes cross-layer signals, including runs and errors, dependency/API failures, logs, and SQL resource metrics.
  3. It correlates failures with SQL timeouts and a spike in failed dependency calls at incident start.
  4. It rules out alternatives such as cache issues, unrelated dependency failures, or broader service outage.
  5. It identifies a short SQL saturation window (often driven by a high-cost query) and concludes that transient SQL pressure caused the failures.

Suggested next steps: optimize expensive queries, improve backend resilience, and monitor SQL resource utilization during peak periods.

Related content

  • Last updated on