Azure monitor timeout errors

Heorhii Teriaiev 0 Reputation points

Hi there!

We run an application on AKS. Collecting k8s events to log analytics. We use Azure Managed Grafana with Azure Monitor as a source for alerting. Everyday we're getting DatasourceError alerts due to timeouts between Grafana and Azure monitor. Errors we're getting:

  • Error = failed to execute query A: request failed, status: 503 Service Unavailable
  • Error = failed to execute query A: request failed, status: 529
  • Error = failed to execute query A: Get ... Error = failed to execute query A: Get

I don't think we cause too much load. We have about 20 alerts in total. Intervals between evaluations are 5-15 minutes. Is this expected that Azure Monitor returns timeouts/is unavailable everyday? Is there anything we can tune to make it more reliable? Thank you in advance

Azure Kubernetes Service (AKS)
Azure Kubernetes Service (AKS)
An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
1,202 questions
Azure Managed Grafana
Azure Managed Grafana
An Azure service used to deploy Grafana dashboards for analytics and monitoring solutions.
38 questions
Azure Monitor
Azure Monitor
An Azure service that is used to collect, analyze, and act on telemetry data from Azure and on-premises environments.
1,942 questions
{count} votes

1 answer

Sort by: Most helpful
  1. JimmySalian-2011 29,741 Reputation points


    Seems like a config issue or timeout between the Grafana and Azure monitor, did you raise this query with Grafana? If not I will suggest you to put this same query with Grafana. However you can check the timeout settings for Azure Monitor APIs -

    Hope this helps



    Please Accept the answer if the information helped you. This will help us and others in the community as well.