Edit

Use Azure Monitor managed service for Prometheus as data source for Grafana using managed system identity

  • Article

Azure Monitor managed service for Prometheus allows you to collect and analyze metrics at scale using a Prometheus-compatible monitoring solution. The most common way to analyze and present Prometheus data is with a Grafana dashboard. This article explains how to configure Prometheus as a data source for both Azure Managed Grafana and self-hosted Grafana running in an Azure virtual machine using managed system identity authentication.

For information on using Grafana with Active Directory, see Configure self-managed Grafana to use Azure Monitor managed Prometheus with Microsoft Entra ID.

Azure Managed Grafana

The following sections describe how to configure Azure Monitor managed service for Prometheus as a data source for Azure Managed Grafana.

Important

This section describes the manual process for adding an Azure Monitor managed service for Prometheus data source to Azure Managed Grafana. You can achieve the same functionality by linking the Azure Monitor workspace and Grafana workspace as described in Link a Grafana workspace.

Configure system identity

Your Grafana workspace requires the following settings:

  • System managed identity enabled
  • Monitoring Data Reader role for the Azure Monitor workspace

Both of these settings are configured by default when you created your Grafana workspace and linked it to an Azure Monitor workspace. Verify these settings on the Identity page for your Grafana workspace.

Screenshot of Identity page for Azure Managed Grafana.

Configure from Grafana workspace
Use the following steps to allow access all Azure Monitor workspaces in a resource group or subscription:

  1. Open the Identity page for your Grafana workspace in the Azure portal.

  2. If Status is No, change it to Yes.

  3. Select Azure role assignments to review the existing access in your subscription.

  4. If Monitoring Data Reader isn't listed for your subscription or resource group:

    1. Select + Add role assignment.

    2. For Scope, select either Subscription or Resource group.

    3. For Role, select Monitoring Data Reader.

    4. Select Save.

Configure from Azure Monitor workspace
Use the following steps to allow access to only a specific Azure Monitor workspace:

  1. Open the Access Control (IAM) page for your Azure Monitor workspace in the Azure portal.

  2. Select Add role assignment.

  3. Select Monitoring Data Reader and select Next.

  4. For Assign access to, select Managed identity.

  5. Select + Select members.

  6. For Managed identity, select Azure Managed Grafana.

  7. Select your Grafana workspace and then select Select.

  8. Select Review + assign to save the configuration.

Create Prometheus data source

Azure Managed Grafana supports Azure authentication by default.

  1. Open the Overview page for your Azure Monitor workspace in the Azure portal.

  2. Copy the Query endpoint, which you'll need in a step below.

  3. Open your Azure Managed Grafana workspace in the Azure portal.

  4. Select on the Endpoint to view the Grafana workspace.

  5. Select Configuration and then Data source.

  6. Select Add data source and then Prometheus.

  7. For URL, paste in the query endpoint for your Azure Monitor workspace.

  8. Select Azure Authentication to turn it on.

  9. For Authentication under Azure Authentication, select Managed Identity.

  10. Scroll to the bottom of the page and select Save & test.

Screenshot of configuration for Prometheus data source.

Self-managed Grafana

The following sections describe how to configure Azure Monitor managed service for Prometheus as a data source for self-managed Grafana on an Azure virtual machine.

Configure system identity

Azure virtual machines support both system assigned and user assigned identity. The following steps configure system assigned identity.

Configure from Azure virtual machine
Use the following steps to allow access all Azure Monitor workspaces in a resource group or subscription:

  1. Open the Identity page for your virtual machine in the Azure portal.

  2. If Status is No, change it to Yes.

  3. Select Save.

  4. Select Azure role assignments to review the existing access in your subscription.

  5. If Monitoring Data Reader isn't listed for your subscription or resource group:

    1. Select + Add role assignment.

    2. For Scope, select either Subscription or Resource group.

    3. For Role, select Monitoring Data Reader.

    4. Select Save.

Configure from Azure Monitor workspace
Use the following steps to allow access to only a specific Azure Monitor workspace:

  1. Open the Access Control (IAM) page for your Azure Monitor workspace in the Azure portal.

  2. Select Add role assignment.

  3. Select Monitoring Data Reader and select Next.

  4. For Assign access to, select Managed identity.

  5. Select + Select members.

  6. For Managed identity, select Virtual machine.

  7. Select your Grafana workspace and then click Select.

  8. Select Review + assign to save the configuration.

Create Prometheus data source

Versions 9.x and greater of Grafana support Azure Authentication, but it's not enabled by default. To enable this feature, you need to update your Grafana configuration. To determine where your Grafana.ini file is and how to edit your Grafana config, review the Configure Grafana document from Grafana Labs. Once you know where your configuration file lives on your VM, make the following update:

  1. Locate and open the Grafana.ini file on your virtual machine.

  2. Under the [auth] section of the configuration file, change the azure_auth_enabled setting to true.

  3. Under the [azure] section of the configuration file, change the managed_identity_enabled setting to true

  4. Open the Overview page for your Azure Monitor workspace in the Azure portal.

  5. Copy the Query endpoint, which you'll need in a step below.

  6. Open your Azure Managed Grafana workspace in the Azure portal.

  7. Click on the Endpoint to view the Grafana workspace.

  8. Select Configuration and then Data source.

  9. Click Add data source and then Prometheus.

  10. For URL, paste in the query endpoint for your Azure Monitor workspace.

  11. Select Azure Authentication to turn it on.

  12. For Authentication under Azure Authentication, select Managed Identity.

  13. Scroll to the bottom of the page and click Save & test.

Screenshot of configuration for Prometheus data source.

Frequently asked questions

This section provides answers to common questions.

I am missing all or some of my metrics. How can I troubleshoot?

You can use the troubleshooting guide for ingesting Prometheus metrics from the managed agent here.

Why am I missing metrics that have two labels with the same name but different casing?

Azure managed Prometheus is a case insensitive system. It treats strings, such as metric names, label names, or label values, as the same time series if they differ from another time series only by the case of the string. For more information, see Prometheus metrics overview.

I see some gaps in metric data, why is this occurring?

During node updates, you might see a 1-minute to 2-minute gap in metric data for metrics collected from our cluster level collectors. This gap occurs because the node that the data runs on is being updated as part of a normal update process. This update process affects cluster-wide targets such as kube-state-metrics and custom application targets that are specified. This occurs when your cluster is updated manually or via autoupdate. This behavior is expected and occurs due to the node it runs on being updated. This behavior doesn't affect any of our recommended alert rules.

Next steps