Collect Prometheus metrics from an Arc-enabled Kubernetes cluster (preview)
Article
This article describes how to configure your Azure Arc-enabled Kubernetes cluster (preview) to send data to Azure Monitor managed service for Prometheus. When you configure your Azure Arc-enabled Kubernetes cluster to send data to Azure Monitor managed service for Prometheus, a containerized version of the Azure Monitor agent is installed with a metrics extension. You then specify the Azure Monitor workspace where the data should be sent.
Note
The process described here doesn't enable Container insights on the cluster even though the Azure Monitor agent installed in this process is the same agent used by Container insights.
For different methods to enable Container insights on your cluster, see Enable Container insights. For details on adding Prometheus collection to a cluster that already has Container insights enabled, see Collect Prometheus metrics with Container insights.
You can use the following optional parameters with the previous commands:
--configurationsettings.AzureMonitorMetrics.KubeStateMetrics.MetricsLabelsAllowlist is a comma-separated list of Kubernetes label keys that will be used in the resource' labels metric. By default the metric contains only name and namespace labels. To include additional labels, provide a list of resource names in their plural form and Kubernetes label keys you would like to allow for them. For example, =namespaces=[kubernetes.io/team,...],pods=[kubernetes.io/team],...
--configurationSettings.AzureMonitorMetrics.KubeStateMetrics.MetricAnnotationsAllowList is a comma-separated list of Kubernetes annotations keys that will be used in the resource' labels metric. By default the metric contains only name and namespace labels. To include additional annotations, provide a list of resource names in their plural form and Kubernetes annotation keys you would like to allow for them. For example, =namespaces=[kubernetes.io/team,...],pods=[kubernetes.io/team],....
Note
A single *, for example '=pods=[*]' can be provided per resource to allow any labels, however, this has severe performance implications.
If the Azure Managed Grafana instance is in a subscription other than the Azure Monitor Workspaces subscription, register the Azure Monitor Workspace subscription with the Microsoft.Dashboard resource provider by following the steps in the Register resource provider section of the Azure resource providers and types article.
The Azure Monitor workspace and Azure Managed Grafana workspace must already exist.
The template must be deployed in the same resource group as the Azure Managed Grafana workspace.
Users with the User Access Administrator role in the subscription of the AKS cluster can enable the Monitoring Data Reader role directly by deploying the template.
Create an extension
Retrieve required values for the Grafana resource
Note
Azure Managed Grafana is not currently available in the Azure US Government cloud.
On the Overview page for the Azure Managed Grafana instance in the Azure portal, select JSON view.
If you're using an existing Azure Managed Grafana instance that's already linked to an Azure Monitor workspace, you need the list of already existing Grafana integrations. Copy the value of the azureMonitorWorkspaceIntegrations field. If the field doesn't exist, the instance hasn't been linked with any Azure Monitor workspace.
Edit the following fields' values in the parameter file.
Parameter
Value
azureMonitorWorkspaceResourceId
Resource ID for the Azure Monitor workspace. Retrieve from the JSON view on the Overview page for the Azure Monitor workspace.
azureMonitorWorkspaceLocation
Location of the Azure Monitor workspace. Retrieve from the JSON view on the Overview page for the Azure Monitor workspace.
clusterResourceId
Resource ID for the Arc cluster. Retrieve from the JSON view on the Overview page for the cluster.
clusterLocation
Location of the Arc cluster. Retrieve from the JSON view on the Overview page for the cluster.
metricLabelsAllowlist
Comma-separated list of Kubernetes labels keys to be used in the resource's labels metric.
metricAnnotationsAllowList
Comma-separated list of more Kubernetes label keys to be used in the resource's labels metric.
grafanaResourceId
Resource ID for the managed Grafana instance. Retrieve from the JSON view on the Overview page for the Grafana instance.
grafanaLocation
Location for the managed Grafana instance. Retrieve from the JSON view on the Overview page for the Grafana instance.
grafanaSku
SKU for the managed Grafana instance. Retrieve from the JSON view on the Overview page for the Grafana instance. Use the sku.name.
Open the template file and update the grafanaIntegrations property at the end of the file with the values that you retrieved from the Grafana instance. For example:
In the example JSON above, full_resource_id_1 and full_resource_id_2 are already in the Azure Managed Grafana resource JSON. They're added here to the Azure Resource Manager template (ARM template). If you don't have any existing Grafana integrations, don't include these entries.
The final azureMonitorWorkspaceResourceId entry is in the template by default and is used to link to the Azure Monitor workspace resource ID provided in the parameters file.
Verify extension installation status
Once you have successfully created the Azure Monitor extension for your Azure Arc-enabled Kubernetes cluster, you can check the status of the installation using the Azure portal or CLI. Successful installations show the status as Installed.
Azure portal
In the Azure portal, select the Azure Arc-enabled Kubernetes cluster with the extension installation.
From the resource pane on the left, select the Extensions item under the Setting' section.
An extension with the name azuremonitor-metrics is listed, with the current status in the Install status column.
Azure CLI
Run the following command to show the latest status of the Microsoft.AzureMonitor.Containers.Metrics extension.
az k8s-extension show \
--name azuremonitor-metrics \
--cluster-name <cluster-name> \
--resource-group <resource-group> \
--cluster-type connectedClusters
Delete the extension instance
To delete the extension instance, use the following CLI command:
az k8s-extension delete --name azuremonitor-metrics -g <cluster_resource_group> -c<cluster_name> -t connectedClusters
The command only deletes the extension instance. The Azure Monitor workspace and its data are not deleted.
Disconnected clusters
If your cluster is disconnected from Azure for more than 48 hours, Azure Resource Graph won't have information about your cluster. As a result, your Azure Monitor Workspace may have incorrect information about your cluster state.