Configure remote write for Azure Monitor managed service for Prometheus using managed identity authentication

Article
09/10/2023

This article describes how to configure remote-write to send data from self-managed Prometheus running in your AKS cluster or Azure Arc-enabled Kubernetes cluster using managed identity authentication. You either use an existing identity created by AKS or create one of your own. Both options are described here.

Cluster configurations

This article applies to the following cluster configurations:

Azure Kubernetes service (AKS)
Azure Arc-enabled Kubernetes cluster

Note

For a Kubernetes cluster running in another cloud or on-premises, see Azure Monitor managed service for Prometheus remote write - Microsoft Entra ID.

Prerequisites

See prerequisites at Azure Monitor managed service for Prometheus remote write.

Locate AKS node resource group

The node resource group of the AKS cluster contains resources that you will require for other steps in this process. This resource group has the name MC_<AKS-RESOURCE-GROUP>_<AKS-CLUSTER-NAME>_<REGION>. You can locate it from the Resource groups menu in the Azure portal. Start by making sure that you can locate this resource group since other steps below will refer to it.

Get the client ID of the user assigned identity

You will require the client ID of the identity that you're going to use. Note this value for use in later steps in this process.

Get the Client ID from the Overview page of your managed identity.

Instead of creating your own ID, you can use one of the identities created by AKS, which are listed in Use a managed identity in Azure Kubernetes Service. This article uses the Kubelet identity. The name of this identity is <AKS-CLUSTER-NAME>-agentpool and located in the node resource group of the AKS cluster.

Assign Monitoring Metrics Publisher role on the data collection rule to the managed identity

The managed identity requires the Monitoring Metrics Publisher role on the data collection rule associated with your Azure Monitor workspace.

From the menu of your Azure Monitor Workspace account, select the Data collection rule to open the Overview page for the data collection rule.
Select Access control (IAM) in the Overview page for the data collection rule.
Select Add and then Add role assignment.
Select Monitoring Metrics Publisher role and select Next.
Select Managed Identity and then select Select members. Choose the subscription the user assigned identity is located in and then select User-assigned managed identity. Select the User Assigned Identity that you're going to use and click Select.
Select Review + assign to complete the role assignment.

Grant AKS cluster access to the identity

This step isn't required if you're using an AKS identity since it will already have access to the cluster.

Important

You must have owner/user access administrator access on the cluster.

Identify the virtual machine scale sets in the node resource group for your AKS cluster.

Run the following command in Azure CLI for each virtual machine scale set.

az vmss identity assign -g <AKS-NODE-RESOURCE-GROUP> -n <AKS-VMSS-NAME> --identities <USER-ASSIGNED-IDENTITY-RESOURCE-ID>

Deploy Side car and configure remote write on the Prometheus server

Copy the YAML below and save to a file. This YAML assumes you're using 8081 as your listening port. Modify that value if you use a different port.

prometheus:
  prometheusSpec:
    externalLabels:
          cluster: <AKS-CLUSTER-NAME>

    ## https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write    
    remoteWrite:
    - url: 'http://localhost:8081/api/v1/write'
  ## Azure Managed Prometheus currently exports some default mixins in Grafana. 
  ## These mixins are compatible with Azure Monitor agent on your Azure Kubernetes Service cluster. 
  ## However, these mixins aren't compatible with Prometheus metrics scraped by the Kube Prometheus stack. 
  ## In order to make these mixins compatible, uncomment remote write relabel configuration below:

  ## writeRelabelConfigs:
  ##   - sourceLabels: [metrics_path]
  ##     regex: /metrics/cadvisor
  ##     targetLabel: job
  ##     replacement: cadvisor
  ##     action: replace
  ##   - sourceLabels: [job]
  ##     regex: 'node-exporter'
  ##     targetLabel: job
  ##     replacement: node
  ##     action: replace
    containers:
    - name: prom-remotewrite
      image: <CONTAINER-IMAGE-VERSION>
      imagePullPolicy: Always
      ports:
        - name: rw-port
          containerPort: 8081
      livenessProbe:
        httpGet:
          path: /health
          port: rw-port
        initialDelaySeconds: 10
        timeoutSeconds: 10
      readinessProbe:
        httpGet:
          path: /ready
          port: rw-port
        initialDelaySeconds: 10
        timeoutSeconds: 10
      env:
      - name: INGESTION_URL
        value: <INGESTION_URL>
      - name: LISTENING_PORT
        value: '8081'
      - name: IDENTITY_TYPE
        value: userAssigned
      - name: AZURE_CLIENT_ID
        value: <MANAGED-IDENTITY-CLIENT-ID>
      # Optional parameter
      - name: CLUSTER
        value: <CLUSTER-NAME>

Replace the following values in the YAML.

Value	Description
`<AKS-CLUSTER-NAME>`	Name of your AKS cluster
`<CONTAINER-IMAGE-VERSION>`	`mcr.microsoft.com/azuremonitor/prometheus/promdev/prom-remotewrite:prom-remotewrite-20230906.1` The remote write container image version.
`<INGESTION-URL>`	Metrics ingestion endpoint from the Overview page for the Azure Monitor workspace
`<MANAGED-IDENTITY-CLIENT-ID>`	Client ID from the Overview page for the managed identity
`<CLUSTER-NAME>`	Name of the cluster Prometheus is running on

Open Azure Cloud Shell and upload the YAML file.

Use helm to apply the YAML file to update your Prometheus configuration with the following CLI commands.

# set context to your cluster 
az aks get-credentials -g <aks-rg-name> -n <aks-cluster-name> 

# use helm to update your remote write config 
helm upgrade -f <YAML-FILENAME>.yml prometheus prometheus-community/kube-prometheus-stack --namespace <namespace where Prometheus pod resides>

Verification and troubleshooting

See Azure Monitor managed service for Prometheus remote write.