Edit

Send Prometheus metrics to multiple Azure Monitor workspaces

  • Article

Routing different metrics to more Azure Monitor workspaces can be done through the creation of additional data collection rules.

You can create Data Collection Rules with corresponding Data Collection Endpoints for different metrics to be sent to additional Azure Monitor workspaces from the same Kubernetes cluster. Currently, this is only available through onboarding through Resource Manager templates. You can follow the regular onboarding process and then edit the same Resource Manager templates to add additional DCRs and DCEs for your additional Azure Monitor workspaces. You'll need to edit the template to add an additional parameters for every additional Azure Monitor workspace, add another DCR for every additional Azure Monitor workspace, add another DCE, add the Monitor Reader Role for the new Azure Monitor workspace and add an additional Azure Monitor workspace integration for Grafana.

  • Add the following parameters:

    "parameters": {
  "azureMonitorWorkspaceResourceId2": {
    "type": "string"
  },
  "azureMonitorWorkspaceLocation2": {
    "type": "string",
    "defaultValue": "",
    "allowedValues": [
      "eastus2euap",
      "centraluseuap",
      "centralus",
      "eastus",
      "eastus2",
      "northeurope",
      "southcentralus",
      "southeastasia",
      "uksouth",
      "westeurope",
      "westus",
      "westus2"
    ]
  },
...
}

  • Add an additional Data Collection Endpoint. You must replace <dceName>:

      {
    "type": "Microsoft.Insights/dataCollectionEndpoints",
    "apiVersion": "2021-09-01-preview",
    "name": "[variables('dceName')]",
    "location": "[parameters('azureMonitorWorkspaceLocation2')]",
    "kind": "Linux",
    "properties": {}
  }

  • Add an additional DCR with the new Data Collection Endpoint. You must replace <dcrName>:

    {
  "type": "Microsoft.Insights/dataCollectionRules",
  "apiVersion": "2021-09-01-preview",
  "name": "<dcrName>",
  "location": "[parameters('azureMonitorWorkspaceLocation2')]",
  "kind": "Linux",
  "properties": {
    "dataCollectionEndpointId": "[resourceId('Microsoft.Insights/dataCollectionEndpoints/', variables('dceName'))]",
    "dataFlows": [
      {
        "destinations": ["MonitoringAccount2"],
        "streams": ["Microsoft-PrometheusMetrics"]
      }
    ],
    "dataSources": {
      "prometheusForwarder": [
        {
          "name": "PrometheusDataSource",
          "streams": ["Microsoft-PrometheusMetrics"],
          "labelIncludeFilter": 
                  "microsoft_metrics_include_label": "MonitoringAccountLabel2"
        }
      ]
    },
    "description": "DCR for Azure Monitor Metrics Profile (Managed Prometheus)",
    "destinations": {
      "monitoringAccounts": [
        {
          "accountResourceId": "[parameters('azureMonitorWorkspaceResourceId2')]",
          "name": "MonitoringAccount2"
        }
      ]
    }
  },
  "dependsOn": [
    "[resourceId('Microsoft.Insights/dataCollectionEndpoints/', variables('dceName'))]"
  ]
}

  • Add an additional Data Collection Rule Association (DCRA) with the relevant Data Collection Rule (DCR). This associates the DCR with the cluster. You must replace <dcraName>:

    {
 "type": "Microsoft.Resources/deployments",
 "name": "<dcraName>",
 "apiVersion": "2017-05-10",
 "subscriptionId": "[variables('clusterSubscriptionId')]",
 "resourceGroup": "[variables('clusterResourceGroup')]",
 "dependsOn": [
   "[resourceId('Microsoft.Insights/dataCollectionEndpoints/', variables('dceName'))]",
   "[resourceId('Microsoft.Insights/dataCollectionRules', variables('dcrName'))]"
 ],
 "properties": {
   "mode": "Incremental",
   "template": {
     "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
     "contentVersion": "1.0.0.0",
     "parameters": {},
     "variables": {},
     "resources": [
       {
         "type": "Microsoft.ContainerService/managedClusters/providers/dataCollectionRuleAssociations",
         "name": "[concat(variables('clusterName'),'/microsoft.insights/', variables('dcraName'))]",
         "apiVersion": "2021-09-01-preview",
         "location": "[parameters('clusterLocation')]",
         "properties": {
           "description": "Association of data collection rule. Deleting this association will break the data collection for this AKS Cluster.",
           "dataCollectionRuleId": "[resourceId('Microsoft.Insights/dataCollectionRules', variables('dcrName'))]"
         }
       }
     ]
   },
   "parameters": {}
 }
}

  • Add an additional Grafana integration:

    {
      "type": "Microsoft.Dashboard/grafana",
      "apiVersion": "2022-08-01",
      "name": "[split(parameters('grafanaResourceId'),'/')[8]]",
      "sku": {
        "name": "[parameters('grafanaSku')]"
      },
      "location": "[parameters('grafanaLocation')]",
      "properties": {
        "grafanaIntegrations": {
          "azureMonitorWorkspaceIntegrations": [
            // Existing azureMonitorWorkspaceIntegrations values (if any)
            // {
            //   "azureMonitorWorkspaceResourceId": "<value>"
            // },
            // {
            //   "azureMonitorWorkspaceResourceId": "<value>"
            // },
            {
              "azureMonitorWorkspaceResourceId": "[parameters('azureMonitorWorkspaceResourceId')]"
            },
            {
              "azureMonitorWorkspaceResourceId": "[parameters('azureMonitorWorkspaceResourceId2')]"
            }
          ]
        }
      }
    }
    • Assign Monitoring Data Reader role to read data from the new Azure Monitor Workspace:
    {
  "type": "Microsoft.Authorization/roleAssignments",
  "apiVersion": "2022-04-01",
  "name": "[parameters('roleNameGuid')]",
  "scope": "[parameters('azureMonitorWorkspaceResourceId2')]",
  "properties": {
      "roleDefinitionId": "[concat('/subscriptions/', variables('clusterSubscriptionId'), '/providers/Microsoft.Authorization/roleDefinitions/', 'b0d8363b-8ddd-447d-831f-62ca05bff136')]",
      "principalId": "[reference(resourceId('Microsoft.Dashboard/grafana', split(parameters('grafanaResourceId'),'/')[8]), '2022-08-01', 'Full').identity.principalId]"
  }
}

Then configure which metrics are routed to which workspace, by adding an extra pre-defined label, microsoft_metrics_account to the metrics. The value should be the same as the corresponding microsoft_metrics_include_label in the DCR for that workspace. To add the label to the metrics, you can utilize relabel_configs in your scrape config. To send all metrics from one job to a certain workspace, add the following relabel config:

relabel_configs:
- source_labels: [__address__]
  target_label: microsoft_metrics_account
  action: replace
  replacement: "MonitoringAccountLabel2"

The source label is __address__ because this label will always exist so this relabel config will always be applied. The target label will always be microsoft_metrics_account and its value should be replaced with the corresponding label value for the workspace.

Example

If you want to configure three different jobs to send the metrics to three different workspaces, then include the following in each data collection rule:

"labelIncludeFilter": {
  "microsoft_metrics_include_label": "MonitoringAccountLabel1"
}

"labelIncludeFilter": {
  "microsoft_metrics_include_label": "MonitoringAccountLabel2"
}

"labelIncludeFilter": {
  "microsoft_metrics_include_label": "MonitoringAccountLabel3"
}

Then in your scrape config, include the same label value for each:

scrape_configs:
- job_name: prometheus_ref_app_1
  kubernetes_sd_configs:
    - role: pod
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_app]
      action: keep
      regex: "prometheus-reference-app-1"
    - source_labels: [__address__]
      target_label: microsoft_metrics_account
      action: replace
      replacement: "MonitoringAccountLabel1"
- job_name: prometheus_ref_app_2
  kubernetes_sd_configs:
    - role: pod
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_app]
      action: keep
      regex: "prometheus-reference-app-2"
    - source_labels: [__address__]
      target_label: microsoft_metrics_account
      action: replace
      replacement: "MonitoringAccountLabel2"
- job_name: prometheus_ref_app_3
  kubernetes_sd_configs:
    - role: pod
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_app]
      action: keep
      regex: "prometheus-reference-app-3"
    - source_labels: [__address__]
      target_label: microsoft_metrics_account
      action: replace
      replacement: "MonitoringAccountLabel3"

Next steps