Edit

Send Prometheus metrics to multiple Azure Monitor workspaces

  • Article

Routing metrics to more Azure Monitor workspaces can be done through the creation of additional data collection rules. All metrics can be sent to all workspaces or different metrics can be sent to different workspaces.

Send same metrics to multiple Azure Monitor workspaces

You can create multiple Data Collection Rules that point to the same Data Collection Endpoint for metrics to be sent to additional Azure Monitor workspaces from the same Kubernetes cluster. In case you have a very high volume of metrics, a new Data Collection Endpoint can be created as well. Please refer to the service limits document regarding ingestion limits. Currently, this is only available through onboarding through Resource Manager templates. You can follow the regular onboarding process and then edit the same Resource Manager templates to add additional DCRs and DCEs (if applicable) for your additional Azure Monitor workspaces. You'll need to edit the template to add an additional parameters for every additional Azure Monitor workspace, add another DCR for every additional Azure Monitor workspace, add another DCE (if applicable), add the Monitor Reader Role for the new Azure Monitor workspace and add an additional Azure Monitor workspace integration for Grafana.

  • Add the following parameters:

    "parameters": {
  "azureMonitorWorkspaceResourceId2": {
    "type": "string"
  },
  "azureMonitorWorkspaceLocation2": {
    "type": "string",
    "defaultValue": "",
    "allowedValues": [
      "eastus2euap",
      "centraluseuap",
      "centralus",
      "eastus",
      "eastus2",
      "northeurope",
      "southcentralus",
      "southeastasia",
      "uksouth",
      "westeurope",
      "westus",
      "westus2"
    ]
  },
...
}

  • For high metric volume, add an additional Data Collection Endpoint. You must replace <dceName>:

      {
    "type": "Microsoft.Insights/dataCollectionEndpoints",
    "apiVersion": "2021-09-01-preview",
    "name": "[variables('dceName')]",
    "location": "[parameters('azureMonitorWorkspaceLocation2')]",
    "kind": "Linux",
    "properties": {}
  }

  • Add an additional DCR with the same or a different Data Collection Endpoint. You must replace <dcrName>:

    {
  "type": "Microsoft.Insights/dataCollectionRules",
  "apiVersion": "2021-09-01-preview",
  "name": "<dcrName>",
  "location": "[parameters('azureMonitorWorkspaceLocation2')]",
  "kind": "Linux",
  "properties": {
    "dataCollectionEndpointId": "[resourceId('Microsoft.Insights/dataCollectionEndpoints/', variables('dceName'))]",
    "dataFlows": [
      {
        "destinations": ["MonitoringAccount2"],
        "streams": ["Microsoft-PrometheusMetrics"]
      }
    ],
    "dataSources": {
      "prometheusForwarder": [
        {
          "name": "PrometheusDataSource",
          "streams": ["Microsoft-PrometheusMetrics"],
          "labelIncludeFilter": {}
        }
      ]
    },
    "description": "DCR for Azure Monitor Metrics Profile (Managed Prometheus)",
    "destinations": {
      "monitoringAccounts": [
        {
          "accountResourceId": "[parameters('azureMonitorWorkspaceResourceId2')]",
          "name": "MonitoringAccount2"
        }
      ]
    }
  },
  "dependsOn": [
    "[resourceId('Microsoft.Insights/dataCollectionEndpoints/', variables('dceName'))]"
  ]
}

  • Add an additional DCRA with the relevant Data Collection Rule. You must replace <dcraName>:

    {
 "type": "Microsoft.Resources/deployments",
 "name": "<dcraName>",
 "apiVersion": "2017-05-10",
 "subscriptionId": "[variables('clusterSubscriptionId')]",
 "resourceGroup": "[variables('clusterResourceGroup')]",
 "dependsOn": [
   "[resourceId('Microsoft.Insights/dataCollectionEndpoints/', variables('dceName'))]",
   "[resourceId('Microsoft.Insights/dataCollectionRules', variables('dcrName'))]"
 ],
 "properties": {
   "mode": "Incremental",
   "template": {
     "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
     "contentVersion": "1.0.0.0",
     "parameters": {},
     "variables": {},
     "resources": [
       {
         "type": "Microsoft.ContainerService/managedClusters/providers/dataCollectionRuleAssociations",
         "name": "[concat(variables('clusterName'),'/microsoft.insights/', variables('dcraName'))]",
         "apiVersion": "2021-09-01-preview",
         "location": "[parameters('clusterLocation')]",
         "properties": {
           "description": "Association of data collection rule. Deleting this association will break the data collection for this AKS Cluster.",
           "dataCollectionRuleId": "[resourceId('Microsoft.Insights/dataCollectionRules', variables('dcrName'))]"
         }
       }
     ]
   },
   "parameters": {}
 }
}

  • Add an additional Grafana integration:

    {
      "type": "Microsoft.Dashboard/grafana",
      "apiVersion": "2022-08-01",
      "name": "[split(parameters('grafanaResourceId'),'/')[8]]",
      "sku": {
        "name": "[parameters('grafanaSku')]"
      },
      "location": "[parameters('grafanaLocation')]",
      "properties": {
        "grafanaIntegrations": {
          "azureMonitorWorkspaceIntegrations": [
            // Existing azureMonitorWorkspaceIntegrations values (if any)
            // {
            //   "azureMonitorWorkspaceResourceId": "<value>"
            // },
            // {
            //   "azureMonitorWorkspaceResourceId": "<value>"
            // },
            {
              "azureMonitorWorkspaceResourceId": "[parameters('azureMonitorWorkspaceResourceId')]"
            },
            {
              "azureMonitorWorkspaceResourceId": "[parameters('azureMonitorWorkspaceResourceId2')]"
            }
          ]
        }
      }
    }
    • Assign Monitoring Data Reader role to read data from the new Azure Monitor Workspace:
    {
  "type": "Microsoft.Authorization/roleAssignments",
  "apiVersion": "2022-04-01",
  "name": "[parameters('roleNameGuid')]",
  "scope": "[parameters('azureMonitorWorkspaceResourceId2')]",
  "properties": {
      "roleDefinitionId": "[concat('/subscriptions/', variables('clusterSubscriptionId'), '/providers/Microsoft.Authorization/roleDefinitions/', 'b0d8363b-8ddd-447d-831f-62ca05bff136')]",
      "principalId": "[reference(resourceId('Microsoft.Dashboard/grafana', split(parameters('grafanaResourceId'),'/')[8]), '2022-08-01', 'Full').identity.principalId]"
  }
}

Send different metrics to different Azure Monitor workspaces

If you want to send some metrics to one Azure Monitor workspace and other metrics to a different one, follow the above steps to add additional DCRs. The value of microsoft_metrics_include_label under the labelIncludeFilter in the DCR is the identifier for the workspace. To then configure which metrics are routed to which workspace, you can add an extra pre-defined label, microsoft_metrics_account to the metrics. The value should be the same as the corresponding microsoft_metrics_include_label in the DCR for that workspace. To add the label to the metrics, you can utilize relabel_configs in your scrape config. To send all metrics from one job to a certain workspace, add the following relabel config:

relabel_configs:
- source_labels: [__address__]
  target_label: microsoft_metrics_account
  action: replace
  replacement: "MonitoringAccountLabel1"

The source label is __address__ because this label will always exist so this relabel config will always be applied. The target label will always be microsoft_metrics_account and its value should be replaced with the corresponding label value for the workspace.

Example

If you want to configure three different jobs to send the metrics to three different workspaces, then include the following in each data collection rule:

"labelIncludeFilter": {
  "microsoft_metrics_include_label": "MonitoringAccountLabel1"
}

"labelIncludeFilter": {
  "microsoft_metrics_include_label": "MonitoringAccountLabel2"
}

"labelIncludeFilter": {
  "microsoft_metrics_include_label": "MonitoringAccountLabel3"
}

Then in your scrape config, include the same label value for each:

scrape_configs:
- job_name: prometheus_ref_app_1
  kubernetes_sd_configs:
    - role: pod
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_app]
      action: keep
      regex: "prometheus-reference-app-1"
    - source_labels: [__address__]
      target_label: microsoft_metrics_account
      action: replace
      replacement: "MonitoringAccountLabel1"
- job_name: prometheus_ref_app_2
  kubernetes_sd_configs:
    - role: pod
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_app]
      action: keep
      regex: "prometheus-reference-app-2"
    - source_labels: [__address__]
      target_label: microsoft_metrics_account
      action: replace
      replacement: "MonitoringAccountLabel2"
- job_name: prometheus_ref_app_3
  kubernetes_sd_configs:
    - role: pod
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_app]
      action: keep
      regex: "prometheus-reference-app-3"
    - source_labels: [__address__]
      target_label: microsoft_metrics_account
      action: replace
      replacement: "MonitoringAccountLabel3"

Next steps