Edit

Collect Prometheus metrics with Container insights

  • Article
  • 8 minutes to read

Prometheus is a popular open-source metric monitoring solution and is the most common monitoring tool used to monitor Kubernetes clusters. Container insights uses its containerized agent to collect much of the same data that is typically collected from the cluster by Prometheus without requiring a Prometheus server. This data is presented in Container insights views and available to other Azure Monitor features such as log queries and log alerts.

Container insights can also scrape Prometheus metrics from your cluster and send the data to either Azure Monitor Logs or to Azure Monitor managed service for Prometheus. This requires exposing the Prometheus metrics endpoint through your exporters or pods and then configuring one of the addons for the Azure Monitor agent used by Container insights as shown the following diagram.

Diagram of container monitoring architecture sending Prometheus metrics to Azure Monitor Logs.

Send data to Azure Monitor managed service for Prometheus

Azure Monitor managed service for Prometheus is a fully managed Prometheus-compatible service that supports industry standard features such as PromQL, Grafana dashboards, and Prometheus alerts. This requires configuring the metrics addon for the Azure Monitor agent, which sends data to Prometheus.

Note

The metrics addon used to collect Prometheus metrics for Managed Prometheus currently only supports AKS clusters and cannot be used as an Arc enabled Kubernetes extension. To collect Prometheus metrics from Kubernetes clusters that are running self-managed Prometheus we recommend looking at the remote write capabilities of Managed Prometheus.

Tip

You don't need to enable Container insights to configure your AKS cluster to send data to managed Prometheus. See Collect Prometheus metrics from AKS cluster (preview) for details on how to configure your cluster without enabling Container insights.

Use the following procedure to add Promtheus collection to your cluster that's already using Container insights.

  1. Open the Kubernetes services menu in the Azure portal and select your AKS cluster.

  2. Click Insights.

  3. Click Monitor settings.

    Screenshot of button for monitor settings for an AKS cluster.

  4. Click the checkbox for Enable Prometheus metrics and select your Azure Monitor workspace.

  5. To send the collected metrics to Grafana, select a Grafana workspace. See Create an Azure Managed Grafana instance for details on creating a Grafana workspace.

    Screenshot of monitor settings for an AKS cluster.

  6. Click Configure to complete the configuration.

See Collect Prometheus metrics from AKS cluster (preview) for details on verifying your deployment and limitations

Send metrics to Azure Monitor Logs

You may want to collect additional data in addition to the predefined set of data collected by Container insights. This data isn't used by Container insights views but is available for log queries and alerts like the other data it collects. This requires configuring the monitoring addon for the Azure Monitor agent, which is the one currently used by Container insights to send data to a Log Analytics workspace.

Prometheus scraping settings

Active scraping of metrics from Prometheus is performed from one of two perspectives:

  • Cluster-wide: Defined in the ConfigMap section [Prometheus data_collection_settings.cluster].
  • Node-wide: Defined in the ConfigMap section [Prometheus_data_collection_settings.node].
Endpoint Scope Example
Pod annotation Cluster-wide prometheus.io/scrape: "true"
prometheus.io/path: "/mymetrics"
prometheus.io/port: "8000"
prometheus.io/scheme: "http"
Kubernetes service Cluster-wide http://my-service-dns.my-namespace:9100/metrics
http://metrics-server.kube-system.svc.cluster.local/metrics​
URL/endpoint Per-node and/or cluster-wide http://myurl:9101/metrics

When a URL is specified, Container insights only scrapes the endpoint. When Kubernetes service is specified, the service name is resolved with the cluster DNS server to get the IP address. Then the resolved service is scraped.

Scope Key Data type Value Description
Cluster-wide Specify any one of the following three methods to scrape endpoints for metrics.
urls String Comma-separated array HTTP endpoint (either IP address or valid URL path specified). For example: urls=[$NODE_IP/metrics]. ($NODE_IP is a specific Container insights parameter and can be used instead of a node IP address. Must be all uppercase.)
kubernetes_services String Comma-separated array An array of Kubernetes services to scrape metrics from kube-state-metrics. Fully qualified domain names must be used here. For example,kubernetes_services = ["http://metrics-server.kube-system.svc.cluster.local/metrics",http://my-service-dns.my-namespace.svc.cluster.local:9100/metrics]
monitor_kubernetes_pods Boolean true or false When set to true in the cluster-wide settings, the Container insights agent will scrape Kubernetes pods across the entire cluster for the following Prometheus annotations:
prometheus.io/scrape:
prometheus.io/scheme:
prometheus.io/path:
prometheus.io/port:
prometheus.io/scrape Boolean true or false Enables scraping of the pod, and monitor_kubernetes_pods must be set to true.
prometheus.io/scheme String http Defaults to scraping over HTTP.
prometheus.io/path String Comma-separated array The HTTP resource path from which to fetch metrics. If the metrics path isn't /metrics, define it with this annotation.
prometheus.io/port String 9102 Specify a port to scrape from. If the port isn't set, it will default to 9102.
monitor_kubernetes_pods_namespaces String Comma-separated array An allowlist of namespaces to scrape metrics from Kubernetes pods.
For example, monitor_kubernetes_pods_namespaces = ["default1", "default2", "default3"]
Node-wide urls String Comma-separated array HTTP endpoint (either IP address or valid URL path specified). For example: urls=[$NODE_IP/metrics]. ($NODE_IP is a specific Container insights parameter and can be used instead of a node IP address. Must be all uppercase.)
Node-wide or cluster-wide interval String 60s The collection interval default is one minute (60 seconds). You can modify the collection for either the [prometheus_data_collection_settings.node] and/or [prometheus_data_collection_settings.cluster] to time units such as s, m, and h.
Node-wide or cluster-wide fieldpass
fielddrop		 String Comma-separated array You can specify certain metrics to be collected or not from the endpoint by setting the allow (fieldpass) and disallow (fielddrop) listing. You must set the allowlist first.

Configure ConfigMaps

Perform the following steps to configure your ConfigMap configuration file for your cluster. ConfigMaps is a global list and there can be only one ConfigMap applied to the agent. You can't have another ConfigMaps overruling the collections.

  1. Download the template ConfigMap YAML file and save it as container-azm-ms-agentconfig.yaml. If you've already deployed a ConfigMap to your cluster and you want to update it with a newer configuration, you can edit the ConfigMap file you've previously used.

  2. Edit the ConfigMap YAML file with your customizations to scrape Prometheus metrics.

    To collect Kubernetes services cluster-wide, configure the ConfigMap file by using the following example:

    prometheus-data-collection-settings: |- ​
# Custom Prometheus metrics data collection settings
[prometheus_data_collection_settings.cluster] ​
interval = "1m"  ## Valid time units are s, m, h.
fieldpass = ["metric_to_pass1", "metric_to_pass12"] ## specify metrics to pass through ​
fielddrop = ["metric_to_drop"] ## specify metrics to drop from collecting
kubernetes_services = ["http://my-service-dns.my-namespace:9102/metrics"]

  3. Run the following kubectl command: kubectl apply -f <configmap_yaml_file.yaml>.

    Example: kubectl apply -f container-azm-ms-agentconfig.yaml.

The configuration change can take a few minutes to finish before taking effect. All ama-logs pods in the cluster will restart. When the restarts are finished, a message appears that's similar to the following and includes the result configmap "container-azm-ms-agentconfig" created.

Verify configuration

To verify the configuration was successfully applied to a cluster, use the following command to review the logs from an agent pod: kubectl logs ama-logs-fdf58 -n=kube-system.

If there are configuration errors from the Azure Monitor Agent pods, the output will show errors similar to the following example:

***************Start Config Processing******************** 
config::unsupported/missing config schema version - 'v21' , using defaults

Errors related to applying configuration changes are also available for review. The following options are available to perform additional troubleshooting of configuration changes and scraping of Prometheus metrics:

  • From an agent pod logs using the same kubectl logs command.

  • From Live Data (preview). Live Data (preview) logs show errors similar to the following example:

    2019-07-08T18:55:00Z E! [inputs.prometheus]: Error in plugin: error making HTTP request to http://invalidurl:1010/metrics: Get http://invalidurl:1010/metrics: dial tcp: lookup invalidurl on 10.0.0.10:53: no such host

  • From the KubeMonAgentEvents table in your Log Analytics workspace. Data is sent every hour with Warning severity for scrape errors and Error severity for configuration errors. If there are no errors, the entry in the table will have data with severity Info, which reports no errors. The Tags property contains more information about the pod and container ID on which the error occurred and also the first occurrence, last occurrence, and count in the last hour.

  • For Azure Red Hat OpenShift v3.x and v4.x, check the Azure Monitor Agent logs by searching the ContainerLog table to verify if log collection of openshift-azure-logging is enabled.

Errors prevent Azure Monitor Agent from parsing the file, causing it to restart and use the default configuration. After you correct the errors in ConfigMap on clusters other than Azure Red Hat OpenShift v3.x, save the YAML file and apply the updated ConfigMaps by running the command kubectl apply -f <configmap_yaml_file.yaml.

For Azure Red Hat OpenShift v3.x, edit and save the updated ConfigMaps by running the command oc edit configmaps container-azm-ms-agentconfig -n openshift-azure-logging.

Query Prometheus metrics data

To view Prometheus metrics scraped by Azure Monitor and any configuration/scraping errors reported by the agent, review Query Prometheus metrics data.

View Prometheus metrics in Grafana

Container insights supports viewing metrics stored in your Log Analytics workspace in Grafana dashboards. We've provided a template that you can download from Grafana's dashboard repository. Use the template to get started and reference it to help you learn how to query other data from your monitored clusters to visualize in custom Grafana dashboards.

Next steps