Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Applies to: AKS on Azure Local
This article describes how to enable and query control plane metrics from AKS on Azure Local. The workflow is as follows:
- Enable Managed Prometheus extension
- Enable control plane metrics
- (Optional) view metrics in Grafana
Control plane metrics provide critical visibility into the availability and performance of Kubernetes control plane components, such as the API server, scheduler, or controller manager. You can use these metrics to maximize observability and maintain operational excellence for your cluster.
Prerequisites
Before you begin, make sure the following prerequisites are met:
- A running AKS on Azure Local instance.
- Install the latest version of the aksarc, connectedk8s, and k8s-extension CLI extensions.
- Download and install kubectl on your development machine.
- Understand the basics of Prometheus Query Language.
- Understand the basics of Kubernetes system component metrics.
Enable Managed Prometheus extension
Azure Monitor collects and aggregates important metrics from your AKS Arc running on Azure Local instance. In addition to platform metrics collected from your cluster, you can view granular Kubernetes metrics using the managed Prometheus extension. This extension collects Prometheus metrics from your deployment and stores them in an Azure Monitor workspace in Azure. Once they are ingested, you can analyze them in Metrics Explorer or use prebuilt dashboards in Azure Managed Grafana.
Step 1: Install the managed Prometheus extension
You can install the extension either from the Azure portal, or using CLI.
Go to your Kubernetes instance, then select Monitoring > Insights > Monitor Settings:
See the guidance for the managed Prometheus extension onboarding.
Step 2: Verify extension and metrics pod deployment
To verify the extension installation, you can run az connectedk8s proxy to connect to the cluster and use kubectl to list the metrics pods. The pods should start with the name ama-metrics- and are in a running state.
kubectl get pods -n kube-system
The output of the command is similar to the following:
NAME READY STATUS RESTARTS AGE
akshci-telemetry-5df56fd5-s5wtm 1/1 Running 1 (37h ago) 44d
ama-logs-nqf9h 3/3 Running 0 5h29m
ama-logs-pvvb2 3/3 Running 2 (5h21m ago) 5h29m
ama-logs-rs-86bc9dd898-4p7pv 2/2 Running 0 5h29m
**ama-metrics-98bb54876-dndrh** 2/2 Running 2 (3h33m ago) 5h30m
**ama-metrics-ksm-6544c98f5f-ph6sp** 1/1 Running 0 5h30m
**ama-metrics-node-6dl7p** 2/2 Running 2 (3h33m ago) 5h30m
**ama-metrics-node-ztwzt** 2/2 Running 1 (3h33m ago) 5h30m
….
Enable control plane metrics with custom configuration
After you enable the extension, you can view Prometheus Metrics from targets scraped by default in the Azure Monitor workspace. The default ON targets include kubelet, kube-state-metrics, node-exporter, etc. To get started with kubelet metrics, use the PromQL below:
kubelet_running_pods{cluster="<cluster_name>", instance="<instance_name>", job="kubelet"}
To view control plane metrics such as APIServer and ETCD, you can customize the scraping of Prometheus metrics by applying the config maps to your cluster. The metrics pods pick up the config maps and pods restart in 2-3 minutes. Follow these steps to enable.
Step 1: connect to Kubernetes
Connect to your cluster using az connectedk8s proxy and run kubectl get pods -A to make sure you're connected.
Step 2: download the configuration files and review the content
Managed Prometheus uses an agent-based solution to collect Prometheus metrics and send them to the Azure Monitor workspace. There are two configuration files to download and review: ama-metrics-settings-configmap.yaml and ama-metrics-prometheus-config-configmap.yaml. For more information about customizing metrics scraping using configuration files, see Customize scraping of Prometheus metrics.
- To enable APIServer metrics, modify the value of
apiserverunder the default OFF targets and set it to true in ama-metrics-settings-configmap.yaml. For the list of metrics, see minimal ingestion for default OFF targets. - To enable metrics from components not listed under default OFF targets such as ETCD, Controller Manager, Kube Scheduler, add a new scraping job in ama-metrics-prometheus-config-configmap.yaml.
You can download these two configuration files to your local machine and review the content before going to the next step.
Step 3: apply custom configuration files
Run the following commands to apply the changes, then wait several minutes for the metrics pods to restart.
kubectl apply -f ama-metrics-settings-configmap.yaml
kubectl apply -f ama-metrics-prometheus-config-configmap.yaml
Step 4: query metrics in Azure Monitor workspace
Go to the linked Azure Monitor Workspace > Metric Explorer and use PromQL to validate that the metrics are ingested. In the following sample query, it shows a stable kube-scheduler metrics scheduler_schedule_attempts_total from a specific Kubernetes cluster.
View metrics in Grafana
Metrics Explorer is convenient for metrics validation. To operationalize Kubernetes monitoring with Azure Monitor, it's recommended that you monitor the metrics using Azure Managed Grafana.
Step 1: install Azure Managed Grafana
Follow these instructions to create a Grafana workspace, link to an Azure Monitor workspace, and view the metrics in Grafana dashboards. You can view the dashboard under Monitoring > Insights > Monitor Settings. Multiple instances can be linked to the same Azure Monitor workspace, so make sure to choose the right dashboard.
Step 2: import a prebuilt dashboard for control plane metrics
Download the API server dashboard to your local machine, copy the JSON content, then import it to the managed Grafana dashboard.
Step 3: view metrics in the dashboard
Ensure that the data source and cluster names are correct. You can view the metrics in Grafana and customize them as needed.
