Monitoring and logging data in AKS hybrid

Applies to: AKS on Azure Stack HCI, AKS on Windows Server

This article describes how to monitor your Azure Kubernetes Service (AKS) deployment and collect logging data in AKS hybrid. You'll learn how to set up and access on-premises monitoring using Prometheus and Graphana, and how to collect and view logs using Elasticsearch, Fluent Bit, and Kibana (EFK).

Two types of monitoring and logging solutions are available, as described in the following table.

Solution Azure connectivity Support and service Cost Deployment
Azure Monitor Requires connecting the AKS cluster to Azure using Azure Arc for Kubernetes. Full support and servicing from Microsoft. Requires signing up for the Azure Monitor service. Use Azure Arc for monitoring clusters.
On-premises monitoring and logging Doesn't require Azure connectivity. Supported as open-source software by Microsoft (with no support agreement or SLAs), the community, and/or external vendors. Vendor-dependent. Customer-driven. See Monitor clusters using on-premises monitoring.

To use Azure Monitor with AKS clusters, see the Azure Monitor overview.

Use on-premises monitoring

It's crucial that you monitor the health, performance, and resource usage of the control plane nodes and workloads on your cluster when running apps in production. The recommended monitoring solution includes the following two tools:

  • Prometheus is a monitoring and alerting toolkit you can use for monitoring containerized workloads. Prometheus works with different types of collectors and agents to collect metrics and store them in a database where you can query the data and view reports. AKS hybrid makes it easy to deploy Prometheus, which is described later in this topic.

  • Grafana is a tool used to view, query, and visualize metrics on the Grafana dashboards. You can also configure Grafana to use Prometheus as the data source. You must have your own licensed copy of Grafana with AKS hybrid.

Monitoring solution overview

As part of Prometheus solution in AKS hybrid, the following components are deployed and automatically configured:

The deployment is based on the publicly available Kube-Prometheus-stack helm chart, which is extended to support the Windows exporter and secures metrics scraping between Prometheus and agents. Once the Prometheus solution is deployed, the Node exporter runs on each Linux node, and the Windows exporter runs on each Windows node.


Since the Prometheus operator, Prometheus, and Kube state metrics components are only supported on Linux, you must provision at least one Linux node in your AKS cluster to deploy this solution.

The objects and endpoints that the Prometheus solution scrapes include the following items:

  • Kube state metrics to collect various metrics provided by Kubernetes
  • Kubernetes API server
  • Kubelet
  • Node exporter to collect metrics for Linux nodes
  • Windows exporter to collect metrics for Windows nodes

To view the Grafana dashboards available in AKS hybrid, see Grafana dashboards available in AKS hybrid.

Deploy monitoring solution using PowerShell

This section describes the two options you can use to deploy monitoring on a workload cluster.

Option 1: Deploy the monitoring solution when creating the workload cluster

To enable monitoring, provide the -enableMonitoring parameter when you use New-AksHciCluster to create the workload cluster, as shown in the following example:

New-AksHciCluster -name mynewcluster -enableMonitoring

Monitoring is installed with the following default configuration:

  • The size of the persistent volume that's provisioned to store metrics (storageSizeGB) is 100 GB.
  • The retention time for collected metrics (retentionTimeHours) is 240 hours (or 10 days).

Option 2: Deploy the monitoring solution on an existing workload cluster

Run the Install-AksHciMonitoring command to deploy the monitoring solution on an existing workload cluster, as shown below:

Install-AksHciMonitoring -Name mycluster -storageSizeGB 100 -retentionTimeHours 240

The -storageSizeGB parameter sets the size of the persistent volume that's provisioned to store metrics, and the -retentionTimeHours parameter sets the amount of time the collected metrics are retained.

The monitoring solution is installed in a separate namespace called monitoring and uses a StorageClass called monitoring-sc. Prometheus is exposed on an internal endpoint that is accessible only within the cluster at http://akshci-monitoring-prometheus-svc.monitoring:9090.

Uninstall monitoring solution using PowerShell

Run the Uninstall-AksHciMonitoring PowerShell command to uninstall the AKS hybrid monitoring solution, as shown below:

Uninstall-AksHciMonitoring -Name <target cluster name>

The uninstall process removes everything, including the namespace, the StorageClass, and the actual data and metrics of the persistent volume.

Deploy Grafana, and configure it to use Prometheus

You can follow any guidance for deploying Grafana that's publicly available. You can also view Microsoft's deployment guidance to use Grafana, which details how to deploy and configure Grafana to connect it to an AKS Prometheus instance. This GitHub page also describes how to add Grafana dashboards that Microsoft makes available for AKS hybrid.

On-premises logging

Logging is crucial for troubleshooting and diagnostics. The logging solution in AKS hybrid is based on Elasticsearch, Fluent Bit, and Kibana (EFK). These components are all deployed as containers: 

  • Fluent Bit is the log processor and forwarder that collects data and logs from different sources. It then formats, unifies, and stores them in Elasticsearch.
  • Elasticsearch is a distributed search and analytics engine capable of centrally storing the logs for fast searches and data analytics. 
  • Kibana provides interactive visualizations on a web dashboard. This tool lets you view and query logs stored in Elasticsearch, and then you can visualize them through graphs and dashboards.

To set up an on-premises logging solution, see the steps to set up logging to access Kibana. This article includes all the components required to collect, aggregate, and query container logs across the cluster. 

For advanced configuration steps, see Windows logging.

Next steps