Setup of Network Observability for Azure Kubernetes Service (AKS) - Azure managed Prometheus and Grafana

Article
10/30/2023

AKS Network Observability is used to collect the network traffic data of your AKS cluster. Network Observability enables a centralized platform for monitoring application and network health. Prometheus collects AKS Network Observability metrics, and Grafana visualizes them. Both Cilium and non-Cilium data plane are supported. In this article, learn how to enable the Network Observability add-on and use Azure managed Prometheus and Grafana to visualize the scraped metrics.

Important

AKS Network Observability is currently in PREVIEW. See the Supplemental Terms of Use for Microsoft Azure Previews for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

For more information about AKS Network Observability, see What is Azure Kubernetes Service (AKS) Network Observability?.

Prerequisites

An Azure account with an active subscription. Create an account for free.

Use the Bash environment in Azure Cloud Shell. For more information, see Quickstart for Bash in Azure Cloud Shell.
If you prefer to run CLI reference commands locally, install the Azure CLI. If you're running on Windows or macOS, consider running Azure CLI in a Docker container. For more information, see How to run the Azure CLI in a Docker container.
- If you're using a local installation, sign in to the Azure CLI by using the az login command. To finish the authentication process, follow the steps displayed in your terminal. For other sign-in options, see Sign in with the Azure CLI.
- When you're prompted, install the Azure CLI extension on first use. For more information about extensions, see Use extensions with the Azure CLI.
- Run az version to find the version and dependent libraries that are installed. To upgrade to the latest version, run az upgrade.

Minimum version of Azure CLI required for the steps in this article is 2.44.0. Run az --version to find the version. If you need to install or upgrade, see Install Azure CLI.

Install the `aks-preview` Azure CLI extension

Important

AKS preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. AKS previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use. For more information, see the following support articles:

# Install the aks-preview extension
az extension add --name aks-preview

# Update the extension to make sure you have the latest version installed
az extension update --name aks-preview

Register the `NetworkObservabilityPreview` feature flag

az feature register --namespace "Microsoft.ContainerService" --name "NetworkObservabilityPreview"

Use az feature show to check the registration status of the feature flag:

az feature show --namespace "Microsoft.ContainerService" --name "NetworkObservabilityPreview"

Wait for the feature to say Registered before preceding with the article.

{
  "id": "/subscriptions/23250d6d-28f0-41dd-9776-61fc80805b6e/providers/Microsoft.Features/providers/Microsoft.ContainerService/features/NetworkObservabilityPreview",
  "name": "Microsoft.ContainerService/NetworkObservabilityPreview",
  "properties": {
    "state": "Registering"
  },
  "type": "Microsoft.Features/providers/features"
}

When the feature is registered, refresh the registration of the Microsoft.ContainerService resource provider with az provider register:

az provider register -n Microsoft.ContainerService

Create a resource group

A resource group is a logical container into which Azure resources are deployed and managed. Create a resource group with az group create command. The following example creates a resource group named myResourceGroup in the eastus location:

az group create \
    --name myResourceGroup \
    --location eastus

Create AKS cluster

Create an AKS cluster with az aks create. The following example creates an AKS cluster named myAKSCluster in the myResourceGroup resource group:

Non-Cilium
Cilium

Non-Cilium clusters support the enablement of Network Observability on an existing cluster or during the creation of a new cluster.

Use az aks create in the following example to create an AKS cluster with Network Observability and non-Cilium.

New cluster

az aks create \
    --name myAKSCluster \
    --resource-group myResourceGroup \
    --location eastus \
    --generate-ssh-keys \
    --network-plugin azure \
    --network-plugin-mode overlay \
    --pod-cidr 192.168.0.0/16 \
    --enable-network-observability

Existing cluster

Use az aks update to enable Network Observability for an existing cluster.

az aks update \
    --resource-group myResourceGroup \
    --name myAKSCluster \
    --enable-network-observability

Use az aks create in the following example to create an AKS cluster with Network Observability and Cilium.

az aks create \
    --name myAKSCluster \
    --resource-group myResourceGroup \
    --generate-ssh-keys \
    --location eastus \
    --max-pods 250 \
    --network-plugin azure \
    --network-plugin-mode overlay \
    --network-dataplane cilium \
    --node-count 2 \
    --pod-cidr 192.168.0.0/16

Azure managed Prometheus and Grafana

Use the following example to install and enable Prometheus and Grafana for your AKS cluster.

Create Azure Monitor resource

az resource create \
    --resource-group myResourceGroup \
    --namespace microsoft.monitor \
    --resource-type accounts \
    --name myAzureMonitor \
    --location eastus \
    --properties '{}'

Create Grafana instance

Use az grafana create to create a Grafana instance. The name of the Grafana instance must be unique. Replace myGrafana with a unique name for your Grafana instance.

az grafana create \
    --name myGrafana \
    --resource-group myResourceGroup

Place the Grafana and Azure Monitor resource IDs in variables

Use az grafana show to place the Grafana resource ID in a variable. Use az resource show to place the Azure Monitor resource ID in a variable. Replace myGrafana with the name of your Grafana instance.

grafanaId=$(az grafana show \
                --name myGrafana \
                --resource-group myResourceGroup \
                --query id \
                --output tsv)

azuremonitorId=$(az resource show \
                    --resource-group myResourceGroup \
                    --name myAzureMonitor \
                    --resource-type "Microsoft.Monitor/accounts" \
                    --query id \
                    --output tsv)

Link Azure Monitor and Grafana to AKS cluster

Use az aks update to link the Azure Monitor and Grafana resources to your AKS cluster.

az aks update \
    --name myAKSCluster \
    --resource-group myResourceGroup \
    --enable-azure-monitor-metrics \
    --azure-monitor-workspace-resource-id $azuremonitorId \
    --grafana-resource-id $grafanaId

Get cluster credentials

az aks get-credentials --name myAKSCluster --resource-group myResourceGroup

Note

The following section requires deployments of Azure managed Prometheus and Grafana.

Use the following example to verify the Azure Monitor pods are running.

kubectl get po -owide -n kube-system | grep ama-

ama-metrics-5bc6c6d948-zkgc9          2/2     Running   0 (21h ago)   26h
ama-metrics-ksm-556d86b5dc-2ndkv      1/1     Running   0 (26h ago)   26h
ama-metrics-node-lbwcj                2/2     Running   0 (21h ago)   26h
ama-metrics-node-rzkzn                2/2     Running   0 (21h ago)   26h
ama-metrics-win-node-gqnkw            2/2     Running   0 (26h ago)   26h
ama-metrics-win-node-tkrm8            2/2     Running   0 (26h ago)   26h

Select Dashboards from the left navigation menu, open Kubernetes / Networking dashboard under Managed Prometheus folder.
Check if the Metrics in Kubernetes / Networking Grafana dashboard are visible. If metrics aren't shown, change time range to last 15 minutes in top right dropdown box.

Note

The following section requires deployments of Azure managed Prometheus and Grafana.

Warning

File should only be named as prometheus-config. Do not add any extensions like .yaml or .txt.

Use the following example to create a file named prometheus-config. Copy the code in the example into the file created.

global:
  scrape_interval: 30s
scrape_configs:
  - job_name: "cilium-pods"
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_container_name]
        action: keep
        regex: cilium-agent
      - source_labels:
          [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        separator: ":"
        regex: ([^:]+)(?::\d+)?
        target_label: __address__
        replacement: ${1}:${2}
        action: replace
      - source_labels: [__meta_kubernetes_pod_node_name]
        action: replace
        target_label: instance
      - source_labels: [__meta_kubernetes_pod_label_k8s_app]
        action: replace
        target_label: k8s_app
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        regex: (.*)
        target_label: pod
    metric_relabel_configs:
      - source_labels: [__name__]
        action: keep
        regex: (.*)

To create the configmap, use the following example:

kubectl create configmap ama-metrics-prometheus-config \
    --from-file=./prometheus-config \
    --namespace kube-system

Azure Monitor pods should restart themselves, if they don't, rollout restart with following command:
```
kubectl rollout restart deploy -n kube-system ama-metrics
```
Once the Azure Monitor pods have been deployed on the cluster, port forward to the ama pod to verify the pods are being scraped. Use the following example to port forward to the pod:
```
kubectl port-forward -n kube-system $(kubectl get po -n kube-system -l rsName=ama-metrics -oname | head -n 1) 9090:9090
```
Open http://localhost:9090 in your browser and navigate to Status > Targets, verify if cilium-pods are present and state says up.
Sign in to Azure Managed Grafana and import dashboard with the ID: 16611. Also, select Dashboards from the left navigation menu, open Kubernetes / Networking dashboard under Managed Prometheus folder. Metrics should be visible in both these dashboards.

Clean up resources

If you're not going to continue to use this application, delete the AKS cluster and the other resources created in this article with the following example:

  az group delete \
    --name myResourceGroup

Next steps

In this how-to article, you learned how to install and enable AKS Network Observability for your AKS cluster.

For more information about AKS Network Observability, see What is Azure Kubernetes Service (AKS) Network Observability?.
To create an AKS cluster with Network Observability and BYO Prometheus and Grafana, see Setup Network Observability for Azure Kubernetes Service (AKS) BYO Prometheus and Grafana.

Setup of Network Observability for Azure Kubernetes Service (AKS) - Azure managed Prometheus and Grafana

Prerequisites

Install the `aks-preview` Azure CLI extension

Register the `NetworkObservabilityPreview` feature flag

Create a resource group

Create AKS cluster

New cluster

Existing cluster

Azure managed Prometheus and Grafana

Create Azure Monitor resource

Create Grafana instance

Place the Grafana and Azure Monitor resource IDs in variables

Link Azure Monitor and Grafana to AKS cluster

Get cluster credentials

Enable visualization on Grafana

Clean up resources

Next steps

Feedback

Additional resources

Additional resources

Setup of Network Observability for Azure Kubernetes Service (AKS) - Azure managed Prometheus and Grafana

Prerequisites

Install the aks-preview Azure CLI extension

Register the NetworkObservabilityPreview feature flag

Create a resource group

Create AKS cluster

New cluster

Existing cluster

Azure managed Prometheus and Grafana

Create Azure Monitor resource

Create Grafana instance

Place the Grafana and Azure Monitor resource IDs in variables

Link Azure Monitor and Grafana to AKS cluster

Get cluster credentials

Enable visualization on Grafana

Clean up resources

Next steps

Feedback

Additional resources

Additional resources

Install the `aks-preview` Azure CLI extension

Register the `NetworkObservabilityPreview` feature flag