Enable Container insights for Azure Kubernetes Service (AKS) cluster

This article describes how to set up Container insights to monitor a managed Kubernetes cluster hosted on an Azure Kubernetes Service (AKS) cluster.

Prerequisites

If you're connecting an existing AKS cluster to a Log Analytics workspace in another subscription, the Microsoft.ContainerService resource provider must be registered in the subscription with the Log Analytics workspace. For more information, see Register resource provider.

New AKS cluster

You can enable monitoring for an AKS cluster when it's created by using any of the following methods:

Existing AKS cluster

Use any of the following methods to enable monitoring for an existing AKS cluster.

Note

Azure CLI version 2.39.0 or higher is required for managed identity authentication.

Use a default Log Analytics workspace

Use the following command to enable monitoring of your AKS cluster by using a default Log Analytics workspace for the resource group. If a default workspace doesn't already exist in the cluster's region, one will be created with a name in the format DefaultWorkspace-<GUID>-<Region>.

az aks enable-addons -a monitoring -n <cluster-name> -g <cluster-resource-group-name>

The output will resemble the following example:

provisioningState       : Succeeded

Specify a Log Analytics workspace

Use the following command to enable monitoring of your AKS cluster on a specific Log Analytics workspace. The resource ID of the workspace will be in the form "/subscriptions/<SubscriptionId>/resourceGroups/<ResourceGroupName>/providers/Microsoft.OperationalInsights/workspaces/<WorkspaceName>".

az aks enable-addons -a monitoring -n <cluster-name> -g <cluster-resource-group-name> --workspace-resource-id <workspace-resource-id>

The output will resemble the following example:

provisioningState       : Succeeded

Verify agent and solution deployment

Run the following command to verify that the agent is deployed successfully.

kubectl get ds ama-logs --namespace=kube-system

The output should resemble the following example, which indicates that it was deployed properly:

User@aksuser:~$ kubectl get ds ama-logs --namespace=kube-system
NAME       DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
ama-logs   2         2         2         2            2           beta.kubernetes.io/os=linux   1d

If there are Windows Server nodes on the cluster, run the following command to verify that the agent is deployed successfully:

kubectl get ds ama-logs-windows --namespace=kube-system

The output should resemble the following example, which indicates that it was deployed properly:

User@aksuser:~$ kubectl get ds ama-logs-windows --namespace=kube-system
NAME                   DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR                   AGE
ama-logs-windows           2         2         2         2            2           beta.kubernetes.io/os=windows   1d

To verify deployment of the solution, run the following command:

kubectl get deployment ama-logs-rs -n=kube-system

The output should resemble the following example, which indicates that it was deployed properly:

User@aksuser:~$ kubectl get deployment ama-logs-rs -n=kube-system
NAME       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE    AGE
ama-logs-rs   1         1         1            1            3h

View configuration with CLI

Use the aks show command to find out whether the solution is enabled or not, what the Log Analytics workspace resource ID is, and summary information about the cluster.

az aks show -g <resourceGroupofAKSCluster> -n <nameofAksCluster>

After a few minutes, the command completes and returns JSON-formatted information about the solution. The results of the command should show the monitoring add-on profile and resemble the following example output:

"addonProfiles": {
    "omsagent": {
      "config": {
        "logAnalyticsWorkspaceResourceID": "/subscriptions/<WorkspaceSubscription>/resourceGroups/<DefaultWorkspaceRG>/providers/Microsoft.OperationalInsights/workspaces/<defaultWorkspaceName>"
      },
      "enabled": true
    }
  }

Migrate to managed identity authentication

This section explains two methods for migrating to managed identity authentication.

Existing clusters with a service principal

AKS clusters with a service principal must first disable monitoring and then upgrade to managed identity. Only Azure public cloud, Azure China cloud, and Azure Government cloud are currently supported for this migration.

  1. Get the configured Log Analytics workspace resource ID:

    az aks show -g <resource-group-name> -n <cluster-name> | grep -i "logAnalyticsWorkspaceResourceID"
    
  2. Disable monitoring with the following command:

    az aks disable-addons -a monitoring -g <resource-group-name> -n <cluster-name> 
    
  3. Upgrade cluster to system managed identity with the following command:

    az aks update -g <resource-group-name> -n <cluster-name> --enable-managed-identity
    
  4. Enable the monitoring add-on with the managed identity authentication option by using the Log Analytics workspace resource ID obtained in step 1:

    az aks enable-addons -a monitoring --enable-msi-auth-for-monitoring -g <resource-group-name> -n <cluster-name> --workspace-resource-id <workspace-resource-id>
    

Existing clusters with system or user-assigned identity

AKS clusters with system-assigned identity must first disable monitoring and then upgrade to managed identity. Only Azure public cloud, Azure China cloud, and Azure Government cloud are currently supported for clusters with system identity. For clusters with user-assigned identity, only Azure public cloud is supported.

  1. Get the configured Log Analytics workspace resource ID:

    az aks show -g <resource-group-name> -n <cluster-name> | grep -i "logAnalyticsWorkspaceResourceID"
    
  2. Disable monitoring with the following command:

    az aks disable-addons -a monitoring -g <resource-group-name> -n <cluster-name>
    
  3. Enable the monitoring add-on with the managed identity authentication option by using the Log Analytics workspace resource ID obtained in step 1:

    az aks enable-addons -a monitoring --enable-msi-auth-for-monitoring -g <resource-group-name> -n <cluster-name> --workspace-resource-id <workspace-resource-id>
    

Use one of the following procedures to enable network isolation by connecting your cluster to the Log Analytics workspace by using Azure Private Link.

Managed identity authentication

Use the following procedure if your cluster is using managed identity authentication with Azure Monitor Agent.

  1. Follow the steps in Enable network isolation for the Azure Monitor agent to create a data collection endpoint and add it to your Azure Monitor private link service.

  2. Create an association between the cluster and the data collection endpoint by using the following API call. For information on this call, see Data collection rule associations - Create. The DCR association name must be configurationAccessEndpoint, and resourceUri is the resource ID of the AKS cluster.

    PUT https://management.azure.com/{cluster-resource-id}/providers/Microsoft.Insights/dataCollectionRuleAssociations/configurationAccessEndpoint?api-version=2021-04-01
    {
    "properties": {
        "dataCollectionEndpointId": "{data-collection-endpoint-resource-id}"
        }
    }
    

    The following snippet is an example of this API call:

    PUT https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.ContainerService/managedClusters/my-aks-cluster/providers/Microsoft.Insights/dataCollectionRuleAssociations/configurationAccessEndpoint?api-version=2021-04-01
    
    {
    "properties": {
        "dataCollectionEndpointId": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/myDataCollectionEndpoint"
        }
    }
    
  3. Enable monitoring with the managed identity authentication option by using the steps in Migrate to managed identity authentication.

Without managed identity authentication

Use the following procedure if you're not using managed identity authentication. This requires a private AKS cluster.

  1. Create a private AKS cluster following the guidance in Create a private Azure Kubernetes Service cluster.

  2. Disable public Ingestion on your Log Analytics workspace.

    Use the following command to disable public ingestion on an existing workspace.

    az monitor log-analytics workspace update --resource-group <azureLogAnalyticsWorkspaceResourceGroup> --workspace-name <azureLogAnalyticsWorkspaceName>  --ingestion-access Disabled
    

    Use the following command to create a new workspace with public ingestion disabled.

    az monitor log-analytics workspace create --resource-group <azureLogAnalyticsWorkspaceResourceGroup> --workspace-name <azureLogAnalyticsWorkspaceName>  --ingestion-access Disabled
    
  3. Configure private link by following the instructions at Configure your private link. Set ingestion access to public and then set to private after the private endpoint is created but before monitoring is enabled. The private link resource region must be same as AKS cluster region.

  4. Enable monitoring for the AKS cluster.

    az aks enable-addons -a monitoring --resource-group <AKSClusterResourceGorup> --name <AKSClusterName> --workspace-resource-id <workspace-resource-id>
    

Limitations

  • Enabling managed identity authentication (preview) isn't currently supported by using Terraform or Azure Policy.
  • When you enable managed identity authentication (preview), a data collection rule is created with the name MSCI-<cluster-region>-<\cluster-name>. Currently, this name can't be modified.

Next steps

  • If you experience issues while you attempt to onboard the solution, review the Troubleshooting guide.
  • With monitoring enabled to collect health and resource utilization of your AKS cluster and workloads running on them, learn how to use Container insights.