Troubleshoot outdated kernel versions in Azure Linux Container Host for Azure Kubernetes Service (AKS) node images

During migration or when adding new node pools to your Azure Linux Container Host, you might encounter issues with outdated kernel versions. Azure Kubernetes Service (AKS) releases a new Azure Linux node image every week, which is used for new node pools and as the starting image for scaling up. However, older node pools might not be updating their kernel versions as expected.

This article provides guidance on how to troubleshoot issues where Azure Linux Container Host node images in AKS are running outdated kernel versions, including identifying the symptoms, understanding the underlying causes, and applying recommended solutions to ensure that your nodes are running the latest kernel versions.

Verify the kernel version of your node pools

1. Check the kernel version of your node pools using the following command:

   kubectl get nodes -o wide
   

2. Compare the kernel version of your node pools with the latest kernel published on packages.microsoft.com.

Symptom

A common symptom of this issue is that the Azure Linux nodes aren't using the latest kernel version.

Causes

You might encounter outdated kernel versions on Azure Linux Container Host node images in AKS due to one or both of the following reasons:

• Automatic node-image upgrades weren't enabled when the node pool was created.
• The base image that AKS uses to start clusters runs two weeks behind the latest kernel versions due to their rollout procedure.

Solution

You can enable automatic upgrades using GitHub Actions and reboot the nodes.

Enable automatic AKS node image upgrades

Azure CLI

Enable automatic node image upgrades on a new AKS cluster using the az aks create command with the parameter --auto-upgrade-channel node-image. Replace the placeholder values with your own values.

az aks create --name <cluster-name> --resource-group <resource-group-name> --os-sku AzureLinux --auto-upgrade-channel node-image

ARM template

Enable automatic node image upgrades in an Azure Resource Manager (ARM) template by setting the upgradeChannel property in autoUpgradeProfile to node-image. For example:

    autoUpgradeProfile: {
      upgradeChannel: 'node-image'
    }

Reboot the nodes

When updating the kernel version, you need to reboot the node to use the new kernel version. We recommend that you set up the kured daemonset. You can use Kured to monitor your nodes for the /var/run/reboot-required file, drain the workload, and reboot the nodes.

Workaround: Manual upgrades

If you need a quick workaround, you can manually upgrade the node image on an AKS cluster using the az aks nodepool upgrade command. For example:

az aks nodepool upgrade \
    --resource-group testAzureLinuxResourceGroup \
    --cluster-name testAzureLinuxCluster \
    --name myAzureLinuxNodePool \
    --node-image-only

Next steps

If the preceding steps don't resolve the issue, open a support ticket.