Azure Kubernetes Service (AKS) node image upgrade issue

Question

Hi,
Node image version not properly returned when executing/checking with az,kubectl, and Azure portal.

I’ve followed https://learn.microsoft.com/en-us/azure/aks/node-image-upgrade to upgrade node images in all node pools.

I have two node pools (systempool and usernodepool) as per MSFT recommendations https://learn.microsoft.com/en-us/azure/aks/use-system-pools?tabs=azure-cli

Executed az aks upgrade --resource-group foo --name bar --node-image-only

It immediately returned a JSON to the terminal, no errors.

As seen on the screenshot, for both node pools the available image version and the current image version is the same (current image version == latest image version).

However, if I run kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels.kubernetes\.azure\.com\/node-image-version}{"\n"}{end}' you can clearly see (last line on the screen shot), that kubectl returns a completely different image version for both nodes (the ones that I was upgrading from).

The Azure portal also reflects the old image versions:

I connected to both nodes to check if they are pending for reboot but /var/run/reboot-required was not present.

What am I missing ?

Environment :
{
"azure-cli": "2.37.0",
"azure-cli-core": "2.37.0",
"azure-cli-telemetry": "1.0.6",
"extensions": {
"aks-preview": "0.5.76"
}
}

Kubernetes version: 1.22.6

Answer 1

@AdamBudzinskiAZA-0329
I confirmed that the node image update should also update the existing nodes. I am reaching out to you privately so we can work with you directly to investigate this.