AKS cluster upgrade from 1.23.x to 1.27.x

Question

Hi All.

I have production AKS running on 1.23.12, and we plan to update to the latest version. In December23, we upgraded the pre-production AKS cluster from 1.24.x to 1.26.x using the following method.

1. Upgraded control plane only from 1.24.x to 1.25.x and later to 1.26.x
2. Create new node pool(ubuntu 22) using the 1.26.6 cluster version.
3. Condron and Drain the old node pool(ubuntu 18).
4. All Kubernetes objects have been migrated from the old node pool (Ubuntu 18) to the new node pool (Ubuntu 22). We had an application issue in the node pool (ubuntu 22) due to a Cgroup version issue, thus we migrated Kubernetes applicaiton from the new node pool (ubuntu 22) to the old node pool (ubuntu 18). Cgroupv2 was disabled in the new node pool (ubuntu 22), and the application was migrated to the new node pool (ubuntu 22), where it is now running properly.

The customer expects to perform a similar update in the production cluster, but the upgrade control plane only option is now disabled in the production AKS cluster, with the message "One or more node pools in the cluster are using a Kubernetes version that is 3 or more minor versions apart from the selected version. Upgrade control plane only is disabled."

What is the best approach for upgrading the ASK cluster from 1.23.x to 1.27.x, including a rollback option?

Thanks
Veera.

Answer 1

Hello Veera

Welcome to Microsoft Q&A Platform, thanks for posting your query here.

Current production AKS is 1.23.12, and the minimum upgradeable version is 1.26, right? If that's true, I think it's no longer possible to upgrade the control plane only.
In this case, it is recommend recreating a new cluster and migrating all workloads. You cannot upgrade only ccp, and even if you upgrade both ccp+worker nodes of this AKS, too many versions will be upgraded at once, which is risky.
Refer to the document: https://learn.microsoft.com/en-us/azure/aks/supported-kubernetes-versions?tabs=azure-cli#can-i-skip-multiple-aks-versions-during-cluster-upgrade

Since your workload is a production cluster, it would be a big issue if the upgrade suddenly fails and gets service impact. Also, even if the upgrade goes well, it might be a problem if application pods compatibility issue occurs on the cgroupv2 nodes. 

Therefore, it is a safe way to create a new cluster first and then deploy existing workloads to the new cluster and see if they are working properly.

Hope this helps.