Upgrades to Kubernetes 1.16 fail when node labels have a kubernetes.io prefix

This article discusses how to troubleshoot a failed upgrade to Kubernetes 1.16 when you're using node labels that have a kubernetes.io prefix.

Symptoms

Your attempts to upgrade to Kubernetes 1.16 have failed, and you're using labels that have a kubernetes.io prefix.

Cause

As of Kubernetes 1.16, the kubelet can only apply a defined subset of labels with the kubernets.io prefix to nodes. Microsoft Azure Kubernetes Service (AKS) can't remove active labels on your behalf without your consent, as it might cause downtime to impacted workloads.

Solution

To mitigate this issue:

1. Upgrade your cluster control plane to 1.16 or later.
2. Add a new node pool on 1.16 or higher without the unsupported kubernetes.io labels.
3. Delete the older node pool.

AKS is investigating the capability to mutate active labels on a node pool to improve this mitigation.

Third-party information disclaimer

The third-party products that this article discusses are manufactured by companies that are independent of Microsoft. Microsoft makes no warranty, implied or otherwise, about the performance or reliability of these products.

Contact us for help

If you have questions or need help, create a support request, or ask Azure community support. You can also submit product feedback to Azure feedback community.