This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 68%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ET%3C/text%3E%3C/svg%3E)
Team,
I was trying to upgrade my cluster to version 1.22.6 but after waiting for 1.5 hour the process threw an error.
Can anyone suggest where to check logs and resolve this issue. Currently, I'm unable to find out the real reason behind this issue. Please help
Regards,
Tanul
Did you click on the link suggested - "click here to go to diagnose and solve problems"? This will provide more information.
Hi Tanul,
Thank you for reaching out to the Microsoft Q&A platform. Happy to answer your question.
I understand you are trying to upgrade ASK cluster and the cluster shows in failed state after you triggered the upgrade. What is the current version of your cluster?
Did you run the diagnostics? running diagnostics will give clues on the cluster state failure.
Firstly, I would suggest fixing failed state. Please do the following:
Upgrade aks to version that is already there. If the current version is say 1.18.10 , you nee to run below
az aks upgrade --resource-group myResourceGroup --name myAKSCluster --kubernetes-version 1.18.10
which fixes the failed state of the cluster
After this you just need to run the upgrade to the correct next version (1.22.6), I would suggest checking for the available version for upgrade using below
az aks get-upgrades --resource-group myResourceGroup --name myAKSCluster --output table
Now the upgrade:
az aks upgrade --resource-group myResourceGroup --name myAKSCluster --kubernetes-version 1.22.6
You could also look at the events using kubectl get events to show events in the default namespaces while running an upgrade. For example
az aks get-upgrades --resource-group myResourceGroup --name myAKSCluster --output table
Reference : https://learn.microsoft.com/en-us/azure/aks/upgrade-cluster
Hope this helps!
--please don't forget to upvote and accept as answer if the reply is helpful--
@Prrudram-MSFT , So the problem is that my AKS version was on 1.21.9 and now it is on 1.22.6. In total, it is showing 8 nodes out of which 7 nodes are on 1.22.6 and only 1 node is on 1.21.9 with the status as Ready,SchedulingDisabled
As per diagnostics the issue is with ingress-nginx pod. Our ingress-nginx is deployed with Replica count of 2 and PDB as 1. Now one pod is running perfectly fine on the node with status scheduled disabled( which is on K8s version 1.21.9).. But the other pod which is trying to spin up on any other node(with K8s version 1.22.6) is failing with these errors:
Exec lifecycle hook ([/wait-shutdown]) for Container "controller" in Pod "ingress-nginx-devtest-controller-6f6cd95b7-jn96c_ingress-nginx(2d3fd173-e845-4278-a178-da0a1635d145)" failed - error: command '/wait-shutdown' exited with 137: , message: ""
Warning Unhealthy 4m47s (x9 over 6m7s) kubelet Readiness probe failed: HTTP probe failed with statuscode: 500
Warning Unhealthy 77s (x22 over 6m7s) kubelet Liveness probe failed: HTTP probe failed with statuscode: 500
Container Logs:-
Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
Due to only 1 replica in running state, PDB caused the deadlock and forced cluster into failed state. Could you please please help me.. And suggest what exactly I have to do
@Prrudram-MSFT , The problem is that my current version on Azure portal and in all the nodes except one with schedule disabled has 1.22.6 version upgraded. Now if I run any of the command with older version then it wont effect because AKS downgrade is not possible. Now my node pool provisioning state is showing as Failed
@Prrudram-MSFT , Issue is resolved.. The problem is that NGINX version(1.19.9) we were using was compatible with maximum 1.21.9 version of kubernetes. The moment we have upgraded K8s to 1.22.6 it started failing. Nginx api's were failing while trying to interact K8s. We have upgraded our nignx to (1.19.10) and its working.
Now the only issue left is that AKS status and node pool provisioning status is showing as Failed. As my kubernetes and nodes are already on 1.22.6.. Should I run this command to bring it back to normal state.
az aks upgrade --resource-group myResourceGroup --name myAKSCluster --kubernetes-version 1.22.6
Please suggest.
Kind Regards
Tanul
Its auto recovered. After resolving the nginx issue, we have manually removed the extra node from our cluster. Now, it is showing in stable state. After 5 hours, resource health event raised and returned the Available status due to which AKS and node pool state is automatically changed to healthy
