This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 37%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENK%3C/text%3E%3C/svg%3E)
Dear community,
Our prod AKS cluster is in failed state(running). Worker nodes is also in NotReady state.
Can you please help resolving this issue? We can not event stop it because it stuck in failed state. We tried running following commands but they failed after unning for hours
az resource update --ids /subscriptions/xxx/resourcegroups/xxx/providers/Microsoft.ContainerService/ManagedClusters/xxx
az aks update --resource-group xxx --name xxx
I see following error as well
Warning NetworkNotReady 2m34s (x14701 over 8h) kubelet network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Below are some output of kubectl commands
[nlweb-prod@AZSAPLPPUSA01 tmp]$ kubectl get no -o wide
E0819 13:10:27.880180 25142 memcache.go:255] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0819 13:10:27.896518 25142 memcache.go:106] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0819 13:10:27.900139 25142 memcache.go:106] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
E0819 13:10:27.904477 25142 memcache.go:106] couldn't get resource list for metrics.k8s.io/v1beta1: the server is currently unable to handle the request
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
aks-primarypool-19920242-vmss000006 NotReady agent 10h v1.25.11 10.82.46.4
If an answer has been helpful, please consider accepting the answer to help increase visibility of this question for other members of the Microsoft Q&A community. If not, please let us know what is still needed in the comments so the question can be answered. Thank you for helping to improve Microsoft Q&A!
@Prrudram-MSFT acctually it didn't help. I tried rebooting. I see following error
Warning NetworkNotReady 2m34s (x14701 over 8h) kubelet network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Thanks for getting back with the detailed explanation. This issue needs deeper investigation and Azure technical support team will be able to help with this. I would recommend you open an azure support case. If you don't have the ability to open a technical support ticket, please let me know and I can help you further with this.
You seem to be running 1 node which has failed.
I'd suggest the node just needs to be rebooted as the first step, you can do this in VMSS view in the Azure Portal, or more lazily by stopping and starting the AKS Cluster from the AKS resource in the portal. Hopefully this remedies the initial issue of the node not being ready. If it doesn't then raising a case with Azure support would be my suggestion.
I'd also suggest verifying your cluster implementation against current design best practices. https://learn.microsoft.com/en-us/azure/architecture/reference-architectures/containers/aks/baseline-aks
I tried rebooting and adding other nodes to nodepool. Even tried adding another system and user nodepool. The nodes are always in NotReady State. While describing nodes I see following error:
Warning NetworkNotReady 2m34s (x14701 over 8h) kubelet network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized