This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 57.00000000000001%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESG%3C/text%3E%3C/svg%3E)
Hi there!
I run a production AKS cluster at the Alan Turing Institute and our internal process of allocating credits to subscriptions often means that mean subs get disabled and re-enabled after some time.
Upon re-enabling the sub, I find the AKS cluster is in a super weird state. In the kube-system namespace, a lot of the pods are duplicated and stuck either in the Pending or Terminating state and this means that helm then can't find a ready tiller pod and I'm kinda stuck. So my questions are as follows:
I'd be hugely grateful for any support you can provide!
Pods output:
$ kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
azure-cni-networkmonitor-8wl2j 1/1 Running 0 214d
azure-cni-networkmonitor-b7wkg 1/1 Running 0 128d
azure-cni-networkmonitor-q8dzp 1/1 Running 0 183d
azure-cni-networkmonitor-tdsqd 1/1 Running 0 72d
azure-ip-masq-agent-859zn 1/1 Terminating 0 183d
azure-ip-masq-agent-c29hr 1/1 Terminating 0 214d
azure-ip-masq-agent-fbch4 1/1 Terminating 0 72d
azure-ip-masq-agent-k2zqj 1/1 Terminating 0 128d
azure-npm-g9ttc 1/1 Terminating 0 65d
azure-npm-jm5ld 1/1 Terminating 0 65d
azure-npm-l97zz 1/1 Terminating 0 65d
azure-npm-mxf4b 1/1 Terminating 0 65d
coredns-6c66fc4fcb-9w5pg 1/1 Running 0 62d
coredns-6c66fc4fcb-s4lbp 1/1 Terminating 0 55d
coredns-869cb84759-4fh5l 0/1 Pending 0 33d
coredns-869cb84759-zkpmz 0/1 Pending 0 33d
coredns-autoscaler-5b867494f-8ffr2 0/1 Pending 0 33d
coredns-autoscaler-78959b4578-hxbht 1/1 Terminating 0 54d
dashboard-metrics-scraper-566c858889-zb67b 0/1 Pending 0 33d
dashboard-metrics-scraper-5f44bbb8b5-72mzn 1/1 Terminating 0 62d
dashboard-metrics-scraper-5f44bbb8b5-xfmb4 0/1 Pending 0 32m
kube-proxy-2hvgb 1/1 Running 0 62d
kube-proxy-sjmvp 1/1 Running 0 62d
kube-proxy-w8n74 1/1 Running 0 62d
kube-proxy-wl2g4 1/1 Running 0 62d
kubernetes-dashboard-785654f667-f8f6z 1/1 Terminating 0 62d
kubernetes-dashboard-7f7d6bbd7f-cschs 0/1 Pending 0 33d
metrics-server-6cd7558856-7gb58 0/1 Pending 0 33d
metrics-server-85c57978c6-dnmml 1/1 Terminating 0 55d
omsagent-k7bs6 1/1 Terminating 0 61d
omsagent-m57lm 1/1 Terminating 0 61d
omsagent-r565v 1/1 Terminating 0 61d
omsagent-rs-669fd8467-78kl7 1/1 Terminating 0 55d
omsagent-rs-6b8f79bd9b-czpwl 0/1 Pending 0 42d
omsagent-rs-7d6b44bbdd-jhn5b 0/1 Pending 0 33d
omsagent-vrg86 1/1 Terminating 0 61d
tiller-deploy-77d5bddbc9-tcm2b 0/1 Pending 0 30m
tiller-deploy-77d5bddbc9-tvl6h 1/1 Terminating 0 62d
tunnelfront-5c54945cc5-gn4j6 0/2 Pending 0 33d
tunnelfront-976977d47-b9v54 2/2 Terminating 0 53d
As the subscription got disabled, all the nodes might have got deleted. From the details shared, looks like there are no nodes available for AKS.
Please check the availability of nodes kubectl get nodes If nothing is healthy, then would recommend to scale up so that you will get new nodes and the pods will get deployed to the healthy nodes.
To avoid running into this is to ensure the subscription is not disabled.
Hope this information is helpful.
Please 'Accept as answer' if it helped, so that it can help others in the community looking for help on similar topics.
You are right that all of my nodes are in the NotReady state. However, I had autoscaling enabled on the cluster, and since the the autoscaler pod is currently in the Terminating state, it can't respond to a request to switch to manual scaling - let alone rescale the nodepools.
To avoid running into this is to ensure the subscription is not disabled.
I don't find this to be very useful advice. We have a set allocation at the Turing and requesting credits involves a review process with people who work on their own timescales. Also, our account with Azure only allows us to see what we've spent not how many credits are remaining. I would love to be able to automate notifications of when my subscription is about to run out so I can avoid it being disabled, but it's just not feasible right now. So as you can see, there are many factors working against me to ensure my subscription doesn't become disabled.
Best practices might suggest that the de-/re-provisioning of nodes in these scenarios is done more gracefully so that users are less likely to run into this issue.