This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 9%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EFR%3C/text%3E%3C/svg%3E)
Hello all.
After manually adding a node with an YAML, AKS cluster stopped working.
The node doesn't appear on any of my node pools using az aks show or az aks nodepool.
The only information about the previous node is a troubleshooting obtained from node pools as shown in image below.
I tried to restart the cluster, but now it won't start either.
Any thoughts on this?
Hi @Fibiger, Roger,
it is not supported to add an AKS node using YAML, you can scale your node pool or add new node pool from Azure portal or using az cli.
https://learn.microsoft.com/en-us/azure/aks/scale-cluster?tabs=azure-cli
If the cluster is stuck in stop/start state, you can try to fix it uisng these commands:
az aks operation-abort --name myAKSCluster --resource-group myResourceGroup
az aks start --name myAKSCluster --resource-group myResourceGroup
Thank you for your reply, mutaz-msft. I tried the commands you provided with no success. Now I'm trying az resource update --ids as presented in this link: https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/cluster-node-virtual-machine-failed-state
The strange is that I'm not able to see any configuration about the node. And even more strange is that the whole cluster went down and won't go up.
Can you try to stop and start the cluster using az cli:
az aks stop --name myAKSCluster --resource-group myResourceGroup
az aks start --name myAKSCluster --resource-group myResourceGroup
if the cluster in updating state, try to use the abort command first.
I also tried to stop and start. It says: (OperationNotAllowed) Unable to perform 'stopping' operation since the cluster is in 'failed' provision state.
I'm still waiting for the result of "az resource update --ids", but it's happening for more than 1 hour.
If it is stuck in updating state then try to abort the operation
az aks operation-abort --name myAKSCluster --resource-group myResourceGroup
then run the update command:
az aks update --name myAKSCluster --resource-group myResourceGroup
I repro the same on my cluster and fixed after running the above commands
az resource update --ids didn't worked
What is the status of the cluster?
if it is in failed state, you can run:
az aks start --name myAKSCluster --resource-group myResourceGroup
Also, can you check the activity log on the VMSS?
Hi @Fibiger, Roger,
If the issue is still not fixed, can you open a support ticket so our team can take a look on your cluster.
Hello Roger,
Thanks for reaching out and I hope you are doing well.
I understand that you manually added a node using a YAML file to your AKS cluster, this is most likely causing the issue as manually adding a node is not a supported way. AKS clusters use nodepools (basically a VMSS) to handle the workload, by default, when you create an AKS Cluster a default nodepool is created. If you use a YAML file to create a node then AKS does not know where this node belongs in terms of nodepool, size, availability zone, etc. and that breaks the cluster, I tested this internally and noticed the same behavior.
Reference link: https://learn.microsoft.com/en-us/azure/aks/concepts-clusters-workloads#nodes-and-node-pools
You can add more nodes via the Azure Portal or by running the az aks nodepool add command:
https://learn.microsoft.com/en-us/azure/aks/create-node-pools#add-a-node-pool
One thing you can try to revert the changes is a cluster reconcile operation:
CLI:
az resource update -n <name of the cluster>-g <ResourceGroup Name> --resource-type Microsoft.ContainerService/ManagedClusters
However, that can take time and there is no guarantee that it will revert the changes, if not, the only option would be recreating the cluster.
Thanks,
Cristian.
That's sad. az resource update didn't worked for me. I saw a few posts asking to use az aks upgrade --resource-group myResourceGroup --name myAKSCluster --kubernetes-version <myAKSVersion> . Do you think this may help?
You can always give that a try but that is similar as running the reconcile operation. At this point, what it seems to be broken is the communication with the api-server, so any commands or actions that involves talking to the control plane will fail, I consider this unrecoverable and creating a new cluster is the best approach.
Sorry to hear that the above solutions did not directly resolve your issue. The suggestions made are all good. However, not everything can be resolved without Azure Support.
To further troubleshoot this, we're going to enable a free, one time technical support ticket. Please email the following to AzCommunity@microsoft.com and we'll get back to you promptly:
• Subject: "Attn: deherman - "
• Email body: Your Subscription ID
• Email body: A link to this thread so we can validate and expedite the request
If you don't receive a response within 24 hours, please reply to the thread so we can investigate.
If you still have questions, please let us know in the "comments" and we would be happy to help you. Comment is the fastest way of notifying the experts.
If the answer has been helpful, we appreciate hearing from you and would love to help others who may have the same question. Accepting answers helps increase visibility of this question for other members of the Microsoft Q&A community.
Thank you for helping to improve Microsoft Q&A!