Setting custom script extension in virtual machine scale set is changing the status of instances to NotReady.
I have set custom script extension using az command az vmss extension set --publisher Microsoft.Azure.Extensions --version 2.0 --name CustomScript --resource-group resource_group_name --vmss-name vmss_name --settings @./customConfig.json
Extension is set successfully and I can log into the instance and task by extension is done as expected. But unable to create any pod since the instance status is NotReady.
It can be viewed by executing command: $kubectl get nodes
By further debugging I got to know that kubelet and kubectl services are inactive(dead).I suspect this is because internally while checking health the expected path should be /usr/local/bin/kubectl But it is unable to find kubectl in /usr/local/bin path . The reason for this is after executing custom script extension many versions of kubelet and kubectl are being created .
How can avoid creating multiple versions of kubelet and kubectl so that My instance will be healthy?
NOTE: I am able to see the status of instance as Running in the Azure portal. I am able to view scaled instances only in the Azure portal but not by the command (kubectl get nodes).
Unable to figure it out! Can anyone help me
what are all the actions done by custom script on that machine?
Is it touching kubelet
anonymous user It is not at all touching kubelet. I tried even with just an echo command to be executed by custom script extension which resulted in same problem!
Are you updating the scale set to the latest version after applying the Custom Script? Or do you have automatic OS upgrades enabled?
Yes , I am updating scale set to the latest version.
I am not sure if updating the VMSS which is used by AKS is supported or not.
I will check with the team and let you know
When we update the scale set to the latest version, VM's may get redeployed. That explains the node not ready and comes back to ready after some time. But it should not add more kubelet executables as shown in the image
Did you considered using Daemonset using kubernetes for executing scripts on all the nodes in the cluster?
This will help if you want to execute a script on all nodes and new nodes as well.
Is there any update on this? Let us know.
Yes, I have tried using daemonset but the constraint is memory. By using concept of Daemonset each node need to have a pod which is consuming around 80 MB.
Sign in to comment