New node replaces node instead of extending the nodepool. Why?

Nuffi 1 Reputation point
2022-01-28T03:32:26.777+00:00

I have been dropped in the deep end, taking over an existing Azure based K8s cluster with literally zero experience or knowledge in Azure or Kubernetes.

First day, I see that it's flatlining 100% memory utilisation.

After a lot of digging around I think I learned that there's no simple way to just increase memory capacity in Azure nodes like there is in a Hyper-V environment. Is this correct?

It looks like I have two options: i) upgrade the node types, which also increases CPU and disk; or ii) scale the nodepool.

I chose to scale the nodepool. It makes sense as there are presently 2 nodes, and 3 is the recommendation.

I have watched it create a new node. And then it transfers the services over to the new node. And then it deletes the oldest node.

How do I get it to create a new node and add it to the nodepool? I have tried both the CLI and using the azure portal, but either way ends up the same.

TIA

Nuffi

Azure Kubernetes Service (AKS)
Azure Kubernetes Service (AKS)
An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
1,854 questions
0 comments

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Vidya Narasimhan 2,201 Reputation points Microsoft Employee
    2022-01-28T05:00:44.377+00:00

    @Nuffi Can you please share the command you used to manually scale? Also please check if cluster-autoscaler is enabled and a max-count for nodes has been set.

    Ideally, you should configure horizontal pod autoscaling and cluster autoscaling to scale your pods and nodes as per the demand. Please follow the recommendations here https://learn.microsoft.com/en-us/azure/architecture/reference-architectures/containers/aks/secure-baseline-aks?toc=https%3A%2F%2Flearn.microsoft.com%2Fen-us%2Fazure%2Faks%2Ftoc.json&bc=https%3A%2F%2Flearn.microsoft.com%2Fen-us%2Fazure%2Fbread%2Ftoc.json#node-and-pod-scalability

    Also ensure that you define pod resource requirements as per this link https://learn.microsoft.com/en-us/azure/aks/developer-best-practices-resource-management

    0 comments
  2. Nuffi 1 Reputation point
    2022-01-28T06:54:42.48+00:00

    Thank you for your help.

    I've done this several times, in different ways. The CLI command I used was:

    az aks scale --resource-group akscluster-rg --name akscluster --node-count 3 --nodepool-name nodepool

    az aks nodepool list --cluster-name akscluster -g akscluster-rg shows
    "maxCount": null,
    "maxPods": 110

    Does that mean that there's no maxlimit set? I couldn't see anything else that would be related to that. Was there a problem in the command I ran to extend the nodepool?

    I totally understand all the very excellent reasons autoscaling is recommended, but I am unable to get approval for setting up autoscaling. Which is why I need to do this manually.

    Thanks again!