Node autoscaler is not working.

sai theja 0

Hi Team,

We had a recent issue, where node autoscaler is not successful,

We kept the node to be in autoscale mode and quota of the nodes to the specific region are available.

And no taints are provided onto nodes.

Yet, one of the pod was in pending state and no node is allocated to the pod.

On describe of the pending pod, it states, insufficient CPU and memory on one of the node.

As per my understanding if node is not available to allocate, it has to create a new node for the allocation of pod and run.

But it didn't happened, later the memory config of pod is reduced it got allocated, and started running.

Please help me on what might be the issue and node is not getting autoscaled.

Details:

Node count : 17

Node range : 40

Node config: Standard DS12V2.

Regional quota is available.

max pods per node is 30.

pod config:

Requests:

CPU: 2

memory : 14 GB.

Later changed to 10 GB and it started running and no new node allocation is done.

shiva patpi 13,141 Reputation points Microsoft Employee

2023-03-26T05:56:35.94+00:00
@sai theja , Can you provide the whole POD yaml file please !!

Also try to check kubectl describe cm cluster-autoscaler-status -n kube-system while repro'ing the issue.

I just tried locally , with those requests autoscaling should happen:

apiVersion: v1 kind: Pod metadata: creationTimestamp: null labels: run: nginx name: nginx2 spec: containers: - image: nginx name: nginx resources: requests: cpu: 2 memory: 14Gi
sai theja 0 Reputation points

2023-03-26T12:14:17.14+00:00

Hi @shiva patpi ,

I have verified the config map, on cluster autoscaler.

It was updating the info, i didn't find any issue on it.

I have provided the both requested info, please look into it.

pod.txt
vipullag-MSFT 24,111 Reputation points Microsoft Employee

2023-03-28T06:17:16.8466667+00:00

Hello sai theja

Thanks for responding back and sharing more details.

This issue needs deeper investigation. Support team will be able to check and help on this. I would recommend you to open a azure support case.

If you don't have a support plan enabled on your subscription, I request you to send an email to AzCommunity@Microsoft.com with Subject as "Attn:Vikas" referencing this thread along with your subscription ID. I can then enable your subscription with a one-time free technical support.

Once the issue is resolved, request you to post the resolution here for the benefit of community.