AKS: Allocatable memory on a node is 2 out of 4 GB. I expected 3... how to increment?

Alessandro Dentella 1 Reputation point
2021-05-06T17:18:58.46+00:00

In a little cluster (2 nodes, 4 GB each) I have problems updating a pod (0/2 nodes are available: 2 Insufficient memory.).

I noticed that allocatable memory is only 2 out of 4 BG, while docs say that I should be able to allocate 75%.

I'm puzzled also by the fact that a similar 1 node cluster I use for testing does not seem to suffer from the same problem and allowed me to update the system. We're using Kubernetes 1.19.

I understand I cannot upgrade the node's VM. Please let me know if I'm wrong and, in case, which is the procedure.

Any help is appreciated. TIA

Azure Kubernetes Service
Azure Kubernetes Service
An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
2,458 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. prmanhas-MSFT 17,946 Reputation points Microsoft Employee Moderator
    2021-05-07T06:59:34.367+00:00

    @Alessandro Dentella Apologies for the delay in response and all the inconvenience caused because of the issue.

    As mentioned here:

    Memory utilized by AKS includes the sum of two values.

    kubelet daemon
    The kubelet daemon is installed on all Kubernetes agent nodes to manage container creation and termination.

    By default on AKS, kubelet daemon has the memory.available<750Mi eviction rule, ensuring a node must always have at least 750 Mi allocatable at all times. When a host is below that available memory threshold, the kubelet will trigger to terminate one of the running pods and free up memory on the host machine.

    A regressive rate of memory reservations for the kubelet daemon to properly function (kube-reserved).

    • 25% of the first 4 GB of memory
    • 20% of the next 4 GB of memory (up to 8 GB)
    • 10% of the next 8 GB of memory (up to 16 GB)
    • 6% of the next 112 GB of memory (up to 128 GB)
    • 2% of any memory above 128 GB

    Memory and CPU allocation rules:

    Keep agent nodes healthy, including some hosting system pods critical to cluster health.
    Cause the node to report less allocatable memory and CPU than it would if it were not part of a Kubernetes cluster.
    The above resource reservations can't be changed.

    For example, if a node offers 7 GB, it will report 34% of memory not allocatable including the 750Mi hard eviction threshold.

    0.75 + (0.25*4) + (0.20*3) = 0.75GB + 1GB + 0.6GB = 2.35GB / 7GB = 33.57% reserved

    In addition to reservations for Kubernetes itself, the underlying node OS also reserves an amount of CPU and memory resources to maintain OS functions.

    Another example is mentioned here as well:

    Here is an example to illustrate Node Allocatable computation:

    Node has 32Gi of memory, 16 CPUs and 100Gi of Storage
    --kube-reserved is set to cpu=1,memory=2Gi,ephemeral-storage=1Gi
    --system-reserved is set to cpu=500m,memory=1Gi,ephemeral-storage=1Gi
    --eviction-hard is set to memory.available<500Mi,nodefs.available<10%
    Under this scenario, Allocatable will be 14.5 CPUs, 28.5Gi of memory and 88Gi of local storage. Scheduler ensures that the total memory requests across all pods on this node does not exceed 28.5Gi and storage doesn't exceed 88Gi. Kubelet evicts pods whenever the overall memory usage across pods exceeds 28.5Gi, or if overall disk usage exceeds 88Gi If all processes on the node consume as much CPU as they can, pods together cannot consume more than 14.5 CPUs.

    If kube-reserved and/or system-reserved is not enforced and system daemons exceed their reservation, kubelet evicts pods whenever the overall node memory usage is higher than 31.5Gi or storage is greater than 90Gi.

    Below URL might help you out with your memory insufficient error:

    https://stackoverflow.com/questions/53192999/pod-dont-run-insufficient-resources

    https://serverfault.com/questions/1042049/why-are-pods-failing-to-schedule-due-to-resources-when-node-has-plenty-available

    When you say you need to upgrade node VM your understanding is correct AKS nodepools don't support resize in place, so the supported way to do this is to create a new nodepool with a new target and delete the previous one. This needs to be done through the AKS portal UX. This maintains the goal state of the AKS node pool, as at the moment the portal is showing the VMSize AKS knows you have because that is what was originally requested.

    For updates regarding AKS cluster you can refer to this.

    You can refer to this for more information.

    Hope it helps!!!

    Please "Accept as Answer" if it helped so it can help others in community looking for help on similar topics

    1 person found this answer helpful.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.