Discrepancy between the result of "kubectl top nodes" Memory Utilization compared to the total memory utilization of "kubectl top pods" deployed under the corresponding nodes

Question

Discrepancy between the result of "kubectl top nodes" Memory Utilization compared to the total memory utilization of "kubectl top pods" deployed under the corresponding nodes

Poojari, Vinaya (Cognizant) 0

when we execute, kubectl top nodes ,we can see below output

for eg:

         Memory-Utilization

node1 60%

node2 79%

but if we verify the sum of the Memory Utilization(%) for all the pods deployed inside the node2 using kubectl top pods then it doesn't match 79% instead the sum would be very less for eg: 45%.

We would like to understand where are the extra utilization is being consumed and how to check that.

kobulloc-MSFT 26,801 Reputation points Microsoft Employee Moderator

2023-07-19T16:34:00.9233333+00:00

Hello, @Poojari, Vinaya (Cognizant) ! I believe the difference is allocated memory versus memory that is actually in use. I'll confirm this and provide an update.
B santhiswaroop naik 405 Reputation points

2023-07-19T17:17:39.31+00:00
Hello , @Poojari, Vinaya (Cognizant)

If the sum of Memory Utilization(%) for all the pods deployed inside node2 doesn't match the reported overall utilization of 79%, there might be additional memory consumption on the node that is not accounted for by the individual pod memory usage. This could be due to system processes, cached data, or other background tasks running on the node.

To investigate where the extra memory utilization is coming from, you can use various tools and commands available on the node. Here are some steps to help you identify the source of the extra memory usage:

Check Node Metrics: You can use kubectl top node to get an overview of the resource usage on the node. This will give you an idea of the overall memory utilization, CPU usage, and other resource metrics. Run the following command to view the node metrics:

kubectl top node <node_name>

Replace <node_name> with the actual name of node2.

Examine System Processes: You can use tools like htop or top to see the list of processes running on the node and their memory consumption. These tools provide a dynamic view of system resource usage, including memory. Install htop if you don't have it:

sudo apt-get install htop # For Debian/Ubuntu

Then, run htop:

htop

You can sort processes by memory usage (press F6 and select %MEM) to identify any processes consuming significant memory.

Check Docker Container Memory Usage: If your pods are running in Docker containers, you can use the docker stats command to view the memory usage of individual containers. SSH into the node and run the following command:

docker stats --no-stream

This will display the resource usage of all containers running on the node, including memory usage.

Check Kubernetes Events: Use the following command to check the events related to the node, which might provide insights into any issues or unusual activities:

kubectl get events --field-selector involvedObject.name=<node_name>

Replace <node_name> with the actual name of node2.

Check Kubernetes Resource Limits: Review the resource limits set on the pods deployed on node2. Ensure that the memory requests and limits are appropriately configured. Incorrect or overly restrictive resource limits can cause memory issues.

Investigate Kubernetes Add-ons and Services: If you have additional Kubernetes add-ons or services running on node2, they could be consuming memory resources. Check the deployment of such add-ons and their resource usage.

By following these steps, you should be able to identify where the extra memory utilization is being consumed on node2. Keep in mind that resource usage can fluctuate over time, so it's essential to monitor the system and investigate any significant variations in memory consumption.

Hope this information helps.

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.
kobulloc-MSFT 26,801 Reputation points Microsoft Employee Moderator

2023-07-19T22:28:00.1+00:00

Hello, @Poojari, Vinaya (Cognizant) !

I hope this has been helpful! Your feedback is important so please take a moment to accept answers.

If you still have questions, please let us know what is needed in the comments so the question can be answered. Thank you for helping to improve Microsoft Q&A!

1 answer

Your answer

kobulloc-MSFT 26,801 Reputation points Microsoft Employee Moderator

2023-07-19T16:34:00.9233333+00:00

Hello, @Poojari, Vinaya (Cognizant) ! I believe the difference is allocated memory versus memory that is actually in use. I'll confirm this and provide an update.
kobulloc-MSFT 26,801 Reputation points Microsoft Employee Moderator

2023-07-19T22:28:00.1+00:00

Hello, @Poojari, Vinaya (Cognizant) !

I hope this has been helpful! Your feedback is important so please take a moment to accept answers.

If you still have questions, please let us know what is needed in the comments so the question can be answered. Thank you for helping to improve Microsoft Q&A!

Answer 1

kobulloc-MSFT 26,801 Microsoft Employee Moderator

Hello, @Poojari, Vinaya (Cognizant) !

Why doesn't the memory usage shown for kubectl top nodes match the memory usage shown for all of the pods in kubectl top pods?

Executing kubectl top nodes includes not only the memory used by the pods but also the memory used by the Kubernetes components and system processes running on the node while kubectl top pods returns all the memory used by the pod.

Additional reading:

René Engel 0 Reputation points

2024-02-16T22:26:19.86+00:00

NO

As said in the kubernetes docs regarding "--show-capacity" argument:

Print node resources based on Capacity instead of Allocatable(default) of the nodes.

From this article https://hwchiu.medium.com/introduction-to-kubernetes-resources-capacity-and-allocatable-4dc1bfbd1caf#:~:text=The%20total%20amount%20of%20resources,through%20%E2%80%9Ckubectl%20describe%20node%E2%80%9D.

The total amount of resources available on the node is referred to as Capacity, while the total amount that can be allocated to Kubernetes Pods is referred to as Allocatable.

This means, that "kubectl node top" should not report memory "used by the Kubernetes components and system processes running on the node".

Is the kubernetes documentation wrong?

Share via

Discrepancy between the result of "kubectl top nodes" Memory Utilization compared to the total memory utilization of "kubectl top pods" deployed under the corresponding nodes

1 answer

Your answer