Azure AKS - Disk bytes used suddenly take GB/TB

Yannick VAN STYVENDAEL 41 Reputation points
2023-07-31T08:51:44.1966667+00:00

Dear Community,

Since the 21 july 2023 at 8pm, for a reason I ignore (not any change on the AKS infrastructure), the disk used bytes switch from some bytes to 800GB for the system pool and more than 2.5TB for the user pool.

User's image

How can I explain this increase since the OS disk size is only 128GB ?

I checked a df -h on the nodes and I can find a lot of overlay files taking lots of GB. Is that normal ?

root@aks-agentpool-95240818-vmss000005:/# df -H

Filesystem Size Used Avail Use% Mounted on

overlay 134G 24G 110G 18% /

tmpfs 68M 0 68M 0% /dev

/dev/root 134G 24G 110G 18% /host

devtmpfs 2.1G 0 2.1G 0% /host/dev

tmpfs 2.1G 0 2.1G 0% /dev/shm

tmpfs 824M 2.2M 822M 1% /host/run

tmpfs 5.3M 0 5.3M 0% /host/run/lock

shm 68M 0 68M 0% /host/run/containerd/io.containerd.grpc.v1.cri/sandboxes/21088829b6b292dcf36a00e0281abfd74a17a9ba4b1cd73aaec3b2d0036c6e9c/shm

shm 68M 0 68M 0% /host/run/containerd/io.containerd.grpc.v1.cri/sandboxes/ce4103bd0bb0e4dba6f6c29068a9868e9e7cf0083f0bbf59ce7ad526f3e17636/shm

shm 68M 0 68M 0% /host/run/containerd/io.containerd.grpc.v1.cri/sandboxes/2e76c3c92b236f261dbc60427aa687f6f123667578349ea617ce6dcd3eebb365/shm

shm 68M 0 68M 0% /host/run/containerd/io.containerd.grpc.v1.cri/sandboxes/6196396872ab86bd09836e83ba36e8cf1bb0ff1d0a60bec4e868909dd349709d/shm

shm 68M 0 68M 0% /host/run/containerd/io.containerd.grpc.v1.cri/sandboxes/cc5e5cd2f11691d894cd9ef79c3bb60ad6941c5c962bba7767f4bf74255bd542/shm

overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/ce4103bd0bb0e4dba6f6c29068a9868e9e7cf0083f0bbf59ce7ad526f3e17636/rootfs

overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/2e76c3c92b236f261dbc60427aa687f6f123667578349ea617ce6dcd3eebb365/rootfs

overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/cc5e5cd2f11691d894cd9ef79c3bb60ad6941c5c962bba7767f4bf74255bd542/rootfs

overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/6196396872ab86bd09836e83ba36e8cf1bb0ff1d0a60bec4e868909dd349709d/rootfs

overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/21088829b6b292dcf36a00e0281abfd74a17a9ba4b1cd73aaec3b2d0036c6e9c/rootfs

overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/7d51b9093ed4d8665cf759ebead7e4d17c10bc235c77393db3b4354b11f22a92/rootfs

overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/a98e0a8fa73acd9e440cf2869107ea2f5a66e1d461d3e1ba0a24c707cbf445ac/rootfs

overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/6c0060027da6483dab55529323bf50e19a5c65da65a974567f70c564bee0b60e/rootfs

overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/5e0e35f60231c224fe3032d1146304af130f81c54752e121782e32c8b87554a0/rootfs

overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/2d9806d118955a82ae98fbe7c1dcfbd27a803b52d48b0dacc7eb9cc1de1a91e4/rootfs

overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/f8207c530e10f5960fdd02663014c467bb7b3b4d6fbc0adf255b411d375cb1cd/rootfs

overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/3bfbb51d475ff69620b47bc352707aa9714c6cdc3fe3b82d4a7aaf0f58df2752/rootfs

overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/4257978d9bac2be7111dc467d97ce570b8dd356f8a55d9cd56b21c1ff5662894/rootfs

overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/7cd8921cedd0d1f690510dcb66e1cfa489b5217482413f3779b7ccecc83b7dda/rootfs

overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/445edc955dab77744c4c43039739f803ea660a1421bcffb1b661f4aa2ab9b423/rootfs

Thanks for your help

Azure Kubernetes Service (AKS)
Azure Kubernetes Service (AKS)
An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
1,779 questions
0 comments No comments
{count} votes

Accepted answer
  1. mutaz-msft 2,256 Reputation points Microsoft Employee
    2023-07-31T15:24:54.1133333+00:00

    Hi @Yannick VAN STYVENDAEL,

    The usage is normal from the output of ( df -h ), but it seems in the screenshot your selection is to show the Sum not the Average, can you select the avg?

    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful