This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(19.2, 66%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EYV%3C/text%3E%3C/svg%3E)
Dear Community,
Since the 21 july 2023 at 8pm, for a reason I ignore (not any change on the AKS infrastructure), the disk used bytes switch from some bytes to 800GB for the system pool and more than 2.5TB for the user pool.
How can I explain this increase since the OS disk size is only 128GB ?
I checked a df -h on the nodes and I can find a lot of overlay files taking lots of GB. Is that normal ?
root@aks-agentpool-95240818-vmss000005:/# df -H
Filesystem Size Used Avail Use% Mounted on
overlay 134G 24G 110G 18% /
tmpfs 68M 0 68M 0% /dev
/dev/root 134G 24G 110G 18% /host
devtmpfs 2.1G 0 2.1G 0% /host/dev
tmpfs 2.1G 0 2.1G 0% /dev/shm
tmpfs 824M 2.2M 822M 1% /host/run
tmpfs 5.3M 0 5.3M 0% /host/run/lock
shm 68M 0 68M 0% /host/run/containerd/io.containerd.grpc.v1.cri/sandboxes/21088829b6b292dcf36a00e0281abfd74a17a9ba4b1cd73aaec3b2d0036c6e9c/shm
shm 68M 0 68M 0% /host/run/containerd/io.containerd.grpc.v1.cri/sandboxes/ce4103bd0bb0e4dba6f6c29068a9868e9e7cf0083f0bbf59ce7ad526f3e17636/shm
shm 68M 0 68M 0% /host/run/containerd/io.containerd.grpc.v1.cri/sandboxes/2e76c3c92b236f261dbc60427aa687f6f123667578349ea617ce6dcd3eebb365/shm
shm 68M 0 68M 0% /host/run/containerd/io.containerd.grpc.v1.cri/sandboxes/6196396872ab86bd09836e83ba36e8cf1bb0ff1d0a60bec4e868909dd349709d/shm
shm 68M 0 68M 0% /host/run/containerd/io.containerd.grpc.v1.cri/sandboxes/cc5e5cd2f11691d894cd9ef79c3bb60ad6941c5c962bba7767f4bf74255bd542/shm
overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/ce4103bd0bb0e4dba6f6c29068a9868e9e7cf0083f0bbf59ce7ad526f3e17636/rootfs
overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/2e76c3c92b236f261dbc60427aa687f6f123667578349ea617ce6dcd3eebb365/rootfs
overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/cc5e5cd2f11691d894cd9ef79c3bb60ad6941c5c962bba7767f4bf74255bd542/rootfs
overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/6196396872ab86bd09836e83ba36e8cf1bb0ff1d0a60bec4e868909dd349709d/rootfs
overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/21088829b6b292dcf36a00e0281abfd74a17a9ba4b1cd73aaec3b2d0036c6e9c/rootfs
overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/7d51b9093ed4d8665cf759ebead7e4d17c10bc235c77393db3b4354b11f22a92/rootfs
overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/a98e0a8fa73acd9e440cf2869107ea2f5a66e1d461d3e1ba0a24c707cbf445ac/rootfs
overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/6c0060027da6483dab55529323bf50e19a5c65da65a974567f70c564bee0b60e/rootfs
overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/5e0e35f60231c224fe3032d1146304af130f81c54752e121782e32c8b87554a0/rootfs
overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/2d9806d118955a82ae98fbe7c1dcfbd27a803b52d48b0dacc7eb9cc1de1a91e4/rootfs
overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/f8207c530e10f5960fdd02663014c467bb7b3b4d6fbc0adf255b411d375cb1cd/rootfs
overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/3bfbb51d475ff69620b47bc352707aa9714c6cdc3fe3b82d4a7aaf0f58df2752/rootfs
overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/4257978d9bac2be7111dc467d97ce570b8dd356f8a55d9cd56b21c1ff5662894/rootfs
overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/7cd8921cedd0d1f690510dcb66e1cfa489b5217482413f3779b7ccecc83b7dda/rootfs
overlay 134G 24G 110G 18% /host/run/containerd/io.containerd.runtime.v2.task/k8s.io/445edc955dab77744c4c43039739f803ea660a1421bcffb1b661f4aa2ab9b423/rootfs
Thanks for your help
The usage is normal from the output of ( df -h ), but it seems in the screenshot your selection is to show the Sum not the Average, can you select the avg?
Hello,
Thanks a lot for your answer.
It seems the Avg shows indeed less data but starting from 21 july at 8pm, it shows now much more data than before.
Maybe your pods logs or data sizes increased, youcan ssh to the nodes and check which directory is consuming the most usage, then you will find which pod is consuming the disk, using these commands:
Hope this helps, and please "Accept as Answer" if it helped, so that it can help others in the community looking for help on similar topics.
Hello,
Thanks a lot for your answer.
With these commands, I've been able to see that the high space was used by /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs which was using 66GB.
We have been able to drastically reduce that space by running a crictl rmi --prune.
I still don't know why suddenly this folder took a lot of space but anyway, the disk space is now back to normal :).
Thanks again and have a nice day !
Hi Yannick VAN STYVENDAEL,
Thank you for your update, and please "Accept as Answer" if it helped, so that it can help others in the community looking for help on similar topics.