linux webapp docker temp images not clearing (30gb used)

Lukasz Andrzejak 126 Reputation points
2021-09-08T12:25:23.197+00:00

We have experienced an outage yesterday caused by lack of disk space for our linux container webapp.
We have found these to be caused by usage of temp files by /mnt/data/docker/images, they appear to not clear automatically (30/60gb currently used).
There appears to be no clear way to get to these for a linux webapp and clear them.

Any guidance on how to purge/prune these would be greatly appreciated.

What we saw and how we found it:

Docker logs:

2021-09-07T07:00:31.127Z ERROR - Container start failed for <our app>_3_27fc9faa with System.AggregateException, One or more errors occurred. (Docker API responded with status code=InternalServerError, response={"message":"OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: rootfs_linux.go:59: mounting \"/var/LWASFiles/Sites/<our app>_Common/appsvctmp\" to rootfs at \"/mnt/data/docker/images/231072.231072/aufs/mnt/6a271eaf624e2ff8a9b686de44fc52380b22bc1f150e2359cd41d37c4e2b3370/appsvctmp\" caused: mkdir /mnt/data/docker/images/231072.231072/aufs/mnt/6a271eaf624e2ff8a9b686de44fc52380b22bc1f150e2359cd41d37c4e2b3370/appsvctmp: no space left on device: unknown"})   
(Docker API responded with status code=InternalServerError, response={"message":"OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: rootfs_linux.go:59: mounting \"/var/LWASFiles/Sites/<our app>_Common/appsvctmp\" to rootfs at \"/mnt/data/docker/images/231072.231072/aufs/mnt/6a271eaf624e2ff8a9b686de44fc52380b22bc1f150e2359cd41d37c4e2b3370/appsvctmp\" caused: mkdir /mnt/data/docker/images/231072.231072/aufs/mnt/6a271eaf624e2ff8a9b686de44fc52380b22bc1f150e2359cd41d37c4e2b3370/appsvctmp: no space left on device: unknown"})  
InnerException: Docker.DotNet.DockerApiException, Docker API responded with status code=InternalServerError, response={"message":"OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: rootfs_linux.go:59: mounting \"/var/LWASFiles/Sites/<our app>_Common/appsvctmp\" to rootfs at \"/mnt/data/docker/images/231072.231072/aufs/mnt/6a271eaf624e2ff8a9b686de44fc52380b22bc1f150e2359cd41d37c4e2b3370/appsvctmp\" caused: mkdir /mnt/data/docker/images/231072.231072/aufs/mnt/6a271eaf624e2ff8a9b686de44fc52380b22bc1f150e2359cd41d37c4e2b3370/appsvctmp: no space left on device: unknown"}  

We are using very little of our app service plan storage:

130177-plan-filesystem-storage.png

When checking disk usage from Kudu we get the following:

130169-inside-kudu-file-usage.png

When checking disk usage from inside the container we get the following:

130225-inside-container-file-usage.png

We use two slots for our webapp (one for dev and one prod).

We temporarly resolved the issue by removing application log files from the dev slot container (which luckily managed to restart).

The webapp is linked to our azure devops build&release pipelines, and as this is an active project we are pushing new images frequently.

When searching for the cause of this we found that the space was being used for temp files - 30gb (Accessed via "Diagnose and solve problems" > "Best Practices for Availability & Performance"):

130242-temp-storage-30gb.png

We currently have 60gb of temp images in /mnt/data/docker/images (Accessed via "Diagnose and solve problems" > "Linux - Host Disk Space Usage"):

130234-docker-images-usage-60gb.png

We have tried the recommended solutions:
WEBSITE_DISABLE_SCM_SEPARATION - this does not appear to work (not surprising as its clearly marked as not supported.
The youtube tutorial provided shows a different version of KUDU from ours and refers to windows.
Restarting the webapps does not result in these files being cleared.
Application insights are not enabled.

Scaling our app service plan will only delay the issue as the files appear to not be cleared automatically.
Tearing down the app and rebuilding it to clear these seems excessive and does not resolve the issue in the long term.

I have found references to this issue occurring here, but our applications do not have application insights enabled.

For now there appears to be no way to clear the these files (/mnt/data/docker/images), and they appear to not clear automatically (30gb seems excessive for the containers - it would take dozens of full images to take that much space).
Any guidance would be greatly appreciated.

Azure App Service
Azure App Service
Azure App Service is a service used to create and deploy scalable, mission-critical web apps.
7,362 questions
{count} vote

Accepted answer
  1. Lukasz Andrzejak 126 Reputation points
    2021-09-20T11:02:43.283+00:00

    We have reached out to microsoft azure support and gotten back the following response:

    "The temp usage we are seeing is to be expected as it is taken up by blessed images used by the app service in other configurations (these are said to take up 23gb of storage)."

    Our solution was to redirect logs to stderr and decrease their verbosity (our application would log detailed http requests and responses for third party services), thus moving log management to azure (we set it to 100mb or 60 days retention).

    One other feature that can take up space here is application insights - application dumps are stored there.

    There is no way to clear the storage used for blessed images (if, like us you are using a custom container and do not need them) - it's just a part of the app service.

    There is no way to access this storage area under linux (tutorials suggest it should be possible under windows by accessing D:\temp after setting the WEBSITE_DISABLE_SCM_SEPARATION flag).

    The flag WEBSITE_DISABLE_SCM_SEPARATION does nothing under linux (there are reports of it actually crashing the container, though I cannot confirm that - it just does nothing).

    Overall Kudu for linux webapps appears to be slightly different to the windows versions and some tutorials/documentation do not apply.

    The size of the app service storage is a bit misleading when using a docker container, what they actually run on is temp storage.
    The actual size is related to these and is described here (if there is a suggestion box anywhere, I would ask that the pricing tables be updated to include these values - finding them is not easy)

    In order to get access to the 250gb storage area you MUST use shared storage if you are using containers. You can do this by setting the flag WEBSITES_ENABLE_APP_SERVICE_STORAGE to true in the configuration blade of your app service. This storage is always mounted under /home (owned by nobody.nogroup - uid.guid is 65534.65534), and will contain a folder structure described here.

    If you want your app to use this storage, be very mindful of Path Traversal vulnerabilities - if you are using a framework your app should be fine, but if you are rolling your own file access (or are accessing it directly) make sure you don't expose yourself to this vulnerability.

    If your container uses isolation techniques (chroot, SELinux, open_basedir), you will need to work around this - mount --bind perhaps, you will need to work within the pre-defined folder structure and manage permissions inside the container. Also prepare to build a default /home to fallback on when running the container locally.

    3 people found this answer helpful.

0 additional answers

Sort by: Most helpful