linux webapp docker temp images not clearing (30gb used)

Question

linux webapp docker temp images not clearing (30gb used)

Lukasz Andrzejak 126

We have experienced an outage yesterday caused by lack of disk space for our linux container webapp.
We have found these to be caused by usage of temp files by /mnt/data/docker/images, they appear to not clear automatically (30/60gb currently used).
There appears to be no clear way to get to these for a linux webapp and clear them.

Any guidance on how to purge/prune these would be greatly appreciated.

What we saw and how we found it:

Docker logs:

2021-09-07T07:00:31.127Z ERROR - Container start failed for <our app>_3_27fc9faa with System.AggregateException, One or more errors occurred. (Docker API responded with status code=InternalServerError, response={"message":"OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: rootfs_linux.go:59: mounting \"/var/LWASFiles/Sites/<our app>_Common/appsvctmp\" to rootfs at \"/mnt/data/docker/images/231072.231072/aufs/mnt/6a271eaf624e2ff8a9b686de44fc52380b22bc1f150e2359cd41d37c4e2b3370/appsvctmp\" caused: mkdir /mnt/data/docker/images/231072.231072/aufs/mnt/6a271eaf624e2ff8a9b686de44fc52380b22bc1f150e2359cd41d37c4e2b3370/appsvctmp: no space left on device: unknown"})   
(Docker API responded with status code=InternalServerError, response={"message":"OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: rootfs_linux.go:59: mounting \"/var/LWASFiles/Sites/<our app>_Common/appsvctmp\" to rootfs at \"/mnt/data/docker/images/231072.231072/aufs/mnt/6a271eaf624e2ff8a9b686de44fc52380b22bc1f150e2359cd41d37c4e2b3370/appsvctmp\" caused: mkdir /mnt/data/docker/images/231072.231072/aufs/mnt/6a271eaf624e2ff8a9b686de44fc52380b22bc1f150e2359cd41d37c4e2b3370/appsvctmp: no space left on device: unknown"})  
InnerException: Docker.DotNet.DockerApiException, Docker API responded with status code=InternalServerError, response={"message":"OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: rootfs_linux.go:59: mounting \"/var/LWASFiles/Sites/<our app>_Common/appsvctmp\" to rootfs at \"/mnt/data/docker/images/231072.231072/aufs/mnt/6a271eaf624e2ff8a9b686de44fc52380b22bc1f150e2359cd41d37c4e2b3370/appsvctmp\" caused: mkdir /mnt/data/docker/images/231072.231072/aufs/mnt/6a271eaf624e2ff8a9b686de44fc52380b22bc1f150e2359cd41d37c4e2b3370/appsvctmp: no space left on device: unknown"}

We are using very little of our app service plan storage:

When checking disk usage from Kudu we get the following:

When checking disk usage from inside the container we get the following:

We use two slots for our webapp (one for dev and one prod).

We temporarly resolved the issue by removing application log files from the dev slot container (which luckily managed to restart).

The webapp is linked to our azure devops build&release pipelines, and as this is an active project we are pushing new images frequently.

When searching for the cause of this we found that the space was being used for temp files - 30gb (Accessed via "Diagnose and solve problems" > "Best Practices for Availability & Performance"):

We currently have 60gb of temp images in /mnt/data/docker/images (Accessed via "Diagnose and solve problems" > "Linux - Host Disk Space Usage"):

We have tried the recommended solutions:
WEBSITE_DISABLE_SCM_SEPARATION - this does not appear to work (not surprising as its clearly marked as not supported.
The youtube tutorial provided shows a different version of KUDU from ours and refers to windows.
Restarting the webapps does not result in these files being cleared.
Application insights are not enabled.

Scaling our app service plan will only delay the issue as the files appear to not be cleared automatically.
Tearing down the app and rebuilding it to clear these seems excessive and does not resolve the issue in the long term.

I have found references to this issue occurring here, but our applications do not have application insights enabled.

For now there appears to be no way to clear the these files (/mnt/data/docker/images), and they appear to not clear automatically (30gb seems excessive for the containers - it would take dozens of full images to take that much space).
Any guidance would be greatly appreciated.

Pierre-Luc Giguere 1,071 Reputation points

2021-09-08T21:03:37.633+00:00

Hi Likasz,

I don't think I have an answer but I'll try to give you ideas to look at.

I confirm that WEBSITE_DISABLE_SCM_SEPARATION does not make anything on Linux (source: https://github.com/projectkudu/kudu/issues/2324)

Unfortunately we can't do a prune or a df on the docker repo since we are not on the VM but the image. So we can't confirm if it is indeed docker who's filling the mount.

The tmp disk looks like /dev/sdb1 but I can't get it to mount:

root@somerandomnumbers:~# lsblk -o NAME,HCTL,SIZE,MOUNTPOINT | grep -i "sd"
sda 0:0:0:0 64G
|-sda1 63.9G /usr/local/azure/certs
|-sda14 4M
`-sda15 106M
sdb 1:0:1:0 50G <<<< P1V2 is Standard_D1_v2 so 50Gb of temp space (https://learn.microsoft.com/en-us/azure/virtual-machines/dv2-dsv2-series)
-sdb1 50G
root@somerandomnumbers:~# mkdir /foo; mount -o ro,noload /dev/sdb1 /foo
mount: /foo: permission denied.

No way we can mount that disk on the docker image (I think?).

You said: "We temporarly resolved the issue by removing application log files from the dev slot container"

Can you check and see if the app generates a lot of logs?
What does the log stream look like?
Have you tried moving the logs to a storage account instead of the filesystem?

Hope it gives you ideas to move forward.
Lukasz Andrzejak 126 Reputation points

2021-09-09T11:39:45.43+00:00

hi @Pierre-Luc Giguere
This is a development slot, so has fairly verbose logging enabled.

I suspect large log files from pervious runs are still in the old images stored in the temp folder.

I have switched over to use stderr for logging in our application, so these should no longer take up space moving forward.

So we can't confirm if it is indeed docker who's filling the mount.

We can - the "Linux - Host Disk Space Usage" page explicitly isolates mount points and shows us:
Pierre-Luc Giguere 1,071 Reputation points

2021-09-09T12:20:07.827+00:00

Thanks for your feedback.

I stand corrected.

If I can spare some time I'll try to figure out something else but for now I can only recommend opening a ticket with Microsoft if you do have support. They should be able to help...
Ryan Hill 18,396 Reputation points Microsoft Employee

2021-09-09T18:08:11.167+00:00

Hi @Lukasz Andrzejak , I just wanted to follow to see if @Rutger Storm 's response help resolve your issue. Please do let me know.
Lukasz Andrzejak 126 Reputation points

2021-09-09T19:45:45.103+00:00

Hi @Ryan Hill ,
sadly - no, I checked and we don't have insights enabled.
though his description (error dumps taking up space) pointed me to a possible cause:
excessive logging in the dev slot might have created very large logfiles which are sitting in our temp folders in stale images.

This doesn't help us clear them down though :(
Lukasz Andrzejak 126 Reputation points

2021-09-09T19:49:22.087+00:00

Thanks for checking @Pierre-Luc Giguere !
My attempts at mounting this (like yours) have so far failed.

We are considering opening a ticket, we figured this forum Is a good place to start looking for solutions :)
Ryan Hill 18,396 Reputation points Microsoft Employee

2021-09-11T00:25:15.73+00:00

@Lukasz Andrzejak , is the dev slot doing standard app service logging? Anything special going?
Joao Moreno 1 Reputation point

2021-09-17T07:13:39.953+00:00

We're currently seeing exactly the same thing on a Linux Web App. @Lukasz Andrzejak , I was wondering if you have learned anything since you posted the question?
Ryan Hill 18,396 Reputation points Microsoft Employee

2021-09-19T00:43:33.51+00:00

@Joao Moreno you don't have Application Insights enabled on your app service either? Any other type of logging you're doing?
Joao Moreno 1 Reputation point

2021-09-20T07:10:30.52+00:00

We actually have a very specific app pattern. Our app caches considerably sized files (1MB-20MB) on disk. These files get rotated often and they always differ. Something tells me this is what creeps up the storage quickly.
Lukasz Andrzejak 126 Reputation points

2021-09-20T11:06:31.81+00:00

@Joao Moreno see my answer to the question.

A solution for you might be to use shared storage for these files (though I personally dislike accessing files outside of the project folder).

Pierre-Luc Giguere 1,071 Reputation points

2021-09-08T21:03:37.633+00:00

Hi Likasz,

I don't think I have an answer but I'll try to give you ideas to look at.

I confirm that WEBSITE_DISABLE_SCM_SEPARATION does not make anything on Linux (source: https://github.com/projectkudu/kudu/issues/2324)

Unfortunately we can't do a prune or a df on the docker repo since we are not on the VM but the image. So we can't confirm if it is indeed docker who's filling the mount.

The tmp disk looks like /dev/sdb1 but I can't get it to mount:

root@somerandomnumbers:~# lsblk -o NAME,HCTL,SIZE,MOUNTPOINT | grep -i "sd"
sda 0:0:0:0 64G
|-sda1 63.9G /usr/local/azure/certs
|-sda14 4M
`-sda15 106M
sdb 1:0:1:0 50G <<<< P1V2 is Standard_D1_v2 so 50Gb of temp space (https://learn.microsoft.com/en-us/azure/virtual-machines/dv2-dsv2-series)
-sdb1 50G
root@somerandomnumbers:~# mkdir /foo; mount -o ro,noload /dev/sdb1 /foo
mount: /foo: permission denied.

No way we can mount that disk on the docker image (I think?).

You said: "We temporarly resolved the issue by removing application log files from the dev slot container"

Can you check and see if the app generates a lot of logs?
What does the log stream look like?
Have you tried moving the logs to a storage account instead of the filesystem?

Hope it gives you ideas to move forward.
Lukasz Andrzejak 126 Reputation points

2021-09-09T11:39:45.43+00:00

hi @Pierre-Luc Giguere
This is a development slot, so has fairly verbose logging enabled.

I suspect large log files from pervious runs are still in the old images stored in the temp folder.

I have switched over to use stderr for logging in our application, so these should no longer take up space moving forward.

So we can't confirm if it is indeed docker who's filling the mount.

We can - the "Linux - Host Disk Space Usage" page explicitly isolates mount points and shows us:
Pierre-Luc Giguere 1,071 Reputation points

2021-09-09T12:20:07.827+00:00

Thanks for your feedback.

I stand corrected.

If I can spare some time I'll try to figure out something else but for now I can only recommend opening a ticket with Microsoft if you do have support. They should be able to help...
Ryan Hill 18,396 Reputation points Microsoft Employee

2021-09-09T18:08:11.167+00:00

Hi @Lukasz Andrzejak , I just wanted to follow to see if @Rutger Storm 's response help resolve your issue. Please do let me know.
Lukasz Andrzejak 126 Reputation points

2021-09-09T19:45:45.103+00:00

Hi @Ryan Hill ,
sadly - no, I checked and we don't have insights enabled.
though his description (error dumps taking up space) pointed me to a possible cause:
excessive logging in the dev slot might have created very large logfiles which are sitting in our temp folders in stale images.

This doesn't help us clear them down though :(
Lukasz Andrzejak 126 Reputation points

2021-09-09T19:49:22.087+00:00

Thanks for checking @Pierre-Luc Giguere !
My attempts at mounting this (like yours) have so far failed.

We are considering opening a ticket, we figured this forum Is a good place to start looking for solutions :)
Ryan Hill 18,396 Reputation points Microsoft Employee

2021-09-11T00:25:15.73+00:00

@Lukasz Andrzejak , is the dev slot doing standard app service logging? Anything special going?
Joao Moreno 1 Reputation point

2021-09-17T07:13:39.953+00:00

We're currently seeing exactly the same thing on a Linux Web App. @Lukasz Andrzejak , I was wondering if you have learned anything since you posted the question?
Ryan Hill 18,396 Reputation points Microsoft Employee

2021-09-19T00:43:33.51+00:00

@Joao Moreno you don't have Application Insights enabled on your app service either? Any other type of logging you're doing?
Joao Moreno 1 Reputation point

2021-09-20T07:10:30.52+00:00

We actually have a very specific app pattern. Our app caches considerably sized files (1MB-20MB) on disk. These files get rotated often and they always differ. Something tells me this is what creeps up the storage quickly.
Lukasz Andrzejak 126 Reputation points

2021-09-20T11:06:31.81+00:00

@Joao Moreno see my answer to the question.

A solution for you might be to use shared storage for these files (though I personally dislike accessing files outside of the project folder).

Answer 1

We have reached out to microsoft azure support and gotten back the following response:

"The temp usage we are seeing is to be expected as it is taken up by blessed images used by the app service in other configurations (these are said to take up 23gb of storage)."

Our solution was to redirect logs to stderr and decrease their verbosity (our application would log detailed http requests and responses for third party services), thus moving log management to azure (we set it to 100mb or 60 days retention).

One other feature that can take up space here is application insights - application dumps are stored there.

There is no way to clear the storage used for blessed images (if, like us you are using a custom container and do not need them) - it's just a part of the app service.

There is no way to access this storage area under linux (tutorials suggest it should be possible under windows by accessing D:\temp after setting the WEBSITE_DISABLE_SCM_SEPARATION flag).

The flag WEBSITE_DISABLE_SCM_SEPARATION does nothing under linux (there are reports of it actually crashing the container, though I cannot confirm that - it just does nothing).

Overall Kudu for linux webapps appears to be slightly different to the windows versions and some tutorials/documentation do not apply.

The size of the app service storage is a bit misleading when using a docker container, what they actually run on is temp storage.
The actual size is related to these and is described here (if there is a suggestion box anywhere, I would ask that the pricing tables be updated to include these values - finding them is not easy)

In order to get access to the 250gb storage area you MUST use shared storage if you are using containers. You can do this by setting the flag WEBSITES_ENABLE_APP_SERVICE_STORAGE to true in the configuration blade of your app service. This storage is always mounted under /home (owned by nobody.nogroup - uid.guid is 65534.65534), and will contain a folder structure described here.

If you want your app to use this storage, be very mindful of Path Traversal vulnerabilities - if you are using a framework your app should be fine, but if you are rolling your own file access (or are accessing it directly) make sure you don't expose yourself to this vulnerability.

If your container uses isolation techniques (chroot, SELinux, open_basedir), you will need to work around this - mount --bind perhaps, you will need to work within the pre-defined folder structure and manage permissions inside the container. Also prepare to build a default /home to fallback on when running the container locally.

SvenGlöckner 396 Reputation points

2021-11-19T09:18:16.533+00:00

Hi @Lukasz Andrzejak

Currently, we're facing exactly the same issue.
I've some questions.

How did you manage redirect logs to stderr? We're running a custom container image with a NodeJS app inside running expressjs. It also logs any http request to console.log.

How did you move log management to Azure?

Thank you!
Sven

linux webapp docker temp images not clearing (30gb used)

0 additional answers

Activity