Azure App Service LInux temp storage 24gB

Question

Azure App Service LInux temp storage 24gB

Jamie Greeve 31

We are currently running into a problem in our Linux App Service resources.

The first thing that we have noticed is that for a brand new, near empty WebAPI deployment – into a fresh WebApp, App Service and Resource group – we have an immediate temp storage usage of +/- 22GB. When we compare this with Windows App Services (+/- 200MB) this seems excessive. Is this normal, or do we have an anomaly?

Ryan Hill 30,276 Reputation points Microsoft Employee Moderator

2021-04-07T22:17:04.347+00:00

Hi @Jamie Greeve ,

This is abnormal. What web stack are you using? What are you deploying? How are you deploying?

Regards,
Ryan
JimmyYang-MSFT 58,631 Reputation points Microsoft External Staff

2021-04-08T01:27:27.383+00:00

@Jamie Greeve

According to your description, your issue is not in Teams for Linux scope. So I will remove teams-for-linux tag from your thread. Thanks for your understanding and patience!
Jamie Greeve 31 Reputation points

2021-04-08T07:01:12.833+00:00

Hi @Ryan Hill ,

Stack .Net
Major version .Net Core 3
Minor version .Net Core 3.1 (LTS)

We are deploying a small app service.

Deployment is done via Azure Devops, but the same happens if we directly deploy from Visual studio.

Regards,
Jamie
Ryan Hill 30,276 Reputation points Microsoft Employee Moderator

2021-04-08T12:30:50.687+00:00

What region are you deploying to? I'll try to repo to see if I get the same result.
Jamie Greeve 31 Reputation points

2021-04-08T12:32:10.437+00:00

In west Europe we are deploying this.
Ryan Hill 30,276 Reputation points Microsoft Employee Moderator

2021-04-10T03:26:25.247+00:00

Hi @Jamie Greeve ,

Sorry for the late reply. I tried to repro but my app measured up to 35.21MiB (pictured below) on an S1 plan. When you browse the temp storage, what folder is showing a lot of space? What do the contents look like?
Jamie Greeve 31 Reputation points

2021-04-13T05:42:55.613+00:00

Hi @Ryan Hill ,

Thanks for your reply, sorry for my late reply.

We are looking in different metrics. This is at the best practices of the webapp. The metrics you showed are indeed almost the same, and the issue is not there.

If we go to the app service, then the 'Diagnose and solve problems'. In the app diagnostic you can search for 'Best Practices for Availability & Performance'. In here is an item 'Temp File Usage On Workers'. This metric shows this huge amount of storage while the webapp is just started. We can not see this metrics anywhere else, but after a while this starts growing. Do you have any idea?

2 answers

Your answer

Ryan Hill 30,276 Reputation points Microsoft Employee Moderator

2021-04-07T22:17:04.347+00:00

Hi @Jamie Greeve ,

This is abnormal. What web stack are you using? What are you deploying? How are you deploying?

Regards,
Ryan
JimmyYang-MSFT 58,631 Reputation points Microsoft External Staff

2021-04-08T01:27:27.383+00:00

@Jamie Greeve

According to your description, your issue is not in Teams for Linux scope. So I will remove teams-for-linux tag from your thread. Thanks for your understanding and patience!
Jamie Greeve 31 Reputation points

2021-04-08T07:01:12.833+00:00

Hi @Ryan Hill ,

Stack .Net
Major version .Net Core 3
Minor version .Net Core 3.1 (LTS)

We are deploying a small app service.

Deployment is done via Azure Devops, but the same happens if we directly deploy from Visual studio.

Regards,
Jamie
Ryan Hill 30,276 Reputation points Microsoft Employee Moderator

2021-04-08T12:30:50.687+00:00

What region are you deploying to? I'll try to repo to see if I get the same result.
Jamie Greeve 31 Reputation points

2021-04-08T12:32:10.437+00:00

In west Europe we are deploying this.
Ryan Hill 30,276 Reputation points Microsoft Employee Moderator

2021-04-10T03:26:25.247+00:00

Hi @Jamie Greeve ,

Sorry for the late reply. I tried to repro but my app measured up to 35.21MiB (pictured below) on an S1 plan. When you browse the temp storage, what folder is showing a lot of space? What do the contents look like?
Jamie Greeve 31 Reputation points

2021-04-13T05:42:55.613+00:00

Hi @Ryan Hill ,

Thanks for your reply, sorry for my late reply.

We are looking in different metrics. This is at the best practices of the webapp. The metrics you showed are indeed almost the same, and the issue is not there.

If we go to the app service, then the 'Diagnose and solve problems'. In the app diagnostic you can search for 'Best Practices for Availability & Performance'. In here is an item 'Temp File Usage On Workers'. This metric shows this huge amount of storage while the webapp is just started. We can not see this metrics anywhere else, but after a while this starts growing. Do you have any idea?

Answer 1

Rutger Storm 6

Hi @Lukasz Andrzejak we ended up working together with Microsoft/Azure Support in identifying what caused the temp files with us to fill up. In the end for us it was the ApplicationInsights profiler which a developer added and forgot to remove when debugging a problem in the past. This caused the temp folder to fill up with dumps etc. We also tuned our app service plans better so they have a little bit more breathing room.

Lukasz Andrzejak 126 Reputation points

2021-09-08T12:10:26.767+00:00

Thank you @Rutger Storm !

Answer 2

Ryan Hill 30,276 Microsoft Employee Moderator

Hi @Jamie Greeve ,

I've been able to confirm that the high temp file usage is expected on Linux app service plans. It contains the blessed docker images of the various supported runtimes.

The Linux - Host Disk Space Usage elaborates on this a bit further and provides insights on what to do should your usage increase.

Regards,
Ryan

Jamie Greeve 31 Reputation points

2021-04-15T09:17:14.113+00:00

Hi @Ryan Hill ,

Okay thanks for looking into, and where can we find the Linux - Host Disk Space Usage?

And is it normal that this storage grows?

We had an issue with one of our app services, that this Disk space was filling up without any usage.

Regards,
Jamie
Ryan Hill 30,276 Reputation points Microsoft Employee Moderator

2021-04-15T14:26:15.83+00:00
It's under Diagnose and solve problems blade.

You will see two different lines

/mnt/data-diskUsage

/mnt/data/docker/images-diskUsage

/mnt/data/docker/images is the default docker storage mount for your docker images and can increase as you add more apps under the app service plan. /mnt/data is used for monitoring and shouldn't be an impact. If you do happen to see that usage steadily increasing, let me know down below.
Rutger Storm 6 Reputation points

2021-04-20T13:45:43.463+00:00

Hi @Ryan Hill

We ran into the problem again today. It seems "/mnt/data/docker/images-diskUsage" is growing. Any ideas how we can find out why? We haven't added any app services under the serviceplan. What we do in our CD process is we first deploy to a blue slot and then swap. Can that have anything to do with this?

And here are the screenshots of the app service.
Ryan Hill 30,276 Reputation points Microsoft Employee Moderator

2021-04-20T13:57:47.743+00:00

Hi @Rutger Storm please respond the private message below so we can work more closely with you.
Lukasz Andrzejak 126 Reputation points

2021-09-08T08:57:51.233+00:00

Hi,
we are experiencing this same problem with our linux webapp container.

Is there a solution available to clear these ?

Currently using up 30gb of temp files, but this will fill up soon, as it does not seem to be cleared.
Scaling up our app service plan would buy us time, but is not a solution.
Restarting the webapp does not clear these.

We are using a CI/CD build & release pipeline via azure devops, the images are updated frequently as this is a project under active development.

All of the available tutorials refer to windows solutions (d:\temp), accessing and clearing these under linux appears impossible (even the video tutorial shows a different kudu then what is available to us).

Share via

Azure App Service LInux temp storage 24gB

2 answers

Your answer