question

StefanN-5644 avatar image
0 Votes"
StefanN-5644 asked VenkateshDodda-MSFT commented

Actual disk space requirements for Azure AppService docker containers

Hi,
We started to run into problems with disk space on our multi-container Azure AppService instance. We have two slots (staging/prod) on an app with 5 docker containers each (rails, rails, node, ngnix, redis) and started to get errors during startup due to no space left for docker images. The «Linux - Host Disk Space Usage» reports near 100% usage.

Here https://github.com/projectkudu/kudu/wiki/Understanding-the-Azure-App-Service-file-system I read, that depending on the App Service Plan the quota varies. But I fear this information is outdated, as we are on a P2V3 plan and seem to still hit a limit of 15 GB.

Can you give us an update on the current quotas with the up-to-date PxV3 plans? Also, once we hit such a limit, what is the quickest way to clean up the disk space, as the service typically starts and endless restart loop that is not quite easy to stop.

Thanks for your help!

azure-webapps
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

VenkateshDodda-MSFT avatar image
1 Vote"
VenkateshDodda-MSFT answered VenkateshDodda-MSFT commented

@StefanN-5644, Thanks for reaching out. As per the documentation if you are using the P2V3 app service plan they you will get 250GB file system storage as shown below 197014-microsoftteams-image.png

App service on Linux offers two types of storage:
1.File System Storage: It's used when files are saved to the persistent storage that's rooted in the /home directory.
2.Host disk space: The host disk space is used to store container images on worker. It's managed by the platform through the docker storage driver. It's not expandable and there is a 15 GB limit for each instance.

If the container's writable layer saves data outside of the /home directory or a mounted azure storage path, the host disk space will also be consumed. If the container writes a large quantity of data to other /home directory, then you may land up with runtime exceptions once the host disk limit is exceeded.

It's recommended that you keep your container images as small as possible and write data to the persistent storage or BYOS(Bring your own storage) when running on Linux App Service. If not possible, you must split the App Service plan because the host disk space is fixed and shared between all containers in the App Service Plan.


· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

The Github documentation you are referring to is more about the persistent files and Temporary files in the app service file system. Feel free to reach back to me if you have any questions.

0 Votes 0 ·

Thank you for the swift answer!
I am still a bit confused. I use «Linux - Host Disk Space Usage» in the Diagnostics section of my App Service as reference. When this goes to 100%, the builds fail. Currently it is at 79%. And this corresponds with the file system usage on both my staging and my production slot:

/app# df -h
Filesystem Size Used Avail Use% Mounted on
none 50G 38G 11G 79% /
...

My main concern is: what can we do to not run into deployment issues due to lack of space. For this to understand, I would need to know:

a) what is the limit (15 GB or 50 GB)

b) how can we keep track on what we consume to keep this in checks

We have currently 2 slots x 4 docker images (Rails: 2.6 GB, Worker: 2.6 GB, NodeJs: 700MB, Nginx: 20MB) and I would like to roughly understand the limits. We are already working on reducing the two 2.6 GB docker images to ~700MB.. We have only minimal tmp/log files and disk space does not vary over time,

It would be helpful to get your feedback on a) and b). Thank you!

0 Votes 0 ·

Just a short update @VenkateshDodda-MSFT

We have now reduced the 2 x 2.6GB images for Rails and Worker to 700MB each. So with the two slots (prod / stage) we went from 2*(2.6+2.6+0.7+0.02) = 11.9GB to 2*(0.7+0.7+0.7+0.02) = 4.2GB. A massive reduction. But our reported «Linux - Host Disk Space Usage» went from 79% to 75%.

We are confused, to say the least.

0 Votes 0 ·

Adding our offline discussion to benefit the other community members.

Host disk space: It is the space used by the container to write files in addition to the read-only image layers. We call this writable layer. These could be log files generated by the container or other files being written to container by customers application.


How the Host Disk Space is calculated : docker ps output provides this information.
For ex, In below docker ps consolidated output 118MB is the disk space consumed by the running container (Writable Layer). 360MB is the total disk space (writable layer + read-only image layer) consumed by the container.


a) what is the limit (15 GB or 50 GB)
Ans: The limit of the host diskspace is 15GB irrespective of the SKU on which your webapp is hosted on.


b) how can we keep track of what we consume to keep this in check
Ans: Currently we don’t have any mechanism to track the amount of host disk space consumed apart from the Diagnose and solve problems (under Availability and Performance  Host disk space Usage). We see there is already feature request is raised for this feature.

I would suggest you UpVote and Comment on that. The product group monitors this site for feedback.

I have also shared the product feedback internally to our team as well.

0 Votes 0 ·