Azure App Service : deployment using pipelines: EnvironmentError: [Errno 28] No space left on device

Question

Azure App Service : deployment using pipelines: EnvironmentError: [Errno 28] No space left on device

Abdelkrim 106

Introduction

Hi, I am deploying a Python-Flask application on the Azure App Service for months.

Today I got the following error message while running my pipeline running on Microsoft DevOps:

ERROR: Could not install packages due to an EnvironmentError: [Errno 28] No space left on device: '/tmp/8d876019152941b/antenv/lib/python3.8/site-packages/numpy-1.18.1.dist-info'

I got the error on other apps stored on the same Plan. All my apps can't be updated anymore

what did I try

I created a new app and deployed the same code.
I got the same error :

> Cleaning up temp folders from previous zip deployments and extracting pushed zip file /tmp/zipdeploy/REDACTED.zip (246.59 MB) to /tmp/zipdeploy/extracted

> No space left on device

> ##[error]Failed to deploy web package to App Service.

The plan states that my threshold quota is 10 Gb and that I use 936Mb or 9% of my quota !

Using 'df-h' by using Kundu Bash

&gt; Filesystem   Size   Used Avail Use%   Mounted on
&gt; none         29G    27G  0     100%   /
&gt; tmpfs        64M     0   64M   0%     /dev
&gt; tmpfs        953M     0  953M  0%     /sys/fs/cgroup
&gt; shm          62M     0   62M   0%     /dev/shm
&gt; //REDACTED/  1000G  9.7G 991G  1% /home
&gt; /dev/sda1    27G    8.6G 18G   34%    /appsvctmp
&gt; /dev/loop6p1 29G    27G  0     100%   /etc/hosts
&gt; udev 920M    0      920M       0%     /dev/tty
&gt; tmpfs 953M   0      953M       0%     /proc/acpi
&gt; tmpfs 953M   0      953M       0%     /proc/scsi
&gt; tmpfs 953M   0      953M       0%     /sys/firmware

The question

What should I do to work again?

I can't access the /dev/loop6p1 directory (what is it?) and delete files

the Azure pipelines logs

&gt; 2020-10-21 ##[section]Starting: Deploy Azure Web App : www.alt-f1.be
&gt; 2020-10-21 ==============================================================================
&gt; 2020-10-21 Task         : Azure Web App
&gt; 2020-10-21 Description  : Deploy an Azure Web App for Linux or Windows
&gt; 2020-10-21 Version      : 1.168.3
&gt; 2020-10-21 Author       : Microsoft Corporation
&gt; 2020-10-21 Help         : https://aka.ms/azurewebapptroubleshooting
&gt; 2020-10-21 ==============================================================================
&gt; 2020-10-21 Got service connection details for Azure App Service:&#39;www.alt-f1.be&#39;
&gt; 2020-10-21 Package deployment using ZIP Deploy initiated.
&gt; 2020-10-21 Updating submodules.
&gt; 2020-10-21 Preparing deployment for commit id &#39;1ea14e5515&#39;.
&gt; 2020-10-21 Repository path is /tmp/zipdeploy/extracted
&gt; 2020-10-21 Running oryx build...
&gt; 2020-10-21 Command: oryx build /tmp/zipdeploy/extracted -o /home/site/wwwroot --platform python --platform-version 3.8 -i /tmp/8d876019152941b -p compress_virtualenv=tar-gz -p virtualenv_name=antenv --log-file /tmp/build-debug.log 
&gt; 2020-10-21 Operation performed by Microsoft Oryx, https://github.com/Microsoft/Oryx
&gt; 2020-10-21 You can report issues at https://github.com/Microsoft/Oryx/issues
&gt; 2020-10-21 
&gt; 2020-10-21 Oryx Version: 0.2.20200917.1, Commit: 59deb778658a124cb74ea8e2c8f39fa87abcc9d9, ReleaseTagName: 20200917.1
&gt; 2020-10-21 
&gt; 2020-10-21 Build Operation ID: |18pOsx/PQ8Y=.d0555319_
&gt; 2020-10-21 Repository Commit : 1ea14e5515974a80bdcdc28eac749d4f
&gt; 2020-10-21 
&gt; 2020-10-21 Detecting platforms...
&gt; 2020-10-21 Detected following platforms:
&gt; 2020-10-21   nodejs: 12.18.3
&gt; 2020-10-21   python: 3.8.5
&gt; 2020-10-21 
&gt; 2020-10-21 Using intermediate directory &#39;/tmp/8d876019152941b&#39;.
&gt; 2020-10-21 
&gt; 2020-10-21 Copying files to the intermediate directory...
&gt; 2020-10-21 Done in 6 sec(s).
&gt; 2020-10-21 
&gt; 2020-10-21 Source directory     : /tmp/8d876019152941b
&gt; 2020-10-21 Destination directory: /home/site/wwwroot
&gt; 2020-10-21 
&gt; 2020-10-21 Python Version: /opt/python/3.8.5/bin/python3.8
&gt; 2020-10-21 Python Virtual Environment: antenv
&gt; 2020-10-21 Creating virtual environment...
&gt; 2020-10-21 Activating virtual environment...
&gt; 2020-10-21 Running pip install...
&gt; 2020-10-21 [20:41:33+0000] Collecting altf1be_helpers==2.1.4
&gt; 2020-10-21 [20:41:33+0000]   Downloading altf1be_helpers-2.1.4-py2.py3-none-any.whl (14 kB)
...

&gt; 2020-10-21 [20:44:13+0000] Installing collected packages: numpy, six, python-dateutil, pytz, pandas, certifi, urllib3, idna, chardet, requests, isodate, oauthlib, requests-oauthlib, msrest, azure-core, pycparser, cffi, cryptography, azure-storage-blob, Unidecode, altf1be-helpers, attrs, Babel, pyparsing, packaging, &gt; webencodings, bleach, cachelib, click, dash-core-components, dash-html-components, dash-table, MarkupSafe, Jinja2, Werkzeug, itsdangerous, Flask, brotli, Flask-Compress, retrying, plotly, dash-renderer, future, dash, docopt, docutils, Flask-Babel, Flask-Session, SQLAlchemy, Flask-SQLAlchemy, gunicorn, zipp, importlib-metadata, jeepney, &gt; SecretStorage, keyring, more-itertools, yarg, pipreqs, pkginfo, pluggy, py, Pygments, pyodbc, pyopenssl, wcwidth, pytest, python-dotenv, readme-renderer, requests-toolbelt, starkbank-ecdsa, python-http-client, sendgrid, tqdm, twine, WTForms, xlrd, XlsxWriter
&gt; 2020-10-21 ERROR: Could not install packages due to an EnvironmentError: [Errno 28] No space left on device: &#39;/tmp/8d876019152941b/antenv/lib/python3.8/site-packages/numpy-1.18.1.dist-info&#39;
&gt; 2020-10-21 
&gt; 2020-10-21 WARNING: You are using pip version 20.1.1; however, version 20.2.4 is available.
&gt; 2020-10-21 You should consider upgrading via the &#39;/tmp/8d876019152941b/antenv/bin/python -m pip install --upgrade pip&#39; command.
&gt; 2020-10-21 ERROR: Could not install packages due to an EnvironmentError: [Errno 28] No space left on device: &#39;/tmp/8d876019152941b/antenv/lib/python3.8/site-packages/numpy-1.18.1.dist-info&#39;\n\nWARNING: You are using pip version 20.1.1; however, version 20.2.4 is available.\nYou should consider upgrading via the &#39;/tmp/&gt; 8d876019152941b/antenv/bin/python -m pip install --upgrade pip&#39; command.\n/opt/Kudu/Scripts/starter.sh oryx build /tmp/zipdeploy/extracted -o /home/site/wwwroot --platform python --platform-version 3.8 -i /tmp/8d876019152941b -p compress_virtualenv=tar-gz -p virtualenv_name=antenv --log-file /tmp/build-debug.log 
&gt; 2020-10-21 ##[error]Failed to deploy web package to App Service.
&gt; 2020-10-21 ##[error]To debug further please check Kudu stack trace URL : https://$www.alt-f1.be:***@www.alt-f1.be.scm.azurewebsites.net/api/vfs/LogFiles/kudu/trace
&gt; 2020-10-21 ##[error]Error: Package deployment using ZIP Deploy failed. Refer logs for more details.
&gt; 2020-10-21 Successfully updated deployment History at https://www.alt-f1.be.scm.azurewebsites.net/api/deployments/2571603313079929
&gt; 2020-10-21 App Service Application URL: https://www.alt-f1.be.azurewebsites.net
&gt; 2020-10-21 ##[section]Finishing: Deploy Azure Web App : www.alt-f1.be

Conclusion

My plan is completely blocked. It is not possible to deploy any version of my code anymore.

2 answers

Your answer

Answer 1

Pratik 6

@Abdelkrim Did you got anything on this? Facing exact same issue and don't know the root cause and permanent solution.

As a temporary fix, I scale up my app service plan and the again scale down to original and it has cleaned space and all started working, but now again the used space in /dev/loop6p1 is increasing and I can't rely on my workaround for each time.

Thanks,
Pratik

SvenGlöckner 446 Reputation points

2021-11-19T09:47:52.977+00:00

Same problem here... Already in touch with MS support.
Shahzad Ali Kulachi 1 Reputation point

2021-11-22T16:05:35.887+00:00

I am facing same issue, any command on it,
Jaime Santos 11 Reputation points

2021-11-22T16:35:29.25+00:00

Hi Guys, did you get and answer about it? I'm facing the same issue
Jaime Santos 11 Reputation points

2021-11-22T16:36:10.27+00:00

Hi, did you get an answer from MS support?I'm facing same issue.
SvenGlöckner 446 Reputation points

2021-11-23T06:52:50.387+00:00

Yes, but it hasn't been very helpful.

My suggestion: “there should be some logic that deletes old images that are no longer needed". This is still under research.
Answer: "yes we totally agree with you ,we suggested this feature request with my engineering team but still In Progress."

Answer 2

Ryan Hill 30,281 Microsoft Employee Moderator

@Abdelkrim had several issues they rectified, see https://stackoverflow.com/questions/59373884/azure-container-generates-error-503-reason-site-unavailable. The following links may also prove useful in isolating your specific issue:

If the above link doesn't help @ShahzadAliKulachi-1894 @Jaime Santos , comment down below.

SvenGlöckner 446 Reputation points

2021-11-23T07:24:58.38+00:00

@Ryan Hill thanks for posting the links, however they are not very helpful.
I already came across them and the solutions are either not applicable or incomprehensible.

I'm already in touch with MS Azure support. However, they have no solution for at least 4 days now.
It seems there is no way to prune the docker image space on the hard disk. For me, this looks like a bad design of App Services; it appears rather useless for running CD pipelines.

Currently I'm trying a different approch. Instead of creating my own docker image, I'm using plain App Service for Linux and deploy the app runnables with my pipeline.
App Service for Linux uses already a pre-built docker image under the hood.

Share via

Azure App Service : deployment using pipelines: EnvironmentError: [Errno 28] No space left on device

Introduction

what did I try

Using 'df-h' by using Kundu Bash

The question

the Azure pipelines logs

Conclusion

2 answers

Your answer