Getting "No Space left on device" error on azure while using ZipDeploy + Bitbucket pipeline

Question

We have an azure Linux App Service Plan with multiple app services. For the specific app services in question, we are using Bitbucket Pipeline to deploy the code to azure using the microsoft/azure-web-apps-deploy:1.0.0. Note: this is happening to two app services and both are deployed builds using the same pipe. Other app services using docker images for example are working fine.

Everything was working fine until suddenly pipelines started failing after taking around 20 mins running the deploy pipe. It freezes specifically at this command

az webapp deployment source config-zip --resource-group $RESOURCE_GROUP --name $APP_NAME --src example-295.zip

Pipeline step.

- step:
          name: "Deploy to Azure"
          deployment: production
          script:
            - pipe: microsoft/azure-web-apps-deploy:1.0.0
              variables:
                AZURE_APP_ID: $AZURE_APP_ID
                AZURE_PASSWORD: $AZURE_PASSWORD
                AZURE_TENANT_ID: $AZURE_TENANT_ID
                AZURE_RESOURCE_GROUP: $RESOURCE_GROUP
                AZURE_APP_NAME: $BAKIER_APP_NAME
                ZIP_FILE: "example-$BITBUCKET_BUILD_NUMBER.zip"

After digging into the kudu logs I was able to find the deployment error trace below

2024-07-21T08:03:07    Error occurred, type: error, text: No space left on device : '/tmp/zipdeploy', stackTrace:    at System.IO.FileSystem.CreateDirectory(String )
   at System.IO.Directory.CreateDirectory(String )
   at System.IO.Abstractions.DirectoryWrapper.CreateDirectory(String path)
   at Kudu.Core.Infrastructure.FileSystemHelpers.CreateDirectory(String path) in /tmp/KuduLite/Kudu.Core/Infrastructure/FileSystemHelpers.cs:line 42
   at Kudu.Core.Environment.get_ZipTempPath() in /tmp/KuduLite/Kudu.Core/Environment.cs:line 292
   at Kudu.Core.Deployment.ArtifactDeploymentInfo.GetRepository() in /tmp/KuduLite/Kudu.Core/Deployment/ArtifactDeploymentInfo.cs:line 24
   at Kudu.Core.Deployment.FetchDeploymentManager.GetDeploymentTracer(DeploymentInfoBase deploymentInfo, ITracer tracer) in /tmp/KuduLite/Kudu.Core/Deployment/FetchDeploymentManager.cs:line 574
   at Kudu.Services.Deployment.PushDeploymentController.PushDeployAsync(ArtifactDeploymentInfo deploymentInfo, Boolean isAsync, HttpContext context, JObject requestJson) in /tmp/KuduLite/Kudu.Services/Deployment/PushDeploymentController.cs:line 665
   at Kudu.Services.Deployment.PushDeploymentController.ZipPushDeploy() in /tmp/KuduLite/Kudu.Services/Deployment/PushDeploymentController.cs:line 157
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
   at Kudu.Services.Web.Tracing.TraceMiddleware.Invoke(HttpContext context) in /tmp/KuduLite/Kudu.Services.Web/Tracing/TraceMiddleware.cs:line 97
2024-07-21T08:03:07      Outgoing response, type: response, statusCode: 400, statusText: BadRequest

What I have tried without success:

1. Run older pipeline to make sure that it's not a recent update that caused this which also failed
2. Restart the app services in question.
3. Change app service plan and change it back again

I am not an expert with Azure but I have been at it for a while without success. Any help or direction to look into is much appreciated.

Answer 1

Mohamed Maher, Apologies for the delayed response from over the weekend.

I understand you have tried multiple things to isolate the issue.

A.File system storage: This is the file system storage that is included in the App Service Plan (ASP) quota, it is a persistent storage for your Linux Web apps apps under /home.

B.Host disk space or Temporary space: It is an assigned disk space for storing container images managed by the platform and custom containers deployed. This is a different storage from the File system storage and it is limited in size.

Kindly try the following steps to further isolate the issue.

1.Avoid writing outside of /home or custom Mounted storage - f your application is writing files to disk, you can redirect those files to /home or any path mapping using custom storage.

For custom containers you need to enable the home storage with this appsetting WEBSITES_ENABLE_APP_SERVICE_STORAGE=true or centralize the logs in a database or use any telemetry tool as Application insights.

2.Reducing image size if a smaller base image can be used .

3.Use .dockerignore - To ensure that any local artifacts aren’t copied into the container image.

4.Leverage multi-stage builds -You may use multi-stage and targets to match the right environment. Avoid having one big build stage when attempting to clean up sensitive data from it or dangling dependencies.

Kindly let us know how it goes, I'll follow-up with you further.

Ref: Azure App Service on Linux FAQ

---

If the answer helped (pointed, you in the right direction) > please click Accept Answer to benefit the community find answers quickly to similar question.