Deploy by Composer fail for missing space

Question

Deploy by Composer fail for missing space

Daniele Somma 21

Hi,
my BoT project was built using Bot Framework Composer, also used to deploy it.
In the last days, when I try to deploy updates to my bot, the process fails with this exception message: "There is not enough space on the disk."
The full error is this:

500 - "{\"Message\":\"An error has occurred.\",\"ExceptionMessage\":\"There is not enough space on the disk.\\r\\n\",\"ExceptionType\":\"System.IO.IOException\",\"StackTrace\":\" at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)\\r\\n at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)\\r\\n at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize)\\r\\n at System.IO.Abstractions.FileWrapper.Create(String path)\\r\\n at Kudu.Core.Infrastructure.FileSystemHelpers.CreateFile(String path) in C:\\\\Kudu Files\\\\Private\\\\src\\\\master\\\\Kudu.Core\\\\Infrastructure\\\\FileSystemHelpers.cs:line 27\\r\\n at Kudu.Services.ByteRanges.HttpRequestMessageExtensions.<CopyToAsync>d__1.MoveNext() in C:\\\\Kudu Files\\\\Private\\\\src\\\\master\\\\Kudu.Services\\\\ByteRanges\\\\HttpRequestMessageExtensions.cs:line 50\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)\\r\\n at Kudu.Services.Deployment.PushDeploymentController.<PushDeployAsync>d__15.MoveNext() in C:\\\\Kudu Files\\\\Private\\\\src\\\\master\\\\Kudu.Services\\\\Deployment\\\\PushDeploymentController.cs:line 423\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n at Kudu.Services.Deployment.PushDeploymentController.<ZipPushDeploy>d__9.MoveNext() in C:\\\\Kudu Files\\\\Private\\\\src\\\\master\\\\Kudu.Services\\\\Deployment\\\\PushDeploymentController.cs:line 98\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n at System.Threading.Tasks.TaskHelpersExtensions.<CastToObject>d__3`1.MoveNext()\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__0.MoveNext()\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__5.MoveNext()\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__5.MoveNext()\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__0.MoveNext()\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__5.MoveNext()\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__5.MoveNext()\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__0.MoveNext()\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__2.MoveNext()\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n at System.Web.Http.Controllers.ExceptionFilterResult.<ExecuteAsync>d__0.MoveNext()\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n at System.Web.Http.Controllers.ExceptionFilterResult.<ExecuteAsync>d__0.MoveNext()\\r\\n--- End of stack trace from previous location where exception was thrown ---\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\\r\\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\\r\\n at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__1.MoveNext()\"}"

My last changes consist only into updated my BoT project, after I update the Bot Framework Composer, passing to version 1.2.0.

Looking the app services, seems there are no problems with the space, also because my project occupy very few space in this moment.
Wich could be the issue?

Thanks in advanced for any help

romungi-MSFT 48,906 Reputation points Microsoft Employee Moderator

2020-12-01T14:24:28.687+00:00

@Daniele Somma The error message seems to be from the app service/plan with insufficient space. Does a restart on the app service from Azure portal help to re-deploy the project?
Daniele Somma 21 Reputation points

2020-12-01T16:23:17.93+00:00

@romungi-MSFT thanks for your reply. I have done the following steps:
1. restart the App Service
2. redeploy the app => works
3. redeploy again the app => fails
4. changed the service plan from free to the next level
5. redeploy the app => works
6. redeploy the app => fails

I checked inside "Quota", press link "File system usage" and checked the value, I have 1% occupied.
I don't understand which is the real problem and if there is a way to clean the "under the hood" of the AppService.

Thanks in advance for any help.

1 answer

Your answer

romungi-MSFT 48,906 Reputation points Microsoft Employee Moderator

2020-12-01T14:24:28.687+00:00

@Daniele Somma The error message seems to be from the app service/plan with insufficient space. Does a restart on the app service from Azure portal help to re-deploy the project?
Daniele Somma 21 Reputation points

2020-12-01T16:23:17.93+00:00

@romungi-MSFT thanks for your reply. I have done the following steps:
1. restart the App Service
2. redeploy the app => works
3. redeploy again the app => fails
4. changed the service plan from free to the next level
5. redeploy the app => works
6. redeploy the app => fails

I checked inside "Quota", press link "File system usage" and checked the value, I have 1% occupied.
I don't understand which is the real problem and if there is a way to clean the "under the hood" of the AppService.

Thanks in advance for any help.

Answer 1

Ryan Hill 30,281 Microsoft Employee Moderator

Hi @Daniele Somma Did you by chance use the following guide https://learn.microsoft.com/en-us/composer/how-to-publish-bot to publish your bot? Although you've checked your Quota and File system usage, check the usage on your app service directly by opening a console/ssh and running ls -la against D:\home\data\sitepackages as well as traversing D:\local directory. Compare this to what your System Info is reporting for your Kudu environment.

It could be that one of the dependencies your bot is using consuming a fair amount of space as it gets deployed. If there aren't any dependencies that can be optimized, you may try moving up to S1, you indicated you moved from Free to next level which would be Basic, to increase the amount of space available to D:\local.

Regards,
Ryan

Daniele Somma 21 Reputation points

2020-12-01T22:06:30.373+00:00

Hi, @Ryan Hill thanks for your reply.
Yes, I used that guide the first time when I created the project and resources. I published for months using the Composer.
This are the results of the execution of your suggested command:

My Bot at the moment is very simple, few dialogs, one connection with a QnA and LUIS, nothing else. I haven't dependencies.

About the size, to test, I switched from F1 to D1 and switch back. I had and I have the issue with both.
Any suggestion?

Regards
Daniele
Ryan Hill 30,281 Reputation points Microsoft Employee Moderator

2020-12-02T01:41:22.353+00:00

Hi @Daniele Somma , what results do you get for ls -la under D:\home\data\sitepackages directory? Furthermore, do you still get the same result if you scale up to an S1 (standard) plan?

Regards,
Ryan
Daniele Somma 21 Reputation points

2020-12-02T10:12:01.38+00:00
Hi @Ryan Hill , I haven't the folder sitepackages, as you can see in the first image of the previous post. Under data folder I have only three folders:

aspnet

kudusync

monaco

I tried switching to B1 plan with 10Gb of storage. After the switch, I have done five deploys with success and now the application deployed is updated, with the last changes done. A secondary defect with the primary problem is the application is not up to date, also when the deployment finishes with success. Looking at quotas for file system storage after the upgrade, I have this situation:

I can't maintain B1 plan and I switched back to F1. After the switch, I have done one deploy with success, the second fails with the same issue. Looking at quotas for file system storage after the downgrade, I have this situation:

Any ideas?

Regards
Daniele
Ryan Hill 30,281 Reputation points Microsoft Employee Moderator

2020-12-02T21:37:00.747+00:00

Hi @Daniele Somma ,

If the scale up worked for a while, the I'm thinking you're hitting the 500MB limitation on Free and Shared tiers. See https://github.com/projectkudu/kudu/wiki/Understanding-the-Azure-App-Service-file-system#temporary-files. Can you go to D:\local and run ls -la? I noticed in your screen shot you did it against C:. Your app contents doesn't deploy to that drive and is locked down.

Regard,
Ryan
Daniele Somma 21 Reputation points

2020-12-03T11:39:35.54+00:00

Hi @Ryan Hill ,
I read the article but I haven't completely understood the point.
Looking at the space used from my application, could be possible, using the free plan, that the remaining space is too less to execute the "unpackaging" of the new app when I run the deploy process?
About your request, I notice you speak always of D: drive. Under that unit, I have folders without the grants to looking inside or analyze.
I execute the command under "C:\local", with these results:

At the moment, the only workaround I thought to fix the problem is to scale-up the plan just before to do the deployment and later the scale down. It's not nice but could work.

Thanks for your help.

Regards
Daniele

Share via

Deploy by Composer fail for missing space

1 answer

Your answer