Azure infrastructure bug: automatic restart of App Service without graceful shutdown

Question

Azure infrastructure bug: automatic restart of App Service without graceful shutdown

Mo Becker 11

We are running an Azure App Service (Single Instance, S1, Windows, 64 bit, .NET 6, Always On enabled, Health Check disabled, Auto Heal disabled) where it is extremely important for data integrity that the service only terminates after a graceful shutdown. The graceful shutdown is implemented via IHostApplicationLifetime.ApplicationStopping, which is working fine (e.g. when the stop button is pressed in the Azure portal).

However, our log files (as well as the eventLog.xml) show that on 1/27/2023 at 3:25 UTC a new instance of the application was started, but the old instance was terminated without a graceful shutdown. In fact, the log file shows that the callback registered with ApplicationStopping was not entered at all, so it looks like the process was just abruptly killed.

The "Diagnose and solve problems - Web App Restarted" tool showed the following information:

Around 1/27/2023 3:25:23 AM (UTC), on Instance DW1SDWK0008DH, your application was recycled as there was a movement of your application from an existing worker instance because Azure platform detected that the worker instance was not working as expected. This resulted in your application being provisioned and started on a new VM instance.

All other health values during the time were normal, and there was nothing in the log suggesting that the application was not "working as expected". [Edit: I have now observed the same behavior due to "normal" Azure infrastructure maintenance/update events.]

1. Is it a bug in the Azure App Service infrastructure that the ApplicationStopping callback was not called on the instance to be replaced?
If graceful shutdown cannot be guaranteed under all circumstances, App Services may not be the right solution for our application. In that case, is there something more suitable?

Mo Becker 11 Reputation points

2023-02-23T15:37:11.0333333+00:00

I have now observed the same behavior (process abruptly killed) when Azure recycled the application because of infrastructure maintenance events.
SnehaAgrawal-MSFT 22,706 Reputation points Moderator

2023-02-23T17:04:30.33+00:00

Thanks for reaching here! To elaborate on

Around 1/27/2023 3:25:23 AM (UTC), on Instance DW1SDWK0008DH, your application was recycled as there was a movement of your appliction from an existing worker instance because Azure platform detected that the worker instance was not working as expected. This resulted in your application being provisioned and started on a new VM instance.

Worker process recycling is the replacing of the instance of the application in memory. Can automatically recycle worker processes by restarting the worker process, or worker processes, that are assigned to an application pool. This helps keep problematic applications running smoothly, and minimizes problems such as memory leaks.
You may want to know that on App service, Apps run inside of low-privileged worker processes using a random application pool identity.
Application code has access to the memory space associated with the worker process, as well as any child processes that may be spawned by CGI processes or other applications.

Also, if you have an application pool with applications that depend on state data, you must decide whether or not to recycle the worker processes that are assigned to that application pool. When you recycle a worker process, the state data for applications that is maintained in process is lost. In this case, do not use recycling.

For Recycle and Reboot insights from App Service diagnostics see Azure App Service Web App troubleshooting blade and tools

More Details:

https://azure.github.io/AppService/2021/09/16/App-Service-Restarts.html

Operating system functionality on Azure App Service

Let us know.
Mo Becker 11 Reputation points

2023-02-24T07:43:12.59+00:00

@SnehaAgrawal-MSFT My question is not why the recycling happened but why no proper shutdown is initiated when the recycling is caused by e.g. an infrastructure update.
Lex Li (Microsoft) 6,037 Reputation points Microsoft Employee

2023-03-02T06:25:32.9433333+00:00

From the good old days of Windows Server, I don't think ever web apps on IIS should expect "graceful shutdown" or "proper shutdown". While you can wish application pool recycle/idle shutdown might give the worker process some time to exit, the process can also be brutally killed for a lot of reasons (recovery from web app crash/hang, power-off, server crash, and etc.). The same applies to the cloud which does not fundamentally change how worker processes are used, and experienced web app developers know how to choose an elegant design that works in all possible cases.

Your answer

Mo Becker 11 Reputation points

2023-02-23T15:37:11.0333333+00:00

I have now observed the same behavior (process abruptly killed) when Azure recycled the application because of infrastructure maintenance events.
SnehaAgrawal-MSFT 22,706 Reputation points Moderator

2023-02-23T17:04:30.33+00:00

Thanks for reaching here! To elaborate on

Around 1/27/2023 3:25:23 AM (UTC), on Instance DW1SDWK0008DH, your application was recycled as there was a movement of your appliction from an existing worker instance because Azure platform detected that the worker instance was not working as expected. This resulted in your application being provisioned and started on a new VM instance.

Worker process recycling is the replacing of the instance of the application in memory. Can automatically recycle worker processes by restarting the worker process, or worker processes, that are assigned to an application pool. This helps keep problematic applications running smoothly, and minimizes problems such as memory leaks.
You may want to know that on App service, Apps run inside of low-privileged worker processes using a random application pool identity.
Application code has access to the memory space associated with the worker process, as well as any child processes that may be spawned by CGI processes or other applications.

Also, if you have an application pool with applications that depend on state data, you must decide whether or not to recycle the worker processes that are assigned to that application pool. When you recycle a worker process, the state data for applications that is maintained in process is lost. In this case, do not use recycling.

For Recycle and Reboot insights from App Service diagnostics see Azure App Service Web App troubleshooting blade and tools

More Details:

https://azure.github.io/AppService/2021/09/16/App-Service-Restarts.html

Operating system functionality on Azure App Service

Let us know.
Mo Becker 11 Reputation points

2023-02-24T07:43:12.59+00:00

@SnehaAgrawal-MSFT My question is not why the recycling happened but why no proper shutdown is initiated when the recycling is caused by e.g. an infrastructure update.
Lex Li (Microsoft) 6,037 Reputation points Microsoft Employee

2023-03-02T06:25:32.9433333+00:00

From the good old days of Windows Server, I don't think ever web apps on IIS should expect "graceful shutdown" or "proper shutdown". While you can wish application pool recycle/idle shutdown might give the worker process some time to exit, the process can also be brutally killed for a lot of reasons (recovery from web app crash/hang, power-off, server crash, and etc.). The same applies to the cloud which does not fundamentally change how worker processes are used, and experienced web app developers know how to choose an elegant design that works in all possible cases.

Share via

Azure infrastructure bug: automatic restart of App Service without graceful shutdown

Your answer