How to switch deployment slot without any downtime

Question

How to switch deployment slot without any downtime

Wallyson Romeiro 5

I implemented the swap for some web apps and noticed that it sometimes fails. Sometimes the swap works normally, but other times it fails, always generating the error below:

"Unexpected error when polling worker for change notification status: System.AggregateException: One or more errors occurred. ---> System.Threading.Tasks.TaskCanceledException: A task was canceled. --- End of inner exception stack trace --- at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification) at Microsoft.Web.Hosting.WorkerManagement.Client.LinuxWorkerManagementServiceClient.GetActiveConfigurationTimeStampForSite(String siteName) at Microsoft.Web.Hosting.Administration.Utilities.SiteWarmer.d__85.MoveNext() in C:__w\1\s\src\Hosting\AdministrationService\Microsoft.Web.Hosting.Administration\Utilities\SiteWarmer.cs

652 ---> (Inner Exception #0) System.Threading.Tasks.TaskCanceledException: A task was canceled.<---"

I would like to understand what could be causing this error, as I followed best practices for creating slots and performing the swap, and now I receive these errors which cause downtime in the application.

SnehaAgrawal-MSFT 22,721 Reputation points Moderator

2024-07-23T10:45:38.14+00:00
@Wallyson Romeiro Thanks for asking question!

The error message you are seeing indicates that the swap operation failed due to a timeout when polling the worker for change notification status. This can happen if the worker process is taking too long to respond or if there is a network issue between the worker and the management service.

Here are some possible reasons why the swap operation is failing and how you can fix the issue:

Worker process: Check that the worker process is running correctly and is not experiencing any issues. You can check the logs for the worker process to see if there are any error messages that can help you diagnose the issue.

Network issues: Check that there are no network issues between the worker and the management service. You can use network diagnostic tools to check for connectivity issues and latency.

Timeout settings: Check that the timeout settings for the swap operation are set up correctly. You can adjust the timeout settings using the WEBSITE_SWAP_WARMUP_PING_PATH and WEBSITE_SWAP_WARMUP_PING_TIMEOUT app settings. - https://learn.microsoft.com/en-us/azure/app-service/deploy-staging-slots?tabs=portal#specify-custom-warm-up

Load balancing: Check that the load balancing settings for the web app are set up correctly. If the web app is using a load balancer, make sure that the load balancer is configured correctly and is not causing issues with the swap operation.

I recommend checking these possible reasons to identify the root cause of the issue.

If you are still experiencing issues, you can check the logs for your web app to see if there are any error messages that can help you diagnose the issue.

Check these links-

Configure swap with preview: https://docs.microsoft.com/en-us/azure/app-service/deploy-staging-slots#configure-swap-with-preview

Troubleshoot swap operation issues: https://docs.microsoft.com/en-us/azure/app-service/deploy-staging-slots#troubleshoot-swap-operation-issues

Let us know if issue remains.
Wallyson Romeiro 5 Reputation points

2024-07-23T14:42:37+00:00

Hi @SnehaAgrawal-MSFT

Thank you for your response.

I have already made the recommended configurations you mentioned, but I still receive the error. However, if the App Service is a managed service, shouldn’t these worker failures be handled by the product team?

For me, this error is random; it happens sometimes and other times it doesn't.

In the following documentation: https://learn.microsoft.com/en-us/azure/app-service/deploy-staging-slots?tabs=portal#specify-custom-warm-up
it does not mention the WEBSITE_SWAP_WARMUP_PING_TIMEOUT.
SnehaAgrawal-MSFT 22,721 Reputation points Moderator

2024-07-26T10:34:35+00:00

@Wallyson Romeiro The instances in your former production instances (those that will be swapped into staging after this swap operation) will be recycled quickly in the last step of the swap process.

In case you have any long running operations in your application, they will be abandoned, when the workers recycle. Ensure your application code should be written in a fault tolerant way.
Wallyson Romeiro 5 Reputation points

2024-08-01T20:19:02.6+00:00

@SnehaAgrawal-MSFT Got it, However, I want to highlight an issue with the current swap process. When I restart or deploy the app, it comes back up in less than 1 minute. Yet, during the swap process, it takes around 10 minutes for the application to become responsive again. This delay is significantly longer and impacts the application's availability.

Additionally, the only error observed in the diagnostics stage is:

"Unexpected error when polling worker for change notification status: System.AggregateException: One or more errors occurred. ---> System.Threading.Tasks.TaskCanceledException: A task was canceled. --- End of inner exception stack trace --- at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification) at Microsoft.Web.Hosting.WorkerManagement.Client.LinuxWorkerManagementServiceClient.GetActiveConfigurationTimeStampForSite(String siteName) at Microsoft.Web.Hosting.Administration.Utilities.SiteWarmer.d__85.MoveNext() in C:__w\1\s\src\Hosting\AdministrationService\Microsoft.Web.Hosting.Administration\Utilities\SiteWarmer.cs 652 ---> (Inner Exception #0) System.Threading.Tasks.TaskCanceledException: A task was canceled.<---"

This error suggests that there might be underlying issues during the swap process that are causing extended downtimes. Considering this, it's not just about making the application fault-tolerant; we also need to address why the swap process is significantly longer compared to a regular restart or deployment. Reducing the swap time would mitigate the impact on long-running operations and improve overall application availability during the swap.
SnehaAgrawal-MSFT 22,721 Reputation points Moderator

2024-08-06T18:33:01.4833333+00:00

@Wallyson Romeiro Thanks for reply!

Some apps might require custom warm-up actions before the swap. The applicationInitialization configuration element in web.config lets you specify custom initialization actions. The swap operation waits for this custom warm-up to finish before swapping with the target slot.

During the swap operation the Web App’s worker process may get restarted in order for some settings to take effect. Even though the swap does not proceed until the restarted worker process comes back online on every VM instance, it may still not be enough for application to be completely ready to take on production traffic.

Try enabling Application Initialization Module to completely warm up your application prior to swapping it into production.

More detailed explanation of that process is available at How to warm up Azure Web App during deployment slots swap

To add to this if the swap operation takes a long time to complete, you can also get information on the swap operation in the activity log. On your app's resource page in the portal, in the left pane, select Activity log. A swap operation appears in the log query as Swap Web App Slots. You can expand it and select one of the sub operations or errors to see the details.

Please refer to below links for more details on this:

https://ruslany.net/2017/11/most-common-deployment-slot-swap-failures-and-how-to-fix-them/

https://ruslany.net/2019/06/azure-app-service-deployment-slots-tips-and-tricks/