Concurrent Swap?

Question

Concurrent Swap?

Mike-E-angelo 591

I currently have five swappable applications within my environment:

gateway/entry/www
content server
commerce server
worker processes (webjobs)
events server (signalr)

Each application takes 2 minutes to swap, with two managed agents, that is a 10-minute operation. The amount of time to deploy bothers me, but what concerns me most is the dependencies. As services are being swapped, I basically have services that are not sharing the same versioned code for 2-6 minutes at a time, depending.

The only way I can think of working around this is to offline.htm the gateway/www application, but this is incredibly disruptive for each deployment, especially if it takes 10 minutes to swap all applications.

I am wanting to reach out here for guidance/suggestions on how to best swap all applications in my environment concurrently.

Thank you for any assistance you can provide,
Michael

SwathiDhanwada-MSFT 18,996 Reputation points Moderator

2022-06-13T09:04:35.357+00:00

@Mike-E-angelo Can you please share which azure services you are using for your applications?
Mike-E-angelo 591 Reputation points

2022-06-13T09:22:16.083+00:00

Hi @SwathiDhanwada-MSFT thank you for your reply. These are all Azure App Service applications.
SwathiDhanwada-MSFT 18,996 Reputation points Moderator

2022-06-13T09:43:23.173+00:00

@Mike-E-angelo Thanks for sharing the details. I have added appropriate tag to the question, one of the engineers will check and respond to your query.
Mike-E-angelo 591 Reputation points

2022-06-13T09:50:51.623+00:00

Thank you @SwathiDhanwada-MSFT . I did try to use the tags, but they seem rather broken and I was not able to see any good suggestions. I appreciate you getting the right one here.
SnehaAgrawal-MSFT 22,706 Reputation points Moderator

2022-06-14T14:20:24.32+00:00

Thanks for reaching here! If I am understanding right you have multiple staging slots and you want to swap,
Check this blog on : Swap Slots in Azure Web Site when there are 2 or more staging slots using Azure PowerShell.

Further Adding some information here regarding Azure App service deployment slot swap might be helpful:

Some apps might require custom warm-up actions before the swap. The applicationInitialization configuration element in web.config lets you specify custom initialization actions. The swap operation waits for this custom warm-up to finish before swapping with the target slot.

During the swap operation the Web App’s worker process may get restarted in order for some settings to take effect. Even though the swap does not proceed until the restarted worker process comes back online on every VM instance, it may still not be enough for application to be completely ready to take on production traffic.

Try enabling Application Initialization Module to completely warm up your application prior to swapping it into production.

More detailed explanation of that process is available at How to warm up Azure Web App during deployment slots swap
SnehaAgrawal-MSFT 22,706 Reputation points Moderator

2022-06-14T14:23:37.767+00:00

Also, You may refer to official document on Which settings are swapped?

If the swap operation takes a long time to complete, you can also get information on the swap operation in the activity log. On your app's resource page in the portal, in the left pane, select Activity log. A swap operation appears in the log query as Swap Web App Slots. You can expand it and select one of the sub operations or errors to see the details.

Please refer to below links for more details on this:

https://learn.microsoft.com/en-us/azure/app-service/deploy-staging-slots#what-happens-during-a-swap
https://ruslany.net/2017/11/most-common-deployment-slot-swap-failures-and-how-to-fix-them/
https://ruslany.net/2019/06/azure-app-service-deployment-slots-tips-and-tricks/

Let us know.
Mike-E-angelo 591 Reputation points

2022-06-14T16:15:25.92+00:00
Thank you for the reply @SnehaAgrawal-MSFT . In my case I have 5 AppServices and 2 slots between them, so 10 slots total (and 5 swap operations).

Application A (Gateway):

Staging

Production

Application B (Commerce):

Staging

Production

Application C (Events):

Staging

Production

Application D (Content):

Staging

Production

Application E (Workers):

Staging

Production

I only have two Azure DevOps agents that can work at any given time on these swaps. So if each swap takes 2 minutes, and there are 2 agents for 5 jobs, that is about 6 minutes of total time that is taken. During this time some applications will be up to date, and others will have the older version. That means if requests are made to the older version there is a chance that operations will be performed with the old code that may result in unintended system state.

Essentially what I want to do is swap all applications at the same time (concurrently) so that they all result in the same version at the same time, and I do not have to worry about old code matching with new code.

I hope that makes sense. Please let me know if there is any further information you need to further assist.
SnehaAgrawal-MSFT 22,706 Reputation points Moderator

2022-06-15T11:34:00.473+00:00

Thanks for reply! If you are using Azure CI/CD pipeline and AzureAppServiceManage task for swapping?
Could you confirm if are using Linux/containers ?

Also suggest you check this blog might be helpful: Using ARM template to swap App Service deployment slots

Let us know.
Mike-E-angelo 591 Reputation points

2022-06-15T11:54:38.137+00:00

Thank you for providing the link, @SnehaAgrawal-MSFT . That appears to be a method to swap, and if I am not mistaken does not address the problem. While that ARM template appears to lock the version, if I have 5 applications to swap and only 2 agents to swap them with, I still run into the potential of having services running on different versions while the swaps are occurring.

So for example:

Agent 1: Swap Application A from v1.0 to 1.1
Agent 2: Swap Application B from v1.0 to 1.1
Application C: v1.0
Application D: v1.0
Application E: v1.0

While Agent 1 & 2 are swapping Applications A & B, if any calls are made to C, D, & E, they are done with v1.0 code and not v1.1 code (which may have additional/different functionality). I hope that further explains the situation here.

Additionally, I am indeed using this task, and all of my images/AppServices are Windows. Thank you for any further insight/suggestions you can provide.

1 answer

Your answer

SwathiDhanwada-MSFT 18,996 Reputation points Moderator

2022-06-13T09:04:35.357+00:00

@Mike-E-angelo Can you please share which azure services you are using for your applications?
Mike-E-angelo 591 Reputation points

2022-06-13T09:22:16.083+00:00

Hi @SwathiDhanwada-MSFT thank you for your reply. These are all Azure App Service applications.
SwathiDhanwada-MSFT 18,996 Reputation points Moderator

2022-06-13T09:43:23.173+00:00

@Mike-E-angelo Thanks for sharing the details. I have added appropriate tag to the question, one of the engineers will check and respond to your query.
Mike-E-angelo 591 Reputation points

2022-06-13T09:50:51.623+00:00

Thank you @SwathiDhanwada-MSFT . I did try to use the tags, but they seem rather broken and I was not able to see any good suggestions. I appreciate you getting the right one here.
SnehaAgrawal-MSFT 22,706 Reputation points Moderator

2022-06-14T14:20:24.32+00:00

Thanks for reaching here! If I am understanding right you have multiple staging slots and you want to swap,
Check this blog on : Swap Slots in Azure Web Site when there are 2 or more staging slots using Azure PowerShell.

Further Adding some information here regarding Azure App service deployment slot swap might be helpful:

Some apps might require custom warm-up actions before the swap. The applicationInitialization configuration element in web.config lets you specify custom initialization actions. The swap operation waits for this custom warm-up to finish before swapping with the target slot.

During the swap operation the Web App’s worker process may get restarted in order for some settings to take effect. Even though the swap does not proceed until the restarted worker process comes back online on every VM instance, it may still not be enough for application to be completely ready to take on production traffic.

Try enabling Application Initialization Module to completely warm up your application prior to swapping it into production.

More detailed explanation of that process is available at How to warm up Azure Web App during deployment slots swap
SnehaAgrawal-MSFT 22,706 Reputation points Moderator

2022-06-14T14:23:37.767+00:00

Also, You may refer to official document on Which settings are swapped?

If the swap operation takes a long time to complete, you can also get information on the swap operation in the activity log. On your app's resource page in the portal, in the left pane, select Activity log. A swap operation appears in the log query as Swap Web App Slots. You can expand it and select one of the sub operations or errors to see the details.

Please refer to below links for more details on this:

https://learn.microsoft.com/en-us/azure/app-service/deploy-staging-slots#what-happens-during-a-swap
https://ruslany.net/2017/11/most-common-deployment-slot-swap-failures-and-how-to-fix-them/
https://ruslany.net/2019/06/azure-app-service-deployment-slots-tips-and-tricks/

Let us know.
Mike-E-angelo 591 Reputation points

2022-06-14T16:15:25.92+00:00

Thank you for the reply @SnehaAgrawal-MSFT . In my case I have 5 AppServices and 2 slots between them, so 10 slots total (and 5 swap operations).

Application A (Gateway):

Staging

Production

Application B (Commerce):

Staging

Production

Application C (Events):

Staging

Production

Application D (Content):

Staging

Production

Application E (Workers):

Staging

Production

I only have two Azure DevOps agents that can work at any given time on these swaps. So if each swap takes 2 minutes, and there are 2 agents for 5 jobs, that is about 6 minutes of total time that is taken. During this time some applications will be up to date, and others will have the older version. That means if requests are made to the older version there is a chance that operations will be performed with the old code that may result in unintended system state.

Essentially what I want to do is swap all applications at the same time (concurrently) so that they all result in the same version at the same time, and I do not have to worry about old code matching with new code.

I hope that makes sense. Please let me know if there is any further information you need to further assist.
SnehaAgrawal-MSFT 22,706 Reputation points Moderator

2022-06-15T11:34:00.473+00:00

Thanks for reply! If you are using Azure CI/CD pipeline and AzureAppServiceManage task for swapping?
Could you confirm if are using Linux/containers ?

Also suggest you check this blog might be helpful: Using ARM template to swap App Service deployment slots

Let us know.
Mike-E-angelo 591 Reputation points

2022-06-15T11:54:38.137+00:00

Thank you for providing the link, @SnehaAgrawal-MSFT . That appears to be a method to swap, and if I am not mistaken does not address the problem. While that ARM template appears to lock the version, if I have 5 applications to swap and only 2 agents to swap them with, I still run into the potential of having services running on different versions while the swaps are occurring.

So for example:

Agent 1: Swap Application A from v1.0 to 1.1
Agent 2: Swap Application B from v1.0 to 1.1
Application C: v1.0
Application D: v1.0
Application E: v1.0

While Agent 1 & 2 are swapping Applications A & B, if any calls are made to C, D, & E, they are done with v1.0 code and not v1.1 code (which may have additional/different functionality). I hope that further explains the situation here.

Additionally, I am indeed using this task, and all of my images/AppServices are Windows. Thank you for any further insight/suggestions you can provide.

Answer 1

I wanted to update this question with an answer as I have finally landed on a path after spending several weeks with Azure Advisors on my Microsoft for Startups plan.

Looks like what I need is blue-green deployments:
https://learn.microsoft.com/en-us/azure/spring-apps/concepts-blue-green-deployment-strategies

And another valuable article that discusses "tagging" environments so Azure DevOps knows which one to deploy to:
https://learn.microsoft.com/en-us/azure/virtual-machines/linux/tutorial-azure-devops-blue-green-strategy

Share via

Concurrent Swap?

1 answer

Your answer