Optimizing CD pipeline when swapping multiple web apps

Question

Hi all,

I am currently building a Azure CD Pipeline which deploys up to eight web apps running on a Linux app service plan. The pipeline deploys to a deployment slot on each of the web apps and then performs a swap after deployment has succeeded.

In Azure, we run all eight web apps under the same Linux App service plan (currently a P2V3 plan) and our pipeline looks like this (I've included only a single web app in this example to avoid bloating the post):

stages:
    - stage: Deploy
    - pool:
        vmImage: 'ubuntu-latest'
    jobs:
        - deployment: Deploy
        environment: Production
        strategy:
            runOnce:
                deploy:
                    steps:
                        - task: AzureWebApp@1
                           displayName: "Deploy My webapp"
                           inputs:
                               azureSubscription: <azureSubscriptionName>
                               appType: 'webAppLinux'        
                               deployToSlotOrASE: true           
                               appName: 'my-web-app'                    
                               slotName: 'my-web-app-slot'
                              package: '$(PIPELINE.WORKSPACE)/MyWebApp/MyWebApp.zip'

                # .... seven more deploys here

                         - task: AzureAppServiceManage@0
                           inputs:
                               azureSubscription: <AzureSubscriptionName>
                               Action: 'Swap Slots'
                               WebAppName: 'my-web-app'
                               ResourceGroupName: 'MyResourceGroup'
                               SourceSlot: 'my-web-app-slot'

                # .... seven more slot swaps here

If I run the pipeline with only a single web app, I'm looking at a swap time of around 2 minutes average. The warmup is nearly instant, so it's more or less the swap operation itself that takes up all the time.

Now, if I run the pipeline with all eight web app deploys and swaps, I get completely random - and often long swap operation times. All warmups are still quite fast. Some of the swap operations takes around 2 minutes while some takes 5, 7 or even 10 minutes. This leads to very long deploment times - and often the pipeline times out at 60 minutes.

So, these questions arise:

1) Is around 2 minutes for the swap operation of a single web app to be expected when deploying/swapping a single web app?

2) Can I do anything to optimize the deployment time here?

3) Am I expecting too much (performance wise) of eight web apps running in a single plan when it comes to deployment/swapping? The plan runs at ~40% average memory percentage and ~10% CPU usages average, so nothing alarming there.

Thanks a lot in advance.

All the best,

Bo

Answer 1

Thanks for asking question! If the swap operation takes a long time to complete, you can get information on the swap operation in the activity log.
On your app's resource page in the portal, in the left pane, select Activity log.

A swap operation appears in the log query as Swap Web App Slots. You can expand it and select one of the suboperations or errors to see the details.

Further you may want to know:

Some apps might require custom warm-up actions before the swap. The applicationInitialization configuration element in web.config lets you specify custom initialization actions. The swap operation waits for this custom warm-up to finish before swapping with the target slot.

During the swap operation the Web App’s worker process may get restarted in order for some settings to take effect. Even though the swap does not proceed until the restarted worker process comes back online on every VM instance, it may still not be enough for application to be completely ready to take on production traffic.

Try enabling Application Initialization Module to completely warm up your application prior to swapping it into production.

More detailed explanation of that process is available at How to warm up Azure Web App during deployment slots swap

Please refer to below links for more details on this:

https://learn.microsoft.com/en-us/azure/app-service/deploy-staging-slots#what-happens-during-a-swap
https://ruslany.net/2017/11/most-common-deployment-slot-swap-failures-and-how-to-fix-them/
https://ruslany.net/2019/06/azure-app-service-deployment-slots-tips-and-tricks/

You can also use App Service Diagnostics feature which helps you to troubleshoot your app with no configuration required. When you run into issues with your app, App Service Diagnostics points out what’s wrong and guide you to the right information to more easily resolving the issues.

Go to Azure Portal > Select your App Service > on left side of menu click on Diagnose and solve problems > select configurations and Management > select check swap operations from left menu.

Let us know If further query or issue remains.

Answer 2

Hi @SnehaAgrawal-MSFT

Thanks a lot for your reply, greatly appreciated! Ideally I should've made this as a comment to your answer, but nothing happens when clicking the submit button when adding a comment.

I have just been through everything you wrote and suggested. I forgot to mention that we're running .NET 5 on our Linux Web apps in Azure. We have set the following app settings on our web app slots (marked as slot settings):

• WEBSITE_SWAP_WARMUP_PING_PATH which has a value of "/healthcheck" (an endpoint we've set up which returns 200 OK)
• WEBSITE_SWAP_WARMUP_PING_STATUSES which has a value of 200.

We can verify that the /healthcheck endpoint is indeed called during warmup by looking at Application Insights transaction logs.

Looking at the activity log of our web apps (not the deployment slot, but the "real" web app) it does seem a little weird to me. It looks like each operation is running twice (see attached screenshot) Is this normal behavior?

Apart from the weird looking activity log, I guess the question leans more towards an advice: how do we optimize our Azure CD pipelines when we want to:

1) Deploy 8 seperate web apps to their deployment slots and
2) Warmup and swap all 8 deployment slots

Each swap operation takes around 2 - 5 minutes which makes a total duration of ~30 minutes for a deployment. Can we do this faster in any way? Attached is a screenshot showing all tasks of our deploy job for reference. The swaps in the screenshot takes around 2 - 4 minutes each, but we have seen swap durations up to 5 - 7 minutes at times.

Again: this is on a P2V3 Linux App Service Plan which "hosts" all of our eight web apps.

Thanks again.

All the best,

Bo

Answer 3

Hey @Bo Damgaard Mortensen ,

Haven't found anything concrete yet but today we noticed something about the AzureAppServiceManage task - if you dig into the task (https://github.com/microsoft/azure-pipelines-tasks/blob/aa9411596dd6713974a1f1eef8936cb93c067a0c/Tasks/AzureAppServiceManageV0/azureappservicemanage.ts#L100) it specifically blocks slot swap with preview for linux/containers because it seems to be using an API version from 2016 which I guess didn't have the functionality.

We switched to the following tasks and on the first run we saw a huge improvement - unfortunately once we submitted the PR the CI swap took about 12 minutes, so we're going to monitor it over the new few days to get a sense if it does make it better or not - maybe it's something you can try and see how it goes, at the very least it's nice to get the slot swap with preview ability. I flagged it with MS so hopefully the AzureAppServiceManage task gets an @1 revision!

Begin with:

az webapp deployment slot swap -g <RG> -n <app> --slot staging --target-slot production --action preview

Complete with:

az webapp deployment slot swap -g <RG> -n <app> --slot staging --target-slot production --action swap

Cancel/revert with:

az webapp deployment slot swap -g <RG> -n <app> --slot staging --target-slot production --action reset

Edit: We've actually seen a huge improvement in most runs, sub-1 minute swap times. That said, we've seen a few 10 minute ones too, we added a --debug on our az cli commands in an attempt to see more about what is happening. The full az cli task (swap) you could try would be something like:

          - task: AzureCLI@2  
            displayName: 'Complete Swap Slots: <web app name>'  
            inputs:  
              azureSubscription: ${{ parameters.serviceConnection }}  
              scriptType: pscore  
              scriptLocation: inlineScript  
              inlineScript: >  
                az webapp deployment slot swap -g ${{ parameters.rgName }} -n 'web app name' --slot staging --target-slot production --action swap --debug