App Service deployment slot replicated to production before we swapped the slots

Stuart 1

We had an issue with our deployment pipeline for our Azure app service on our last push to production, and I wanted to ask if anyone has seen similar, or has any ideas why it's happening?

Our typical flow is to push from our staging environment to a prep slot off our main production environment. We'll then make final checks of the new code on the prep slot before switching the prep and production slots, pushing the new code live.

On our most recent push, however, something unexpected happened. We pushed code from staging to the prep slot and, once deployed, I checked the production site to ensure that process hadn't disrupted production (no reason it should, but I like to check). Everything was fine, and production was running the soon-to-be-replaced version of the site.

As we were testing the code on the prep slot (a process which took about an hour) we noticed that the code on the live production site had been replaced with the code we had just pushed to the prep slot. We hadn't taken any steps to request this, and the prep and production slots hadn't swapped; they were both running the latest code we had pushed to the prep slot only.

One thing we did notice was that, a few minutes after deployment, we received an Azure Monitor notification that our resource capacity was being auto-scaled from 1 to 2. Could this auto-scaling have confused the two slots and taken our prep slot code live?

Any advice or suggestions greatly appreciated :)

brtrach-MSFT 15,251 Reputation points Microsoft Employee

2022-01-19T05:08:05.21+00:00
@Stuart We are sorry to hear you encountered this unexpected situation with your deployment.

Ultimately, reviewing the swap logs would be the best approach as we otherwise would be left throwing many darts at a board and hoping one of them would stick.

Customer's are empowered to use the same tool for troubleshooting web apps that are paid support team uses. To access this feature:

Navigate to portal.azure.com

Navigate to your web app

Select the 'diagnose and solve' blade of your web app

Select "configuration and management" blade

Configure the the range

Select "Check swap operations"

Do you see any insightful details listed here? If so, please share a sanitized version of them here with us. We look forward to your reply.
Stuart 1 Reputation point

2022-01-19T18:06:31.53+00:00

Thanks for your help on this.

I ran the report you recommended, and it came back with green ticks for everything except:

Autoscale rule configuration has very low value for maximum CPU or memory Percentage
Percentage configured CpuPercentage = 50
Tip We recommend choosing a high value for maximum CPU or memory Percentage.

It also listed:

Scaling Operations 2
No. of Workers 1

No requests rejected due to IP address restrictions

Interestingly, it states "No swap operations occured during this timeframe". Which follows what we saw. The prep and production slots didn't swap, the code deployed to the prep slot duplicated to the production slot.

Does this give you any insight into what could have occurred?
Stuart 1 Reputation point

2022-01-24T20:15:11.647+00:00

I also checked to confirm, and auto-swap is disabled on the prep slot.
brtrach-MSFT 15,251 Reputation points Microsoft Employee

2022-01-27T05:28:00.437+00:00

@Stuart Thank you for the updates and I apologize for the delay in getting back to you as we have had a surge in customer engagements recently.

Reviewing the information you shared, it certainly does not align with your experience. I think it would be best to have a support engineer work one on one with you to dig through your site files and see if they can locate the root cause as this does not sound like it should have happened.

Can you please reach out to me at azcommunity@microsoft.com with the subject as ATTN: Bryan and in the body of the email please include your Azure subscription ID. We look forward to your reply.