ARM template deployments take much longer then normal

Question

ARM template deployments take much longer then normal

Dan Wyatt 20

Over the last few weeks (starting early Feb) I have seen that ARM deployments using linked templates have started getting delay on seemingly random artifacts within the template. The deployment takes 30 minutes on average when it is running well, but with no change on our side we can see certain linked templates get 'stuck' for hours at a time.

Is anyone else experiencing this issue? It has caused some pretty massive issues with our CI/CD process by this point.

Musaif Shaikh 85 Reputation points

2023-03-07T22:10:38.96+00:00
There have been some reports of similar issues from other users as well. Here are some troubleshooting steps you can take to investigate and potentially resolve the issue:

Check the Azure Status page for any ongoing issues or outages that may be affecting your deployment.

Review the logs for the ARM deployment to see if there are any errors or warnings that may be causing the delay. You can use Azure Monitor or the Azure portal to access the deployment logs.

Review the linked templates to see if there are any issues with the templates themselves that may be causing the delay. You can use a tool like ARMVIZ to visualize your ARM templates and identify potential issues.

Check if there are any changes in your environment or resources that may be impacting the deployment. For example, changes to resource allocation or quotas may cause delays in deployments.

Consider optimizing your ARM templates to reduce the overall deployment time. This could involve optimizing the order of deployment, reducing the number of resources being deployed, or using deployment scripts to automate the process.

If you are still experiencing issues, consider reaching out to Azure Support for assistance. They can provide additional guidance and help troubleshoot the issue.

I hope these steps help you identify and resolve the issue you are experiencing with your ARM deployments.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-03-07T22:21:47.8433333+00:00

Hello Dan Wyatt

Are you using Github or AzuredevOps as your source control here?

Is this a new issue? Or may I know how long you have been having this issue?

Can you try to deploy your ARM template using custom deployment option from the Azure portal and see if it helps?
Dan Wyatt 20 Reputation points

2023-03-08T10:48:54.5633333+00:00

Hello @Bhargava-MSFT
We are using AzureDevOps as our Source control.
This is something we have started seeing from around the 8/10th of February. There haven't been any major changes to our code base in this time and the same ARM template can run fine in 30 minutes on one deployment, then take 2 hours on the next (same environment/infrastructure etc) the ADF we are deploying uses linked templates and different templates seem to get 'stuck' on each deployment.

I will run a few deployments today using the manual process and get back to you on the performance.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-03-13T23:34:38.11+00:00

Hello Dan Wyatt

Sorry for the delayed response. I tried checking with my internal team, and no one has experienced this issue before.

It might be an issue with agent pools. Can you please follow this document and see if you can update the agent pool from your end?

If this doesn't help, I suggest posting the issue in our dedicated DevOps forum or opening a support case for a deeper investigation. If you don't have a support plan, please let me know. I can provide you one-time free support request.

I am looking forward to hearing from you.
John McDonnell 15 Reputation points

2023-03-14T19:55:32.93+00:00

Hi,

I've started to see this issue from today. We're using Azure DevOps, and triggering an ARM template deployment from a Microsoft-Hosted agent.

We've seen Service Bus queues taking 7+ mins to deploy, which is very strange.
Actually its more than the Service Bus, evertything seems slow. What is usually a 10-12 minute deployment of resources, we're now seeing the pipeline expire after an hour of running - we need to increase the timeout to find out how long the deployment actually takes.

I've raised an support ticket against Azure for assistance.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-03-15T01:39:22.5866667+00:00

Hi John McDonnell,

Thanks for your comment. Please update us if you find a resolution for this issue, as it helps the community look for answers to similar questions.
Martin Nenseth 5 Reputation points

2023-03-15T08:34:37.1033333+00:00

I am experiencing the same behaviour.

My ARM template deployment previously took about 15 min. Now its running at 1h and 20m and counting...
Anonymous

2023-03-15T10:27:37.4333333+00:00

@Martin Nenseth This is being face by our team too starting at 7:30 UTC on March 14th.

Reported related issues:

https://developercommunity.visualstudio.com/t/Pipeline-Jobs-Timing-Out-Since-Morning/10309304

Slack discussion:

https://azuredevopsclub.slack.com/archives/CHLTU6CCC/p1678815194870249
John McDonnell 15 Reputation points

2023-03-15T10:38:29.1133333+00:00

I received this update from MS Support this morning:

Starting at 12:00 UTC on 10 Mar 2023 you have been identified as an Azure Resource Manager (ARM) customer who may experience higher than expected latency and timeouts when deploying resources in West Europe and North Europe. Retries of whole deployments may be successful.

Current Status: Our investigation indicates that backend resources that ARM utilizes to process incoming requests have reached operational thresholds. This is causing high latency and timeouts for customers deploying to the West and North Europe regions. We are currently working on identifying appropriate mitigation strategies to alleviate customer impact.
Anonymous

2023-03-15T11:10:04.51+00:00

Thank you very much @John McDonnell for the detail. Kindly keep this thread updated in case of further updates received and in case i hear anything will update this thread for syncing all on current state of issue.
Dan Wyatt 20 Reputation points

2023-03-16T02:17:55.7033333+00:00

I also have a ticket out with Microsoft but have yet to hear anything constructive back from them. (ticket has been open for over a week at this point in time)
I wonder if this is the same issue, my assets are all located in UK South.

Please keep this thread updated with any news/mitigations that are suggested.
Anonymous

2023-03-16T07:25:33.4466667+00:00

"Summary of Impact: Between 12:00 UTC on 10 Mar 2023 and 05:50 UTC on 16 Mar 2023, you were identified as an Azure Resource Manager (ARM) customer who may have experienced higher than expected latency and timeouts when deploying resources in Resource Groups with a location set to West Europe. Retries of whole deployments may have been successful.

Mitigation: We determined this to be a transient impact and applied a configuration change which addressed the latencies and timeouts during deployments in this region."

================================================================

So far, it seems the issue fixed for West Europe consumers. As per response received by John the issue was for West and North Europe regions so perhaps North Europe consumers are still facing it along with others as well @Dan Wyatt
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-03-16T16:49:29.99+00:00

Thank you, John McDonnell and M Maroof Subhani, for your updates.
Dan Wyatt 20 Reputation points

2023-03-20T09:20:50.7833333+00:00

We have had an answer from our Microsoft support ticket. There was nothing actually revealed as to why we saw these delays (despite asking - the person on the ticket is currently asking the technical team to put a document together for this)
We were told the issue has been mitigated, which seems to match up with what we are seeing with our ARM deployments.

Thanks for all the messages on this. I was just glad to know it wasnt 'just' an issue with us and others were seeing the same.

4 answers

Your answer

Musaif Shaikh 85 Reputation points

2023-03-07T22:10:38.96+00:00

There have been some reports of similar issues from other users as well. Here are some troubleshooting steps you can take to investigate and potentially resolve the issue:

Check the Azure Status page for any ongoing issues or outages that may be affecting your deployment.

Review the logs for the ARM deployment to see if there are any errors or warnings that may be causing the delay. You can use Azure Monitor or the Azure portal to access the deployment logs.

Review the linked templates to see if there are any issues with the templates themselves that may be causing the delay. You can use a tool like ARMVIZ to visualize your ARM templates and identify potential issues.

Check if there are any changes in your environment or resources that may be impacting the deployment. For example, changes to resource allocation or quotas may cause delays in deployments.

Consider optimizing your ARM templates to reduce the overall deployment time. This could involve optimizing the order of deployment, reducing the number of resources being deployed, or using deployment scripts to automate the process.

If you are still experiencing issues, consider reaching out to Azure Support for assistance. They can provide additional guidance and help troubleshoot the issue.

I hope these steps help you identify and resolve the issue you are experiencing with your ARM deployments.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-03-07T22:21:47.8433333+00:00

Hello Dan Wyatt

Are you using Github or AzuredevOps as your source control here?

Is this a new issue? Or may I know how long you have been having this issue?

Can you try to deploy your ARM template using custom deployment option from the Azure portal and see if it helps?
Dan Wyatt 20 Reputation points

2023-03-08T10:48:54.5633333+00:00

Hello @Bhargava-MSFT
We are using AzureDevOps as our Source control.
This is something we have started seeing from around the 8/10th of February. There haven't been any major changes to our code base in this time and the same ARM template can run fine in 30 minutes on one deployment, then take 2 hours on the next (same environment/infrastructure etc) the ADF we are deploying uses linked templates and different templates seem to get 'stuck' on each deployment.

I will run a few deployments today using the manual process and get back to you on the performance.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-03-13T23:34:38.11+00:00

Hello Dan Wyatt

Sorry for the delayed response. I tried checking with my internal team, and no one has experienced this issue before.

It might be an issue with agent pools. Can you please follow this document and see if you can update the agent pool from your end?

If this doesn't help, I suggest posting the issue in our dedicated DevOps forum or opening a support case for a deeper investigation. If you don't have a support plan, please let me know. I can provide you one-time free support request.

I am looking forward to hearing from you.
John McDonnell 15 Reputation points

2023-03-14T19:55:32.93+00:00

Hi,

I've started to see this issue from today. We're using Azure DevOps, and triggering an ARM template deployment from a Microsoft-Hosted agent.

We've seen Service Bus queues taking 7+ mins to deploy, which is very strange.
Actually its more than the Service Bus, evertything seems slow. What is usually a 10-12 minute deployment of resources, we're now seeing the pipeline expire after an hour of running - we need to increase the timeout to find out how long the deployment actually takes.

I've raised an support ticket against Azure for assistance.
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-03-15T01:39:22.5866667+00:00

Hi John McDonnell,

Thanks for your comment. Please update us if you find a resolution for this issue, as it helps the community look for answers to similar questions.
Martin Nenseth 5 Reputation points

2023-03-15T08:34:37.1033333+00:00

I am experiencing the same behaviour.

My ARM template deployment previously took about 15 min. Now its running at 1h and 20m and counting...
Anonymous

2023-03-15T10:27:37.4333333+00:00

@Martin Nenseth This is being face by our team too starting at 7:30 UTC on March 14th.

Reported related issues:

https://developercommunity.visualstudio.com/t/Pipeline-Jobs-Timing-Out-Since-Morning/10309304

Slack discussion:

https://azuredevopsclub.slack.com/archives/CHLTU6CCC/p1678815194870249
John McDonnell 15 Reputation points

2023-03-15T10:38:29.1133333+00:00

I received this update from MS Support this morning:

Starting at 12:00 UTC on 10 Mar 2023 you have been identified as an Azure Resource Manager (ARM) customer who may experience higher than expected latency and timeouts when deploying resources in West Europe and North Europe. Retries of whole deployments may be successful.

Current Status: Our investigation indicates that backend resources that ARM utilizes to process incoming requests have reached operational thresholds. This is causing high latency and timeouts for customers deploying to the West and North Europe regions. We are currently working on identifying appropriate mitigation strategies to alleviate customer impact.
Anonymous

2023-03-15T11:10:04.51+00:00

Thank you very much @John McDonnell for the detail. Kindly keep this thread updated in case of further updates received and in case i hear anything will update this thread for syncing all on current state of issue.
Dan Wyatt 20 Reputation points

2023-03-16T02:17:55.7033333+00:00

I also have a ticket out with Microsoft but have yet to hear anything constructive back from them. (ticket has been open for over a week at this point in time)
I wonder if this is the same issue, my assets are all located in UK South.

Please keep this thread updated with any news/mitigations that are suggested.
Anonymous

2023-03-16T07:25:33.4466667+00:00

"Summary of Impact: Between 12:00 UTC on 10 Mar 2023 and 05:50 UTC on 16 Mar 2023, you were identified as an Azure Resource Manager (ARM) customer who may have experienced higher than expected latency and timeouts when deploying resources in Resource Groups with a location set to West Europe. Retries of whole deployments may have been successful.

Mitigation: We determined this to be a transient impact and applied a configuration change which addressed the latencies and timeouts during deployments in this region."

================================================================

So far, it seems the issue fixed for West Europe consumers. As per response received by John the issue was for West and North Europe regions so perhaps North Europe consumers are still facing it along with others as well @Dan Wyatt
Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2023-03-16T16:49:29.99+00:00

Thank you, John McDonnell and M Maroof Subhani, for your updates.
Dan Wyatt 20 Reputation points

2023-03-20T09:20:50.7833333+00:00

We have had an answer from our Microsoft support ticket. There was nothing actually revealed as to why we saw these delays (despite asking - the person on the ticket is currently asking the technical team to put a document together for this)
We were told the issue has been mitigated, which seems to match up with what we are seeing with our ARM deployments.

Thanks for all the messages on this. I was just glad to know it wasnt 'just' an issue with us and others were seeing the same.

Answer 1

John McDonnell 15

I've no further update from Microsoft, I'm awaiting on an RCA when they resolve the issue. We managed to unblock ourselves by moving the deployments of those resources temporarily to ukwest. This appears to be working fine for us.

Answer 2

Dan Wyatt 20

We have had an answer from our Microsoft support ticket. There was nothing actually revealed as to why we saw these delays (despite asking - the person on the ticket is currently asking the technical team to put a document together for this)
We were told the issue has been mitigated, which seems to match up with what we are seeing with our ARM deployments.

Answer 3

prathibha p 0

I am also facing same kind of issue, ARM template deployment run time consuming more time for every run, for the first time it will consume more time, but while updating it should reduce time. Any suggestions

Answer 4

Jose Nicolas Zambrano Benavides 0

I have the same issue in East US 2 location... a basic ARM template to deploy SQL server and database, times out at 30 min approx. In the second run of the pipeline, it works 80% of the times, otherwise, it times out again at 30 min.

Is there actually any solution for this issue?

TY in advance!

Share via

ARM template deployments take much longer then normal

4 answers

Your answer