OSProvisioningTimedOut when deploying multiple virtual machines using Compute Gallery Image

Question

OSProvisioningTimedOut when deploying multiple virtual machines using Compute Gallery Image

Mitchell, Daniel 35

I am consistently getting OSProvisioningTimedOut errors when attempting to deploy 10+ virtual machines. My image is generalized and added to an azure compute gallery with 9 replicas. My understanding is I should be able to deploy 180 VMs concurrently with that image based on the number of replicas. But it seems to time out closer to 30.

azure.core.exceptions.HttpResponseError: (OSProvisioningTimedOut) OS Provisioning for VM 'VMNAME' did not finish in the allotted time. The VM may still finish provisioning successfully. Please check provisioning state later. Also, make sure the image has been properly prepared (generalized).
 * Instructions for Windows: https://azure.microsoft.com/documentation/articles/virtual-machines-windows-upload-image/ 
 * Instructions for Linux: https://azure.microsoft.com/documentation/articles/virtual-machines-linux-capture-image/ 
 * If you are deploying more than 20 Virtual Machines concurrently, consider moving your custom image to shared image gallery. Please refer to https://aka.ms/movetosig for the same.
Code: OSProvisioningTimedOut

kobulloc-MSFT 26,861 Reputation points Microsoft Employee Moderator

2024-01-16T22:35:17.5966667+00:00
Hello, @Mitchell, Daniel !

I have a couple questions to help us determine what is causing this:

Are these single instance VMs, availability sets, or scale sets (uniform or flex)?

How are you deploying these VMs?

What is the VM size and OS being used?
Mitchell, Daniel 35 Reputation points

2024-01-17T03:24:25.2366667+00:00

These are single instance VMs that I deploy using the python sdk. Each VM is created individually and then polled until provisioning is completed. The VM size is Standard_D2s_v3.
kobulloc-MSFT 26,861 Reputation points Microsoft Employee Moderator

2024-01-18T08:02:44.9566667+00:00

Thank you for the information, @Mitchell, Daniel ! From the information you provided, it may be that you are running into rate limits as multiple single VM create calls would add up and may prevent you from completing the remaining creates. If all of the VMs are the same version, the recommendation would be to use Scale Sets. Does that look like it would work for your scenario?
Mitchell, Daniel 35 Reputation points

2024-01-18T14:30:39.9166667+00:00

@kobulloc-MSFT Thanks. Can you provide more information on the rate limits? Is it specific to the VM size selected? I attempted to redeploy a custom compute gallery image definition based off a capture of the win10-22h2-pro-g2 marketplace image. I was able to deploy this more consistently and in line with the replica limits (20*num_replica). The only thing I can think of is that the original image was not "properly prepared (generalized)" as the error message suggests. However, if it wasn't generalized the provisioning should fail 100% of the time, not > 50% of the time, right? I can look into scale sets if you think that would be different. I had looked briefly and it did not look like it would suit our use case.
kobulloc-MSFT 26,861 Reputation points Microsoft Employee Moderator

2024-01-19T21:03:20.06+00:00
Hello, @Mitchell, Daniel !

Rate limits end up being a somewhat complicated answer that isn't cut and dry as it depends on a variety of factors including subscription type but the example given in the documentation is 20 calls per 90 seconds.

Troubleshooting API throttling errors (Virtual Machines)

Throttling Resource Manager requests

If you are seeing expected behavior with a marketplace image but not with the image that you prepared, then I agree that it's worth revisiting the generalization process of the original image. Sometimes it's possible for an error to occur that will need to time out before the process completes. Were these generalization steps followed when the image was created?

https://learn.microsoft.com/en-us/azure/virtual-machines/generalize
Mitchell, Daniel 35 Reputation points

2024-01-19T21:15:26.19+00:00

@kobulloc-MSFT yes I did follow those generalize steps. The person who created the original image said they had as well but maybe they had missed a step. It does appear that may have been the issue so will mark your response as accepted. I appreciate the rate limit information.

Answer accepted by question author

kobulloc-MSFT 26,861 Microsoft Employee Moderator

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to "Accept " the answer.

Why am I receiving the following error? OS Provisioning for VM 'VMNAME' did not finish in the allotted time. The VM may still finish provisioning successfully. Please check provisioning state later. Also, make sure the image has been properly prepared (generalized).

Solution from @Mitchell, Daniel :

The issue is caused by the way the image was generalized. Microsoft support recommended the following options when running Sysprep and this appears to have solved my problem. Engineering team has seen that it’s helping in these scenarios where the specialize pass takes a very long time and recommends to sysprep the image with the option /mode:vm.
 Sysprep /generalize /shutdown /oobe /mode:vm

0 comments

2 additional answers

Your answer

kobulloc-MSFT 26,861 Reputation points Microsoft Employee Moderator

2024-01-16T22:35:17.5966667+00:00

Hello, @Mitchell, Daniel !

I have a couple questions to help us determine what is causing this:

Are these single instance VMs, availability sets, or scale sets (uniform or flex)?

How are you deploying these VMs?

What is the VM size and OS being used?
Mitchell, Daniel 35 Reputation points

2024-01-17T03:24:25.2366667+00:00

These are single instance VMs that I deploy using the python sdk. Each VM is created individually and then polled until provisioning is completed. The VM size is Standard_D2s_v3.
kobulloc-MSFT 26,861 Reputation points Microsoft Employee Moderator

2024-01-18T08:02:44.9566667+00:00

Thank you for the information, @Mitchell, Daniel ! From the information you provided, it may be that you are running into rate limits as multiple single VM create calls would add up and may prevent you from completing the remaining creates. If all of the VMs are the same version, the recommendation would be to use Scale Sets. Does that look like it would work for your scenario?
Mitchell, Daniel 35 Reputation points

2024-01-18T14:30:39.9166667+00:00

@kobulloc-MSFT Thanks. Can you provide more information on the rate limits? Is it specific to the VM size selected? I attempted to redeploy a custom compute gallery image definition based off a capture of the win10-22h2-pro-g2 marketplace image. I was able to deploy this more consistently and in line with the replica limits (20*num_replica). The only thing I can think of is that the original image was not "properly prepared (generalized)" as the error message suggests. However, if it wasn't generalized the provisioning should fail 100% of the time, not > 50% of the time, right? I can look into scale sets if you think that would be different. I had looked briefly and it did not look like it would suit our use case.
kobulloc-MSFT 26,861 Reputation points Microsoft Employee Moderator

2024-01-19T21:03:20.06+00:00

Hello, @Mitchell, Daniel !

Rate limits end up being a somewhat complicated answer that isn't cut and dry as it depends on a variety of factors including subscription type but the example given in the documentation is 20 calls per 90 seconds.

Troubleshooting API throttling errors (Virtual Machines)

Throttling Resource Manager requests

If you are seeing expected behavior with a marketplace image but not with the image that you prepared, then I agree that it's worth revisiting the generalization process of the original image. Sometimes it's possible for an error to occur that will need to time out before the process completes. Were these generalization steps followed when the image was created?

https://learn.microsoft.com/en-us/azure/virtual-machines/generalize
Mitchell, Daniel 35 Reputation points

2024-01-19T21:15:26.19+00:00

@kobulloc-MSFT yes I did follow those generalize steps. The person who created the original image said they had as well but maybe they had missed a step. It does appear that may have been the issue so will mark your response as accepted. I appreciate the rate limit information.

Answer 1

Mitchell, Daniel 35

I worked with Microsoft support on this and we have a solution. The issue is caused by the way the image was generalized. Microsoft support recommended the following options when running Sysprep and this appears to have solved my problem.

Engineering team has seen that it’s helping in these scenarios where the specialize pass takes a very long time and recommends to sysprep the image with the option /mode:vm.

 Sysprep /generalize /shutdown /oobe /mode:vm

kobulloc-MSFT 26,861 Reputation points Microsoft Employee Moderator

2024-02-10T20:19:51.5833333+00:00

Thank you for following up, @Mitchell, Daniel ! I'm going to upvote your answer to increase visibility for anyone else who may run into this issue. I'll also add a quoted version of your answer that you can accept to increase the visibility even more. Thank you again!

Answer 2

kobulloc-MSFT 26,861 Microsoft Employee Moderator

Hello, @Mitchell, Daniel ! Thank you for following up!

Why am I receiving the following error? OS Provisioning for VM 'VMNAME' did not finish in the allotted time. The VM may still finish provisioning successfully. Please check provisioning state later. Also, make sure the image has been properly prepared (generalized).

In this case, it appears that the issue is related to the last line in the error message:

Also, make sure the image has been properly prepared (generalized).

To ensure that the VM has been properly prepared/generalized, use the following steps: https://learn.microsoft.com/en-us/azure/virtual-machines/generalize

I hope this has been helpful! Your feedback is important so please take a moment to accept answers. If you still have questions, please let us know what is needed in the comments so the question can be answered. Thank you for helping to improve Microsoft Q&A! User's image

Mitchell, Daniel 35 Reputation points

2024-01-25T03:06:13.2433333+00:00

@kobulloc-MSFT I've unfortunately had to unaccept the answer. I've confirmed that the image is properly generalized and that the previous image was as well. We are still hitting OSProvisioningTimedOut errors fairly frequently which end up being rather costly. I've even tried increasing the number of replicas to 20. I'm currently failing to deploy just 10 at a time. Also, I've confirmed that VM scale sets do not fit our use case. What other options do we have for investigation? I don't really see any useful errors in the resource group activity logs.
kobulloc-MSFT 26,861 Reputation points Microsoft Employee Moderator

2024-01-25T08:09:53.6633333+00:00
No worries, @Mitchell, Daniel ! Let's get someone to take a look at your resources to see what the root of the issue is. Please email the following to AzCommunity@microsoft.com and we'll get back to you promptly:

Subject: "Attn: kobulloc - OSProvisioningTimedOut on VM deployment"

Email body: Your Subscription ID

Email body: A link to this thread so we can validate and expedite the request

If you don't receive a response within 24 hours, please reply to the thread so we can investigate.
kobulloc-MSFT 26,861 Reputation points Microsoft Employee Moderator

2024-01-25T19:00:41.86+00:00

I responded to your email, @Mitchell, Daniel. Let me know in the comments if you run into any issues and if you get a minute, we'd love to get a follow up with the resolution for anyone else who may run into this. Thank you!

Share via

OSProvisioningTimedOut when deploying multiple virtual machines using Compute Gallery Image

2 additional answers

Your answer