Help to resolve VM size not available issue while creating cluster in Databricks

Question

I am trying to create a cluster in Databricks. I have chosen worker type, spent more than 3 hours. I am getting the message "The VM size you are specifying is not available. [details] QuotaExceeded: Operation could not be completed as it results in exceeding approved Total Regional Cores quota. Additional details - Deployment Model: Resource Manager, Location: centralus..."

I have also gone through the steps explained in the video.

https://www.youtube.com/watch?v=qpRVQYShez8

No resources are available.

Can anyone help me with an easy way to resolve this issue?

Answer 1

To resolve the issue of the VM size not being available while creating a cluster in Databricks, you can consider the following steps:

1. Check Quota Limits: The error message indicates that you may be exceeding your approved Total Regional Cores quota. You can check your current quota limits in the Azure portal and request an increase if necessary.
2. Choose a Different VM Size: If the specific VM size you are trying to use is not available, consider selecting a different VM size that is available in your chosen region. You can do this by checking the available VM sizes for the region you are working in.
3. Retry Later: Sometimes, the availability of VM sizes can change due to demand. If possible, try to create the cluster at a later time.
4. Create in a Different Region: If the VM size is not available in the "centralus" region, you might want to consider creating the cluster in a different region where the desired VM size is available.
5. Contact Support: If you continue to face issues, you can create a support request through the Azure portal for further assistance.

---

References:

• Common Issues: Unavailable Virtual Machine SKU
• Azure Windows VM deployment top issues
• Allocation failures when deploying custom/gallery/marketplace images

Answer 2

Hello Krishnakanth Manivannan,

Thanks for your question.

This error message usually occurs when there is not enough capacity in the region to provision the requested VM size.

To resolve this issue, you can try one of the following options:

• Try a different VM size: You can try selecting a different VM size that is available in the region. You can refer to the Azure documentation to find the list of available VM sizes in each region.
• Deploy to a different location or zone: You can try deploying the cluster to a different location or zone where the requested VM size is available.
• Request a quota increase: If you need to use the requested VM size in the specified location, you can request a quota increase for that VM size in that location. You can refer to the Azure documentation to learn how to request a quota increase.

See similar threead here:

https://learn.microsoft.com/en-us/answers/questions/611894/cluster-terminated-quota-exceeded-expectation-whil

The above information is cited from:
https://learn.microsoft.com/en-us/answers/questions/1782661/while-creating-cluster-in-databricks-i-am-getting

You can mark it 'Accept Answer' and 'Upvote' if this helped you

Regards,

Abiola

Answer 3

Hello, @Krishnakanth Manivannan !

You may get a better answer using the Databricks tag but in the video you linked, it looks like they are using 44 cores at one point (2 workers at 20 cores each and a driver at 4 cores) with only 30 cores available on the subscription (around 9m 30s). I'll explain below why this would cause a quota error, even though they are in different VM families.

Why am I getting a Total Regional Cores quota error when trying to deploy VMs?

There are two tiers of vCPU quota types for standard vCPU quotas:

• Regional: The first tier is the total regional vCPU quota.
• VM Family: The second tier is the VM-family vCPU quota such as D-series vCPUs

The error you are encountering indicates that you need to increase the total number of cores for the region, regardless of which VM families you are using:

https://learn.microsoft.com/en-us/azure/quotas/regional-quota-requests#request-an-increase-for-regional-vcpu-quotas

It's worth checking your current vCPU usage to make sure that you don't have unused resources that are counting against your vCPU quota for the region that you are interested in.

If a quota increase to your desired vCPU amount isn't available for any reason, then you can try a different region or you can try a VM size with a lower vCPU count.

Resources:

• Increase regional vCPU quotas
• (VM) Check vCPU quota

---

I hope that this has been helpful! Your feedback is important, so please take a moment to accept an answer if it was useful.