Not able to create a DLT Pipeline in Azure Databricks to move the data from silver to gold layer

Question

Not able to create a DLT Pipeline in Azure Databricks to move the data from silver to gold layer

Shrishti Kumari 0

I am trying to create DLT Pipeline to move the enriched data from silver layer to gold layer in Azure Databricks. This was the configuration, I chose 1 worker node with worker type as Standard_D3_v2, Driver Type as Standard_D3_v2 which together makes 8 cores which is within the limit of 10 cores for Trial Premium version of databricks.

But I am getting an error as the VM size (SKU) requested for launching your Databricks pipeline cluster is currently not available in the region or under your current capacity quota.

When I checked the Quotas > Compute, It shows quota limit exceeded for each and every of the region:

User's image

Please provide a resolution on this as I just have 10 more days left to practice this project.

Krupal Bandari 770 Reputation points Microsoft External Staff Moderator

2025-05-06T00:40:08.5666667+00:00

Hi @Shrishti Kumari
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

1 answer

Your answer

Krupal Bandari 770 Reputation points Microsoft External Staff Moderator

2025-05-06T00:40:08.5666667+00:00

Hi @Shrishti Kumari
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Answer 1

Hi @Shrishti Kumari
You're encountering an error while trying to create a Delta Live Tables (DLT) pipeline in Azure Databricks due to VM size (SKU) unavailability or quota limitations. Even though you've selected a configuration with 1 driver and 1 worker node using the Standard_D3_v2 VM type well within the 10-core limit of your Databricks trial Azure is unable to provision the required VMs in your selected region (Australia Central). This is because your current subscription either has no quota assigned for the requested VM family in that region or the VM type is not available due to capacity constraints. Your quota page confirms that most VM families, including Dv2, DSv2, and Dv3, have a quota of 0 cores, which effectively blocks any deployment in that region.

To resolve this, you should create a new Databricks workspace in a different region where VM quotas are available and try using smaller VM types like Standard_DS1_v2 or DS2_v2, which consume fewer cores and are more likely to be available under your trial limits. If needed and allowed by your subscription type, you may also request a quota increase through the Azure portal.

You can refer to Microsoft’s official documentation to verify and guide these steps:
https://learn.microsoft.com/en-us/azure/databricks/getting-started/free-trial
https://learn.microsoft.com/en-us/azure/quotas/quickstart-increase-quota-portal
https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/overview?tabs=breakdownseries%2Cgeneralsizelist%2Ccomputesizelist%2Cmemorysizelist%2Cstoragesizelist%2Cgpusizelist%2Cfpgasizelist%2Chpcsizelist#general-purpose

I hope this information helps.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues

Shrishti Kumari 0 Reputation points

2025-05-06T13:39:59.7066667+00:00

Hi Krupal, Good day!

Thanks for your quick reply and suggestions. I tried to use the serverless compute. I went back to the DLT pipeline and changed the settings to serverless compute. I didn't try to create a new workspace in different region as you can see in the screenshot above, there were no VM quotas available in any other regions as well apart from Canada Central where I had created the Databricks workspace first. Also creating new workspace from the start may have lead me to again code the entire notebook which I was not sure of to do.

Apologies for the delayed response. Attaching the DLT pipeline settings for the reference.

Thanks,

Shrishti
Krupal Bandari 770 Reputation points Microsoft External Staff Moderator

2025-05-07T01:40:36.6966667+00:00

HI @Shrishti Kumari
it’s great to see that you’ve switched to Serverless compute for your DLT pipeline.

Using serverless is a smart move, especially under quota constraints, as it abstracts away the underlying VM provisioning and typically bypasses quota issues tied to specific VM SKUs. This should allow your pipeline to run without being blocked by region-based capacity or core limits, which is a common challenge with free-tier or trial accounts.

Since you mentioned that Canada Central is the only region with available quota and you've already set up your workspace there, continuing with serverless is the most efficient and effective path forward. Also, good call on avoiding workspace recreation if it risks disrupting your current notebooks and setup.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Share via

Not able to create a DLT Pipeline in Azure Databricks to move the data from silver to gold layer

1 answer

Your answer