I can't create databricks cluster, timeout error

HI,

Recently I can't create databricks clusters. I'm using visual studio enterprise subscription with quota limit of 20 for vCPUs. When I try to create even single cluster with 4 cores after a while I exceed my limit of 20 vCPU cores. It looks like databricks create constantly new clusters and sometimes even if I terminate cluster, the number of taken vCPU is still equal to 20. I need to delete whole databricks workspace to free it. Maybe someone had similar error and could help me. I tried to set up a databricks resource in different locations but to no avail. Thanks in advance.

Bartłomiej Bogajewicz (PL) 21 Reputation points

2023-01-11T08:19:30+00:00

Thanks for answer. I understand how vCores allocates and I'm using single node which creates 4 vCores. My problem is that during cluster set up number of vCores increase constantly to 4,8,12,16 and 20 which is my limit. After that I hit timeout error. I have only one databricks service in my whole account.
BhargavaGunnam-MSFT 26,306 Reputation points Microsoft Employee

2023-01-13T01:15:52.4+00:00

Hello @Bartłomiej Bogajewicz (PL) ,

Thank you for providing the screenshot. Let me check with my internal team and get back to you with more details.
BhargavaGunnam-MSFT 26,306 Reputation points Microsoft Employee

2023-01-13T04:27:50.1866667+00:00

Hello @Bartłomiej Bogajewicz (PL) ,

Could you please send me the time-out error message screenshot?

I see from your screenshot, usage at low level for virtual machines, and usage showing 3 of 25,000. Meaning, 3 VMs got spun up in the Azure data bricks managed resource group.

I see two VMs when there are 2 data bricks clusters from my end.

Can you try the new data bricks cluster in a different region and see if you have the same issue?
Bartłomiej Bogajewicz (PL) 21 Reputation points

2023-01-13T14:44:31.63+00:00

I have created clusters in two additional regions: West US and UK West. In UK West I had the same behavior with quotes going to 20 with timeout error but in West US only 4 vCores were created but after a while I got error that I saved as txt file (error_statement.txt). I added also some screenshots to show you what is going on in more details. This problem started around October, previously I had no problems with cluster creation.
BhargavaGunnam-MSFT 26,306 Reputation points Microsoft Employee

2023-01-14T00:35:14.56+00:00
Hello @Bartłomiej Bogajewicz (PL) ,

Welcome to the MS Q&A platform.

Here is the formula to calculate the number of Vcores on the data bricks cluster.

Driver node + max workers * worker nodes

In my below screenshot, I have selected Standard_D15_v2, which has 20 Vcores and Max workers as 8

So my total number of Vcores = 20+ (8*20)= 180 Vcores

As my workspace has only 100 vcores available, I see the information message saying the account may not have enough CPU cores.

In most cases, the cluster usually requires more than one node, and each node may have at least 4 cores to run (the recommended worker VM is DS3_v2 which has 4 vCores).

Suppose if we are limited to having only 1 node running, which becomes useless, as Spark needs its distributed features to expedite the calculations. Please note that the driver node will need another 4 cores if we choose “Same as worker”.

In this case, total vcores = 4+4 = 8 Vcores

If multiple clusters run, their vCPU usage can add up quickly, leading to an overage of your vCPU quota. Additionally, even if a cluster is idle, it will still consume vCPUs, as Databricks uses them for cluster management and monitoring tasks. So, if you've created a cluster with 4 vCPUs and you're not using it, it's still consuming vCPUs

One consideration is to delete the cluster they are no longer in use to free up the Vcores. Also, you can consider downscaling the clusters that are not in use.

You can also review vCPU usage from the Azure portal, and check for unexpected usage spikes or long-running clusters.

For this, you will need to send the Databricks cluster logs to either one of the below solutions to analyze the logs.

log analytics

Storage account

Event hub

Send to partner solution

You can also submit a request to increase your Vcore limit from 'help+support'

I hope this helps in troubleshooting your issue. Please let me know if you have any further questions.

Please don't forget to click on and upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

Want a reminder to come back and check responses? Here is how to subscribe to a notification

If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Accepted answer

BhargavaGunnam-MSFT 26,306 Reputation points Microsoft Employee

2023-01-14T00:35:32.2233333+00:00

Hello @Bartłomiej Bogajewicz (PL) ,
Thank you for providing the detailed error message. The issue is with a special character in your data bricks cluster name.
I was able to repro the same from my end. Please see the below screenshot for your reference.

Once you remove the special character then you should able to create the data bricks cluster. Meanwhile, I will check with PG if there is any known issue with the special character on the cluster name.

If this resolves the issue, please accept the answer, as it helps the community members looking for answers to similar questions.
Please sign in to rate this answer.

1 person found this answer helpful.
Bartłomiej Bogajewicz (PL) 21 Reputation points

2023-01-16T10:28:16.2766667+00:00

It worked. Thanks a lot!

BhargavaGunnam-MSFT 26,306 Reputation points Microsoft Employee

2023-01-16T17:14:14.3166667+00:00

Thank you for the confirmation, @Bartłomiej Bogajewicz (PL)
Sign in to comment

I can't create databricks cluster, timeout error

0 additional answers