Cannot create a compute in databricks

Question

Hi, I was trying to create a compute in databricks but it is running forever and I can never connect my notebook to it, justing showing 'starting' all the time...unable to run any code in the notebook

Answer 1

Hi Zeen,

That endlessly "starting" compute cluster in Databricks is super frustrating when you're trying to get work done )) it's like watching a loading screen that never ends.

This usually happens when there's some kind of resource allocation issue or networking problem preventing the cluster from fully initializing. Even though it shows as "starting," the underlying VMs might not be provisioning correctly. look at your cluster configuration. If you're using a spot instance or a VM size that's in high demand in your region, Azure might be struggling to allocate the resources. Try switching to a different VM family or use on-demand instances instead of spot to see if that helps. check your subscription quotas. Even if you had capacity before, your quota might have been reduced or other projects might be consuming all available vCPUs for that VM family in your region. The Azure portal shows your current usage and limits under "Subscriptions" > "Usage + quotas." other common culprit is network security groups or firewall rules. If your workspace is VNet-injected, make sure the required outbound rules are in place for Databricks to communicate with its control plane. The cluster needs outbound access on port 443 to several Azure service tags.

You might also want to check the cluster event log in Databricks. Sometimes there are more detailed error messages there that don't show up in the main cluster status. Look for any initialization failures or resource allocation errors.

If you recently made any changes to your workspace configuration, try creating a cluster with the most basic settings possible - single node, smallest VM size, no init scripts. If that works, you can gradually add back your custom configurations to identify what's causing the hang.

Also worth trying a different region temporarily if your data location allows it. Sometimes specific regions have temporary capacity constraints that affect cluster startup times.

regards,

Alex

and "yes" if you would follow me at Q&A - personaly thx.
P.S. If my answer help to you, please Accept my answer

https://ctrlaltdel.blog/