Issue with spin the service Databricks cluster on free account.

Question

Issue with spin the service Databricks cluster on free account..

Databricks offers 14 days free trial as per the below link,

https://www.databricks.com/try-databricks?scid=7018Y000001Fi1AQAS&utm_medium=paid+search&utm_source=google&utm_campaign=15424459433&utm_adgroup=150300911441&utm_content=trial&utm_offer=try-databricks&utm_ad=665996163291&utm_term=databricks%20community%20edition%20free%20trial&gad=1&gclid=Cj0KCQjwz8emBhDrARIsANNJjS75pjXz7gv1z6UcVrSKwoJMTYHKmyTbW1YkMFR3ovnaV-DmBNJ65KAaApSoEALw_wcB#account

However, when it takes me to Azure portal, and I tried to spin a cluster inside Databricks workspace, Azure does not provide any supporting VM to Host the cluster.

Below is the description of the error-

Azure Quota Exceeded Exception:

Error code: QuotaExceeded, error message: Operation could not be completed as it results in exceeding approved Total Regional Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus, Current Limit: 4, Current Usage: 4, Additional Required: 16, (Minimum) New Limit Required: 20. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%221453fbdf-5a74-4324-98ca-d20dd5c73005%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22eastus%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22cores%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:20,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22cores%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/regional-quota-requests

Answer 1

@Ravi Kumar - - Thanks for the question and using MS Q&A platform.

This is execpted behaviour when you are trying to use more CPU cores which are not available in your subscription (Azure free trial/Student/Pass).

Reason: Azure free trial/Student/Pass subscription has a limit of 4 cores, and you cannot create Databricks cluster multi node using a Student Subscription because it requires more than 8 cores.

You need to upgrade to a Pay-As-You-Go subscription to create Azure Databricks clusters with multi mode.

Note: Azure free trial/Student/Pass subscriptions aren't eligible for limit or quota increases. If you have a Student subscription, you can upgrade to a Pay-As-You-Go subscription.

You can use Azure Free Trial subscription to create a Single node cluster which will have one Driver node with 4 cores.

A Single Node cluster is a cluster consisting of a Spark driver and no Spark workers. Such clusters support Spark jobs and all Spark data sources, including Delta Lake. In contrast, Standard clusters require at least one Spark worker to run Spark jobs.

Single Node clusters are helpful in the following situations:

• Running single node machine learning workloads that need Spark to load and save data
• Lightweight exploratory data analysis (EDA)

Reference: Azure Databricks - Single Node clusters

To use multi-node cluster in Azure Databricks, you need to have “Pay-As-You-Go” subscription.

Why can't I use Azure Free Trial subscription?

Azure Free Trial has a limit of 4 cores, and you cannot create Azure Databricks cluster using a Free Trial Subscription because to create a spark cluster which requires more than 4 cores i.e (Minimum 4 cores for Driver type and 4 cores for Worker type).

For more details, refer to Quickstart: Run a Spark job on Azure Databricks Workspace using the Azure portal

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.