How do I create cluster in Azure Databricks?

Question

Hi,

I am trying to use Databricks on Azure the first time and am having a lot of difficulties. There are so many confusions and hurdles for me.

First, I wanted to use Databricks free trial on Azure. After creating a new subscription, I wasn't able to launch the workspace as Azure asked me to upgrade my option to Pay As You Go to do the same and create compute. Why is so? We cannot create a cluster on Databricks on Azure then what's the meaning of the free trial?

However, I did as suggested. But even then I haven't been able to create a compute, not even Personal Compute. There is no guideline on how a new user should go forward or what steps they should follow to keep going without any problem. Anytime I click on Create Compute after entering the instance details, I get an error like "ab777a75396 has a quota of 3 for resources of type Public|pAddress with sku SkuNotSpecified. Subscription currently has 0 resources and the template contains 8 new resources of the this type which exceeds the quota. Please contact support to increase the quota for resource type Public|pAddress". I don't understand what this error is for. I didn't expect it as I wanted to use a trial version of Databricks on Azure.

I got a hint on Databricks Community to go to Subscriptions -> Usage + Quotas and ask for quota increase. When I go there and select my instance type to raise the request for quota increase, I get message that my region "US West" doesn't have quota for my instance. How do I go from here? How can I resolve this issue? I am stuck at this place for more than one day. I am really confused how Free trail of Azure is working vs free trial of Databricks is working. At this moment, I have cancelled the Azure Subscription on Azure Portal and deleted all the resources.

Please provide me step by step guidelines to resolve this issue. I am also looking for a live chat option or even a video call if I don't understand your instructions properly. As I am looking at Azure portal first time, I am not familiar with its layout.

Answer 1

Hi @Abhishek Pradhan

Thank you for using Microsoft Q&A platform and thanks for posting your question here.

I understand your frustration with the difficulties you're facing while trying to use Databricks on Azure for the first time.

Free Trial Clarification

The Azure Databricks free trial is a limited-time offer that lets you explore the service at no cost, but it comes with restrictions like limited resources and quotas. To create a Databricks cluster, you'll need a Pay-As-You-Go subscription, which is why you're being asked to upgrade. The Azure Free Trial allows only 4 cores, and since creating a Databricks cluster requires more than 4 cores (at least 4 for the Driver and 4 for the Worker), it can't be done with a Free Trial subscription.

For more details, refer to Quickstart: Run a Spark job on Azure Databricks Workspace using the Azure portal

Regarding the quota limit for public IP addresses in your region (US West)

Problem:

Databricks clusters use one public IP address per node (including the driver node). Azure subscriptions have public IP address limits per region. Thus, cluster creation and scale-up operations may fail if they would cause the number of public IP addresses allocated to that subscription in that region to exceed the limit. This limit also includes public IP addresses allocated for non-Databricks usage, such as custom user-defined VMs.

In general, clusters only consume public IP addresses while they are active. However, PublicIPCountLimitReached errors may continue to occur for a short period of time even after other clusters are terminated. This is because Databricks temporarily caches Azure resources when a cluster is terminated. Resource caching is by design, since it significantly reduces the latency of cluster startup and autoscaling in many common scenarios.

Solution:

If your subscription has already reached its public IP address limit for a given region, then you should do one or the other of the following.

• Create new clusters in a different Databricks workspace. The other workspace must be located in a region in which you have not reached your subscription's public IP address limit.
• Request to increase your public IP address limit. Choose Quota as the Issue Type, and Networking as the Quota Type.
• In Details, request a Public IP Address quota increase. For example, if your limit is currently 60, and you want to create a 100-node cluster, request a limit increase to 160.

For more details, refer to Issue: Cloud provider launch failure while setting up the cluster (PublicIPCountLimitReached).

Important! Note: Azure free trial/Student/Pass subscription are not eligible for a quota request. You need to upgrade to a Pay-As-You-Go subscription to increase the quota

If you still have questions, you can chat with sales team to get more information.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.