Running Spark cluster having issue in Databrick workspace

databricks_Debi 1 Reputation point
2022-07-11T08:03:39.797+00:00

Hi Team ,

I am getting error while running scala code in databricks notebook
below is the error

Waiting for cluster to start: Encountered Quota Exhaustion issue in your account: azure_error_code: InvalidTemplateDeployment, azure_error_message: The template deployment failed with error: 'The resource with id: '/subscripti...

![219369-image.png]
1219328-image.png

219445-image.png

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,327 questions
{count} votes

1 answer

Sort by: Most helpful
  1. PRADEEPCHEEKATLA 90,536 Reputation points
    2022-07-12T06:31:25.577+00:00

    Hello @databricks_Debi ,

    Thanks for the question and using MS Q&A platform.

    You can use Azure Free Trial subscription to create a Single node cluster which will have one Driver node with 4 cores.

    A Single Node cluster is a cluster consisting of a Spark driver and no Spark workers. Such clusters support Spark jobs and all Spark data sources, including Delta Lake. In contrast, Standard clusters require at least one Spark worker to run Spark jobs.

    Single Node clusters are helpful in the following situations:

    • Running single node machine learning workloads that need Spark to load and save data
    • Lightweight exploratory data analysis (EDA)

    29653-image.png

    Reference: Azure Databricks - Single Node clusters

    To use multi-node cluster in Azure Databricks, you need to have “Pay-As-You-Go” subscription.

    157703-image.png

    Why can't I use Azure Free Trial subscription?

    Azure Free Trial has a limit of 4 cores, and you cannot create Azure Databricks cluster using a Free Trial Subscription because to create a spark cluster which requires more than 4 cores i.e (Minimum 4 cores for Driver type and 4 cores for Worker type).

    alt text

    For more details, refer to Quickstart: Run a Spark job on Azure Databricks Workspace using the Azure portal

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.