Azure Databricks Sizing

Gopinath Rajee 646 Reputation points
2022-04-11T01:22:37.84+00:00

All,

With respect to Sizing the Azure Databricks cluster, how does one know if the current Worker Nodes need additional RAM or if an additional Worker Node needs to be added?

Thanks,
grajee

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,010 questions
{count} votes

Accepted answer
  1. PRADEEPCHEEKATLA-MSFT 81,646 Reputation points Microsoft Employee
    2022-04-11T12:33:55.12+00:00

    Hello @Gopinath Rajee ,

    Thanks for the question and using MS Q&A platform.

    Azure Databricks provides a number of options when you create and configure clusters to help you get the best performance at the lowest cost. This flexibility, however, can create challenges when you’re trying to determine optimal configurations for your workloads. Carefully considering how users will utilize clusters will help guide configuration options when you create new clusters or configure existing clusters. Some of the things to consider when determining configuration options are:

    • What type of user will be using the cluster? A data scientist may be running different job types with different requirements than a data engineer or data analyst.
    • What types of workloads will users run on the cluster? For example, batch extract, transform, and load (ETL) jobs will likely have different requirements than analytical workloads.
    • What level of service level agreement (SLA) do you need to meet?
    • What budget constraints do you have?

    This article - Best practices: Cluster configuration provides cluster configuration recommendations for different scenarios based on these considerations. This article also discusses specific features of Azure Databricks clusters and the considerations to keep in mind for those features.

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

0 additional answers

Sort by: Most helpful