Azure Databricks Sizing

Question

All,

With respect to Sizing the Azure Databricks cluster, how does one know if the current Worker Nodes need additional RAM or if an additional Worker Node needs to be added?

Thanks,
grajee

Answer 1

Hello @Gopinath Rajee ,

Thanks for the question and using MS Q&A platform.

Azure Databricks provides a number of options when you create and configure clusters to help you get the best performance at the lowest cost. This flexibility, however, can create challenges when you’re trying to determine optimal configurations for your workloads. Carefully considering how users will utilize clusters will help guide configuration options when you create new clusters or configure existing clusters. Some of the things to consider when determining configuration options are:

• What type of user will be using the cluster? A data scientist may be running different job types with different requirements than a data engineer or data analyst.
• What types of workloads will users run on the cluster? For example, batch extract, transform, and load (ETL) jobs will likely have different requirements than analytical workloads.
• What level of service level agreement (SLA) do you need to meet?
• What budget constraints do you have?

This article - Best practices: Cluster configuration provides cluster configuration recommendations for different scenarios based on these considerations. This article also discusses specific features of Azure Databricks clusters and the considerations to keep in mind for those features.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators