Databricks Cluster Size

Question

Hi,

We have a below use case.

We are developing ML Development using azure data bricks cluster service's. Our data bricks will receive 20 GB size of dataset records and we are running our logic/algorithms against this dataset's.

We have to run every day once this ML algorithms against 20 GB size of dataset's. I'm not able to Identify the required cluster size.

I have checked while creating compute in data bricks available ML based GPU supported clusters but its cost more.

Please let me the best compute size to perform this operations.

Thanks.

Answer 1

Hello james vasanth,

There are several factors and best practices to consider. Here are some key points:

• User Type: The type of user (data scientist, data engineer, data analyst) can influence the compute configuration.
• Workload Type: Different workloads (ETL jobs, analytical workloads) have different requirements.
• Service Level Agreement (SLA): The level of SLA you need to meet can influence your configuration.
• Budget Constraints: Your budget can determine the size and type of cluster you choose.
• Compute Features: Understanding the features of Databricks compute (all-purpose compute, job compute, single-node, multi-node, on-demand, spot instances) can help you make an informed decision.
• Concurrent Queries: Databricks recommends a cluster for every 10 concurrent queries.
• Data Size: The size of your data can also influence the size of your cluster.

Your configuration decisions will require a tradeoff between cost and performance. It’s recommended to start with a smaller cluster and then monitor the performance. You can then adjust the size of the cluster as needed based on your observations.

References:

• https://docs.databricks.com/en/compute/cluster-config-best-practices.html
• https://docs.databricks.com/en/compute/sql-warehouse/warehouse-behavior.html
• https://community.databricks.com/t5/data-engineering/choosing-the-optimal-cluster-size-specs/td-p/11334

If the information helped address your question, please Accept the answer.

Luis