Share via

Azure Databricks - How to calculate the cost for a job that is configured to use a job cluster?

Anusha S Raj 21 Reputation points
2023-03-09T14:14:11.49+00:00

Azure Databricks - How do i calculate the cost of a databricks job that uses a job cluster? How do I get the number of workers being utilized for a job in real time? Is the cost dependant on the number of workers that are up and running?

Azure Cost Management
Azure Cost Management
A Microsoft offering that enables tracking of cloud usage and expenditures for Azure and other cloud providers.
2,355 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,080 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. PRADEEPCHEEKATLA-MSFT 85,746 Reputation points Microsoft Employee
    2023-03-15T09:41:50+00:00


    Anusha S Raj    , Thanks for the question and using MS Q&A platform.

    To calculate the cost of a Databricks job that uses a job cluster, you will need to consider the following factors:

    1. Cluster specifications: The cost of the job cluster depends on the specifications of the cluster, such as the number of workers, type of workers, and amount of memory and storage. You can choose to use different types of worker nodes based on your requirements and budget.
    2. Duration of job execution: The cost of the job cluster will also depend on how long the job runs. Databricks charges for each second of cluster usage, so the longer the job runs, the higher the cost.

    To get the number of workers being utilized for a job in real-time, you can monitor the job cluster metrics in the Databricks workspace. You can do this by navigating to the "Clusters" tab, selecting the cluster that the job is running on, and clicking on the "Metrics" tab. Here, you will find information about the number of workers that are currently up and running.

    The cost of the job cluster is dependent on the number of workers that are up and running. Databricks charges per second for each worker node, so the more workers you have, the higher the cost. You can manage the number of workers in the job cluster by setting the minimum and maximum number of workers based on your workload requirements.

    In summary, to calculate the cost of a Databricks job that uses a job cluster, you need to consider the cluster specifications and the duration of the job execution. You can monitor the number of workers being utilized for the job in real-time by checking the cluster metrics. The cost of the job cluster is dependent on the number of workers that are up and running.

    Depending on your workloads we recommend different compute configurations. Please follow this guide for best practices.

    Hope this helps. Do let us know if you any further queries.

    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.