How to calculate the price of a databricks job with Photon engine?

Jiapeng Zhang 0 Reputation points
2024-04-05T14:53:12.39+00:00

It's trivial to calculate a databricks job cost using Azure pricing calculator. You add up Azure VM price and Azure Databricks DBU consumption price and you would get your answer.

However when it comes to Databricks with Photon runtime, the pricing is opaque.

So how on earth can we calculate a databricks job cost when using Photon engine?

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,937 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. PRADEEPCHEEKATLA-MSFT 77,676 Reputation points Microsoft Employee
    2024-04-08T03:43:37.7366667+00:00

    @Jiapeng Zhang - Thanks for the question and using MS Q&A platform.

    Calculating the cost of a Databricks job with Photon engine can be a bit tricky as it is not explicitly mentioned on the Azure pricing calculator page. However, enabling Photon will increase the DBU count, which means that the cost will be higher than a regular Databricks job.

    Regarding the question: How to calculate the price of a databricks job with Photon engine? - I will work with the product team to publish an article or update the Azure Databricks pricing page to explain an sample estimation to understand the price of a databricks job with Photon engine.

    Meanwhile, to estimate the cost of a Databricks job with Photon engine, you can use the following steps:

    • Determine the number of DBUs required for your job without Photon engine enabled for one hour.
    • Enable Photon engine for your job and run it for a specific period of time for one hour.
    • Monitor the DBU consumption for the job during this period of time. You can monitor and view the DBU consumption from Cost Analysis as shown below.User's image
    • Calculate the additional DBUs consumed during the period of time that Photon engine was enabled. You can do this by subtracting the DBU consumption during the period of time that Photon engine was not enabled from the DBU consumption during the period of time that Photon engine was enabled.
    • Multiply the additional DBUs consumed by the cost per DBU to get the additional cost of enabling Photon engine for your job.

    Hope this helps. Do let us know if you any further queries.

    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.