Azure databricks workload type

Question

I am calling a azure databricks cluster from a AI/ML pipeline.
I want to understand it will get executed under which feature type: Data analytics or data engineering.

I want to understand this to get my databricks usage cost calculation.

Answer 1

Hello @Sandhya Prusty ,

Welcome to Microsoft Q&A platform.

Azure Databricks cluster for AI/ML pipelines will be billed as ALL-PURPOSE COMPUTE price.

Effective September 1, 2020, the meter names of Azure Databricks meters have changed.

• ALL-PURPOSE COMPUTE - Interactive workloads to analyze data collaboratively with notebooks
• JOBS COMPUTE - Automated workloads to run fast and robust jobs via API or UI
• JOBS LIGHT COMPUTE - Automated workloads to run robust jobs via API or UI

To calculate the databricks usage cost, here is the formula.

Formula:

Total cost of databricks = VM Cost + DBU Cost  
VM Cost = [Total Hours] X [No. of Instances] X [Linux VM Price]  
DBU Cost = [Total Hours] X [No. of Instances] X [DBU] X [DBU Price/hour - Standard / Premium Tier]  

Here is an example on how Azure Databricks billing works?

Depending on the type of workload your cluster runs, you will either be charged for Jobs Compute or All-Purpose Compute workload. For example, if the cluster runs workloads triggered by the Databricks jobs scheduler, you will be charged for the Jobs Compute workload. If your cluster runs interactive features such as ad-hoc commands, you will be billed for All-Purpose Compute workload.

• If you run Premium tier cluster for 100 hours in East US 2 with 10 DS13v2 instances, the billing would be the following for All-Purpose Compute workload: VM cost for 10 DS13v2 instances —100 hours x 10 instances x $0.598/hour = $598 DBU cost for All-Purpose Compute workload for 10 DS13v2 instances —100 hours x 10 instances x 2 DBU per node x $0.55/DBU = $1,100 The total cost would therefore be $598 (VM Cost) + $1,100 (DBU Cost) = $1,698.
• If you run Premium tier cluster for 100 hours in East US 2 with 10 DS13v2 instances, the billing would be the following for Jobs Compute workload: VM cost for 10 DS13v2 instances —100 hours x 10 instances x $0.598/hour = $598 DBU cost for Jobs Compute workload for 10 DS13v2 instances —100 hours x 10 instances x 2 DBU per node x $0.30/DBU = $600 The total cost would therefore be $598 (VM Cost) + $600 (DBU Cost) = $1,198.
• If you run Premium tier cluster for 100 hours in East US 2 with 10 DS13v2 instances, the billing would be the following for Jobs Light Compute workload: VM cost for 10 DS13v2 instances —100 hours x 10 instances x $0.598/hour = $598 DBU cost for Jobs Light Compute workload for 10 DS13v2 instances —100 hours x 10 instances x 2 DBU per node x $0.22/DBU = $440 The total cost would therefore be $598 (VM Cost) + $440 (DBU Cost) = $1,038.

In addition to VM and DBU charges, you may also be charged for managed disks, public IP address or any other resource such as Azure Storage, Azure Cosmos DB depending on your application.

For more details, refer "Azure Databricks pricing".

Hope this helps. Do let us know if you any further queries.

------------

• Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
• Want a reminder to come back and check responses? Here is how to subscribe to a notification.