Visual Studio $50 Credit and Databricks Issues

Question

Hey everyone.

We use Azure Databricks at work. I set up a resource group for databricks and have not used it to do any computes. However, it is draining my $50 credit at a crazy rate, $20 over the last 3 days. The other devs I work with said this isn't an issue, and that they typically have so much money left over at the end of every month. I am trying to understand what the issue might be as none of us have been able to troubleshoot it.

Answer 1

@Alec Patsey - Thanks for the question and using MS Q&A platform.

I set up a resource group for databricks and have not used it to do any computes. However, it is draining my $50 credit at a crazy rate, $20 over the last 3 days.

In addition to VM and DBU charges, you may also be charged for managed disks, public IP address, or any other resource such as Azure Storage, Azure Cosmos DB depending on your application. The VMs provisioned for any cluster will be charged from the VM "Starting" phase until the cores are no longer allocated to the virtual machine.

Let's understand the Azure Databricks costs in depth:

In general, majority of Azure Databricks costs are expected to come from DBU charges and the underlying Virtual Machines used by the clusters so the bare minimum need is to get better handle on Databricks costs is to capture costs for DBU and the Virtual Machines. In case you just try to find the costs by resource or resource groups using Azure Cost Management it is not easy to capture both of them in one place. The guidance shared here will help with getting a more wholistic view of costs including sub-components like disks, networking resources, etc. which will give you even more accuracy beyond DBU and VM costs.

This official documentation: Monitor usage using tags covers the tags and its propagation to resources in detail. A few things to highlight from the documentation about default tags:

• A default tag named Vendor with value Databricks is propagated to all the Azure Databricks workspace resources (VMs, Network Interface, Disks, etc.) with the exception of default storage account created in the Managed Resource for DBFS.
• ClusterName is another default tag is propagated to underlying resources.
• RunName default tag is propagated to underlying resources with value Job Name for Job Cluster, ClusterName tags are not super useful for Job Clusters if they are running on schedule because job clusters get assigned a new name for each run so RunName is a better tag for figuring out job costs.
• For clusters created from Cluster Pools, Workspace and Pool tags are propagated but ClusterName default tags are not propagated to underlying resources.

To browse to Cost Analysis section on Azure Portal, search Cost Management + Billing from Azure Portal, then browse to Cost Management and then Cost Analysis.

For more details, refer to Understanding Azure Databricks Costs using Azure Cost Management for Observability and Chargebacks — Effective Tag Usage.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.