Share via

How to track cost for Azure Databricks Lakehouse monitoring per table from system tables

Raja Asokan Charuvil 5 Reputation points
2025-11-07T18:52:52.4033333+00:00

Lakehouse Monitoring or Data Profiling (https://learn.microsoft.com/en-us/azure/databricks/data-quality-monitoring/data-profiling/) is applied on Unity Catalog tables/views and materialized views. I am trying to figure out how to track cost for Azure Databricks Lakehouse monitoring per table from system tables. I have referenced Azure Databricks documentation https://learn.microsoft.com/en-us/azure/databricks/data-quality-monitoring/data-profiling/expense and found out the way to get cost for whole Data Profiling feature.

SELECT usage_date, sum(usage_quantity) as dbus
FROM system.billing.usage
WHERE
  usage_date >= DATE_SUB(current_date(), 30) AND
  sku_name like "%JOBS_SERVERLESS%" AND
  custom_tags["LakehouseMonitoring"] = "true"
GROUP BY usage_date
ORDER BY usage_date DESC

It is using the ´system.billing.usage´ table. But I could not find any way to associate this cost for table. I went through the detailed documentation about ´system.billing.usage´ table and figured out that ´usage_metadata´ field should contain the table information like ´uc_table_name´, ´uc_table_schema´ and ´uc_table_catalog´.

I created some sample monitoring on some of the UC tables and scheduled to run it over a period time. But when I checked the ´system.billing.usage´ tables later for ´billing_origin_product = 'LAKEHOUSE_MONITORING'´, all the fields of usage_metadata fields are null. So I have no possibility to find which table this usage was occur.

It would be really helpful if someone can guide me in the right direction.

Azure Databricks
Azure Databricks

An Apache Spark-based analytics platform optimized for Azure.


1 answer

Sort by: Most helpful
  1. Smaran Thoomu 35,045 Reputation points Microsoft External Staff Moderator
    2025-11-13T14:26:05.8766667+00:00

    Hi Raja Asokan Charuvil
    Good question - and you’re absolutely right in your observation. At this time, the system.billing.usage table doesn’t include per-table details for Lakehouse Monitoring workloads. The usage_metadata fields such as uc_table_name, uc_table_schema, and uc_table_catalog are not yet populated for the LAKEHOUSE_MONITORING billing origin.

    In other words, it’s currently not possible to break down Lakehouse Monitoring costs by individual Unity Catalog tables. The cost data is aggregated at the feature or SKU level (for example, JOBS_SERVERLESS or LAKEHOUSE_MONITORING) rather than by table.

    If you need some level of attribution, one option is to correlate job or task run IDs from your monitoring jobs (using system.job_runs) and map them to your table configurations. That won’t give you precise billing data per table, but it can help approximate usage.

    This is a known limitation, and Databricks is looking at extending metadata coverage for Lakehouse Monitoring in future updates.

    Hope this helps. Do let us know if you any further queries.


    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

    2 people found this answer helpful.
    0 comments No comments

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.