Monitor the cost of serverless compute
This article explains how to use the billable usage system table (Public Preview) to monitor the cost of your serverless compute usage.
You can monitor the usage of serverless compute for notebooks and workflows by querying the billable usage system table (system.billing.usage
), which includes user and workload attributes related to serverless compute costs. The applicable fields include:
- The
identity_metadata
column includes therun_as
field, which shows the user or service principal whose credentials were used to run the workload. - The
usage_metadata
column has fields that describe the workload:job_run_id
andnotebook_id
.
Considerations for serverless usage records
When analyzing your serverless usage, consider the following:
- You may see multiple records associated with a given serverless compute workload in a given hour. For example, you may see multiple records with the same
job_id
andjob_run_id
but with different DBU consumption values for each. The sum of these DBUs collectively represents the hourly DBU consumption for a given job run. - You may also see records with DBU consumption billed using a serverless SKU, but with null values for
run_as
,job_id
,job_run_id
, andnotebook_id
. These represent costs associated with shared resources that are not directly attributable to any particular workload. As you increase your usage of serverless compute and add more workloads, the proportion of these shared costs on your bill will decrease as they are shared across more workloads.
Cost observability dashboard
To help you get started monitoring your serverless costs, download the following cost observability dashboard from Github. See Serverless cost observability dashboard.
After you download the JSON file, import the dashboard into your workspace. For instructions on importing dashboards, see Import a dashboard file.
Use alerts to track serverless spend
Alerts are a powerful way to stay informed about your serverless spend. With alerts, you can receive notifications when certain conditions are met in your query results. To learn how to create alerts, see Create an alert.
You can add alerts to the following queries to monitor budgets. In each query, replace {budget}
with your chosen budget.
Alert when any workspace spend exceeds a threshold in the last 30 days
You can set an alert to trigger whenever this query returns a row. Replace {budget}
with your chosen budget.
SELECT
t1.workspace_id,
SUM(t1.usage_quantity * list_prices.pricing.default) as list_cost
FROM system.billing.usage t1
INNER JOIN system.billing.list_prices on
t1.cloud = list_prices.cloud and
t1.sku_name = list_prices.sku_name and
t1.usage_start_time >= list_prices.price_start_time and
(t1.usage_end_time <= list_prices.price_end_time or list_prices.price_end_time is null)
WHERE
t1.sku_name LIKE '%SERVERLESS%'
AND billing_origin_product IN ("JOBS", "NOTEBOOKS")
AND t1.usage_date >= CURRENT_DATE() - INTERVAL 30 DAYS
GROUP BY
t1.workspace_id
HAVING
list_cost > {budget}
Alert when a user exceeds the threshold in the last 30 days
You can set an alert to trigger whenever this query returns a row. Replace {budget}
with your chosen budget.
SELECT
t1.identity_metadata.run_as,
SUM(t1.usage_quantity * list_prices.pricing.default) as list_cost
FROM system.billing.usage t1
INNER JOIN system.billing.list_prices on
t1.cloud = list_prices.cloud and
t1.sku_name = list_prices.sku_name and
t1.usage_start_time >= list_prices.price_start_time and
(t1.usage_end_time <= list_prices.price_end_time or list_prices.price_end_time is null)
WHERE
t1.sku_name LIKE '%SERVERLESS%'
AND billing_origin_product IN ("JOBS", "NOTEBOOKS")
AND t1.usage_date >= CURRENT_DATE() - INTERVAL 30 DAYS
GROUP BY
t1.identity_metadata.run_as
HAVING
list_cost > {budget}
Alert when a job exceeds the threshold in the last 30 days
You can set an alert to trigger whenever this query returns a row. Replace {budget}
with your chosen budget.
SELECT
t1.workspace_id,
t1.usage_metadata.job_id,
SUM(t1.usage_quantity * list_prices.pricing.default) as list_cost
FROM system.billing.usage t1
INNER JOIN system.billing.list_prices on
t1.cloud = list_prices.cloud and
t1.sku_name = list_prices.sku_name and
t1.usage_start_time >= list_prices.price_start_time and
(t1.usage_end_time <= list_prices.price_end_time or list_prices.price_end_time is null)
WHERE
t1.sku_name LIKE '%SERVERLESS%'
AND billing_origin_product IN ("JOBS")
AND t1.usage_date >= CURRENT_DATE() - INTERVAL 30 DAYS
GROUP BY
t1.workspace_id, t1.usage_metadata.job_id,
HAVING
list_cost > {budget}
Sample queries
Use the following queries to gain insights into serverless usage in your account:
- Identify expensive serverless compute notebooks
- Identify expensive serverless compute jobs
- Report on DBUs consumed by a particular user
- Report on serverless compute DBUs consumed by workloads that share a custom tag
Identify expensive serverless compute notebooks
This query returns a list of notebooks and how many DBUs each notebook consumed, in descending order by DBU consumption:
SELECT
usage_metadata.notebook_id,
SUM(usage_quantity) as total_dbu
FROM
system.billing.usage
WHERE
usage_metadata.notebook_id is not null
and billing_origin_product = 'INTERACTIVE'
and product_features.is_serverless
and usage_unit = 'DBU'
and usage_date >= DATEADD(day, -30, current_date)
GROUP BY
1
ORDER BY
total_dbu DESC
Identify expensive serverless compute jobs
This query returns a list of jobs and how many DBUs each job consumed, in descending order by DBU consumption:
SELECT
usage_metadata.job_id,
SUM(usage_quantity) as total_dbu
FROM
system.billing.usage
WHERE
usage_metadata.job_id is not null
and billing_origin_product = 'JOBS'
and product_features.is_serverless
and usage_unit = 'DBU'
and usage_date >= DATEADD(day, -30, current_date)
GROUP BY
1
ORDER BY
total_dbu DESC
Report on DBUs consumed by a particular user
This query returns a list of notebooks and jobs that use serverless compute run by a particular user or service principal, and the number of DBUs consumed by each workload:
SELECT
usage_metadata.job_id,
usage_metadata.notebook_id,
SUM(usage_quantity) as total_dbu
FROM
system.billing.usage
WHERE
identity_metadata.run_as = '<emailaddress@domain.com>'
and billing_origin_product in ('JOBS','INTERACTIVE')
and product_features.is_serverless
and usage_unit = 'DBU'
and usage_date >= DATEADD(day, -30, current_date)
GROUP BY
1,2
ORDER BY
total_dbu DESC
Report on serverless compute DBUs consumed by workloads that share a custom tag
This query returns a list of jobs that use serverless compute that share the same custom tag, and the number of DBUs consumed by each workload:
SELECT
usage_metadata.job_id,
usage_metadata.notebook_id,
SUM(usage_quantity) as total_dbu
FROM
system.billing.usage
WHERE
custom_tags.<key> = '<value>'
and billing_origin_product in ('JOBS','INTERACTIVE')
and product_features.is_serverless
and usage_unit = 'DBU'
and usage_date >= DATEADD(day, -30, current_date)
GROUP BY
1,2
ORDER BY
total_dbu DESC
Feedback
https://aka.ms/ContentUserFeedback.
Coming soon: Throughout 2024 we will be phasing out GitHub Issues as the feedback mechanism for content and replacing it with a new feedback system. For more information see:Submit and view feedback for