Azure Databricks, How to get usage statistics from Databricks or SQL Databricks?

Question

Hi,

 

I am looking for a way to get usage statistics from Databricks (Data Science & Engineering and SQL persona).

 

For example:

• I created a table. I want to know how many times a specific user queried that table.
• How many times a pipeline was triggered?
• How long it took to run a DLT pipeline?

 

Is there any way to get usage statistics?

Answer 1

Hi Mohammad Saber

Thanks for using Microsoft Q&A forum and posting your query.

You can get usage statistics from Azure Databricks by several ways.

1. Databricks Rest API : REST API can also be used to retrieve usage statistics for SQL Databricks. You can retrieve information about query history, query run time, and other statistics. Please refer to this document for more information: Query History API 2.0.
2. Databricks UI: You can use Databricks UI to retrieve usage statistics. You can filter the list by user, service principal, date range, SQL warehouse, and query status. Please refer to this document for more information: Query history
3. Audit logs: If you have a Premium plan, then Databricks provides access to audit logs of activities performed by Databricks users, allowing your enterprise to monitor detailed Databricks usage patterns. Ref doc : Databircks Configure audit logging.
4. For DLT Pipeline monitoring: An event log is created and maintained for every Delta Live Tables pipeline. The event log contains all information related to the pipeline, including audit logs, data quality checks, pipeline progress, and data lineage. You can use the event log to track, understand, and monitor the state of your data pipelines. The event log for each pipeline is stored in a Delta table in DBFS. You can view event log entries in the Delta Live Tables user interface, the Delta Live Tables API, or by directly querying the Delta table. For more info please ref to this doc: Monitor pipelines with the Delta Live Tables event log

Hope this info helps.

---

Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.