Databricks Audit Logs, Where can I find table usage information or queries?

Question

Hi,

 

I want to access the Databricks Audit Logs to check the table usage information.

 

I created a Databricks workspace on the premium pricing tier and enabled it for the Unity Catalogue.

I configured Audit logs to be sent to Azure Diagnostic log delivery. What I got in the "Log Analytics Workspace":

I can see table usage information in "DatabricksUnityCatalog “ for tables managed by Unity Catalogue.

After a few days of testing, I noticed that I don't get any logs when I query tables either in a notebook running on a cluster or in the SQL persona.

There is an ActionName "getTable" where I can see table names. But, its timestamp is corresponding to the time when I created a table (not when I queried tables).

I queried tables on Feb 22, 23, and 24th at specific times, but there are no logs related to those actions. Also, I cannot find queries I run in other log tables like "DatabircksSQL".

Please let me know where I can find information about table usage or queries (if there is any).

Also, note that I get logs after at least 1 day, even though it is supposed to be updated every 15 min according to the documentation.

Answer 1

@Mohammad Saber Got the confirmation from the internal team:

You can audit Metadata (Unity Catalog), queries from SQL, and Notebooks - but there's no logging on table level, so you'd have to extract from the logs what table was accessed by whom and when. Consider that in some countries doing this type of analysis (per user basis) may not be allowed.

Hope this helps.