Can I get cluster logs in an external storage account?

Question

Hi,

I have two questions about cluster logs.

Q1) It seems that I can only store logs on DBFS. But, is there any way I can send the cluster logs to an Azure storage account?

Note that workspace is not enabled for the Unity Catalog (and is not aimed to be enabled based on the my client's request.).

Q2) We can query tables in the notebook running on the cluster (different from the query editor in SQL persona).

I am interested in table usage information (which user queried a specific table at what time).

Do the logs contain table usage information?

Answer 1

@Mohammad Saber Thanks for the question and using MS Q&A platform.

You can use send cluster logs to Azure Storage account by mounting the storage account and specific the cluster logs location as the mounted storage account.

Step1: Mount the storage account as metioned in the document: Access Azure Data Lake Storage Gen2 and Blob Storage

Step2: Update the logging location pointing to the storage account mounted in step1.

Azure Databricks provides several ways to get information about table usage in the SQL persona. One way is to use the query history feature. The query history feature provides information about the queries that have been run in the past, including the tables that were used in those queries. You can use this information to understand how your tables are being used and to optimize your queries for better performance

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.