How to monitor Azure Databricks cluster event logs

Question

Hi,

We need the event logs from Azure Databricks Cluster to monitor its health:

For example, if the metastore or dbfs goes down, actions should be taken.

My team followed the Azure Docs and set up the Databricks Diagnostic logging in Azure monitor. But we couldn't find these event logs from the Databricks Diagnostic logging in Azure monitor.

So is there a way for us to monitor the cluster's event logs in Azure Monitor?

Thank you!

Answer 1

Hi @Fangzhou Zhang ,

Thank you for posting your query on Microsoft Q&A Platform.

I did research on this and found that Unfortunately, Azure Data bricks does not natively support sending event logs data to Azure monitor.

To get metric data related to an Apache Spark job, stages of the job, and tasks, you need to use a library for this functionality is available in GitHub.

Kindly check below link there similar kind of issue discussed.
https://learn.microsoft.com/en-us/answers/questions/132025/azure-databricks-queries-for-monitoring-using-log.html

Hope this will help. Please let us know if any further queries. Thank you.

---------------------------------

• Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
• Want a reminder to come back and check responses? Here is how to subscribe to a notification.