Azure Databricks queries for monitoring using log analytics

Question

Hi,

I have premium version of azure databricks and i have enabled monitoring as well in this .. so i have sent its all data to log analytics workspace ..

now i want to query on databricks .. can u provide me with written queries so i can use them in my log analytics worksapce ..

1) i want see the executor memory and no of executors running

i have sent its dignostics information to log analytics so now can i monitor azure databricks ?

or

i need to follow this one https://learn.microsoft.com/en-us/azure/architecture/databricks-monitoring/ to monitor azure databricks

and could you please provide me with some common databricks queries for log analytics ..

Regards,
Rohit

Answer 1

Hello @Rohit Boddu ,

Unfortunately, Azure Databricks does not natively support sending log data to Azure monitor.

To get metric data related to an Apache Spark job, stages of the job, and tasks, you need to use a library for this functionality is available in GitHub.

You can use the library enables logging of Azure Databricks service metrics as well as Apache Spark structure streaming query event metrics. Once you've successfully deployed this library to an Azure Databricks cluster, you can further deploy a set of Grafana dashboards that you can deploy as part of your production environment.

This article shows how to set up a Grafana dashboard to monitor Azure Databricks jobs for performance issues.

Hope this helps. Do let us know if you any further queries.

------------

• Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
• Want a reminder to come back and check responses? Here is how to subscribe to a notification.