Azure Databricks queries for monitoring using log analytics

Rohit Boddu 461 Reputation points
2020-10-20T06:30:51.913+00:00

Hi,

I have premium version of azure databricks and i have enabled monitoring as well in this .. so i have sent its all data to log analytics workspace ..

now i want to query on databricks .. can u provide me with written queries so i can use them in my log analytics worksapce ..

1) i want see the executor memory and no of executors running

i have sent its dignostics information to log analytics so now can i monitor azure databricks ?

or

i need to follow this one https://learn.microsoft.com/en-us/azure/architecture/databricks-monitoring/ to monitor azure databricks

and could you please provide me with some common databricks queries for log analytics ..

Regards,
Rohit

Azure Monitor
Azure Monitor
An Azure service that is used to collect, analyze, and act on telemetry data from Azure and on-premises environments.
2,679 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,854 questions
0 comments No comments
{count} votes

Accepted answer
  1. PRADEEPCHEEKATLA-MSFT 73,731 Reputation points Microsoft Employee
    2020-10-20T12:04:24.027+00:00

    Hello @Rohit Boddu ,

    Unfortunately, Azure Databricks does not natively support sending log data to Azure monitor.

    To get metric data related to an Apache Spark job, stages of the job, and tasks, you need to use a library for this functionality is available in GitHub.

    You can use the library enables logging of Azure Databricks service metrics as well as Apache Spark structure streaming query event metrics. Once you've successfully deployed this library to an Azure Databricks cluster, you can further deploy a set of Grafana dashboards that you can deploy as part of your production environment.

    This article shows how to set up a Grafana dashboard to monitor Azure Databricks jobs for performance issues.

    Hope this helps. Do let us know if you any further queries.

    ------------

    • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification.

0 additional answers

Sort by: Most helpful