sending databricks driver logsc (stderr) to azure monitor

Question

Hi Everyone, Thanks in advance for helping out..
I want to send driver logs generated (stderr logs) in databricks to azure monitor, how can I achieve this?
the logs are getting saved in a dbfs location. I have been looking for a solution for 2 weeks now. :(

Answer 1

To send driver logs (stderr logs) from Databricks to Azure Monitor, you need to perform several integrated steps across both platforms. Firstly, it's essential to enable diagnostic logging in your Azure Databricks workspace. This involves configuring your workspace settings to ensure that the driver logs are captured and stored in the DBFS location you mentioned.

Once the logging is configured in Databricks, the next step involves setting up Azure Monitor to collect these logs. You would need to either create a new Log Analytics workspace or use an existing one within Azure Monitor. In this workspace, the configuration focuses on integrating with Databricks. This typically includes installing the Azure Monitor agent on your Databricks clusters, enabling Azure Monitor to fetch logs from Databricks.

The data sources in Azure Monitor need to be set up to point to the DBFS location where your Databricks logs are being saved. This configuration ensures that Azure Monitor can access and retrieve the logs. Additionally, a custom log collection configuration might be necessary. This configuration defines how logs are collected from Databricks, including specifying the log format and setting up any necessary filters to capture the relevant data.

Answer 2

@Antony Micheal - Thanks for the question and using MS Q&A platform.

To send driver logs generated in Databricks to Azure Monitor, you can use the Azure Monitor Logs integration with Databricks. This integration allows you to stream logs from Databricks to Azure Monitor Logs in near real-time.

The below articles helps you to configure Azure Databricks with Azure Monitor.

Send Azure Databricks application logs to Azure Monitor

Comprehensive look at Azure Databricks Monitoring & Logging

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.