How to send custom log messages from Azure Databricks Notebook to AppInsights?

DevelKP 1

Hi,

I have a working Notebook (Scala) in Azure Databricks. Now I want to create custom log messages and send them to Azure Log Analytics and view the log messages in Azure Application Insights.

Until now I'm using a custom logger to write log messages to 'log4j-active.log'.

import org.slf4j.{Logger, LoggerFactory}    
val alogger: Logger = LoggerFactory.getLogger("MyLogger")

Can you please guide me on what I have to change to have these messages sent to Azure Log Anayltics?

Greetings and thanks

PRADEEPCHEEKATLA-MSFT 77,081 Reputation points Microsoft Employee

2022-04-27T11:05:21.44+00:00
Hello @DevelKP ,

Following up to see if the below suggestion was helpful. And, if you have any further query do let us know.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you.

2 answers

Jeffrey Lai 21 Reputation points Microsoft Employee

2022-11-28T10:37:08.903+00:00

Hi @DevelKP ,

Do we have any other alternatives to do logging if this method is not supporting ADB 11.x?
Please sign in to rate this answer.

1 person found this answer helpful.

0 comments No comments
Sign in to comment
PRADEEPCHEEKATLA-MSFT 77,081 Reputation points Microsoft Employee

2022-04-21T13:37:55.5+00:00
Hello @DevelKP ,

Thanks for the question and using MS Q&A platform.

This article - Send Azure Databricks application logs to Azure Monitor shows how to send application logs and metrics from Azure Databricks to a Log Analytics workspace. It uses the Azure Databricks Monitoring Library, which is available on GitHub.

Hope this will help. Please let us know if any further queries.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

Want a reminder to come back and check responses? Here is how to subscribe to a notification

If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
Please sign in to rate this answer.
DevelKP 1 Reputation point

2022-04-22T12:32:39.213+00:00

Thank you for the link. I managed to create both of the JAR files and uploaded them. I also uploaded the shell script for the cluster initialization. Now I want to upload the 'log4j.properties' file as described in step 2 in the link you have posted. But, as I said, I'm working with a Azure Databricks notebook. With what should

<your application package name>

be replaced then? Also, looking at step 4, how should that be adjusted to make it work from inside a notebook?

PRADEEPCHEEKATLA-MSFT 77,081 Reputation points Microsoft Employee

2022-04-25T06:22:19.88+00:00

Hello @DevelKP ,

This information is already mentioned in the documentation.

You can find a sample configuration file here.

log4j.appender.A1=com.microsoft.pnp.logging.loganalytics.LogAnalyticsAppender log4j.appender.A1.layout=com.microsoft.pnp.logging.JSONLayout log4j.appender.A1.layout.LocationInfo=false log4j.additivity.com.microsoft.pnp.samplejob=false log4j.logger.com.microsoft.pnp.samplejob=INFO, A1

For more details, refer to Configure Azure Data bricks to send events to Application insights — Simplified.

Hope this will help. Please let us know if any further queries.

PRADEEPCHEEKATLA-MSFT 77,081 Reputation points Microsoft Employee

2022-04-28T12:06:02.497+00:00

Hello @DevelKP ,

Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

mpsr2022 1 Reputation point

2022-10-07T13:17:35.697+00:00

Hi @PRADEEPCHEEKATLA-MSFT

The proposed solutions does not work on Databricks Runtime 11.x, because the library involved is not supported anymore.

DBR 11.0: Not currently supported due to changes in Log4j version

1

Library does not support Log4j 2

2

Do you have any details on how to achieve this on a cluster running DBR 11.x?

Thanks!
Sign in to comment