How to send custom log messages from Azure Databricks Notebook to AppInsights?

DevelKP 1 Reputation point


I have a working Notebook (Scala) in Azure Databricks. Now I want to create custom log messages and send them to Azure Log Analytics and view the log messages in Azure Application Insights.

Until now I'm using a custom logger to write log messages to 'log4j-active.log'.

import org.slf4j.{Logger, LoggerFactory}    
val alogger: Logger = LoggerFactory.getLogger("MyLogger")

Can you please guide me on what I have to change to have these messages sent to Azure Log Anayltics?

Greetings and thanks

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,391 questions
{count} votes

2 answers

Sort by: Most helpful
  1. Jeffrey Lai 11 Reputation points Microsoft Employee

    Hi @DevelKP ,

    Do we have any other alternatives to do logging if this method is not supporting ADB 11.x?

    1 person found this answer helpful.
    0 comments No comments

  2. PRADEEPCHEEKATLA-MSFT 59,326 Reputation points Microsoft Employee

    Hello @DevelKP ,

    Thanks for the question and using MS Q&A platform.

    This article - Send Azure Databricks application logs to Azure Monitor shows how to send application logs and metrics from Azure Databricks to a Log Analytics workspace. It uses the Azure Databricks Monitoring Library, which is available on GitHub.

    Hope this will help. Please let us know if any further queries.


    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators