Databricks notebook output exception log to Azure data factory

sakuraime 2,316 Reputation points

Are there anyway that the Databricks notebook output exception log to Azure data factory ? Like set variable and then log to Azure Log analytics ?

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,901 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,444 questions
{count} votes

1 answer

Sort by: Most helpful
  1. ShaikMaheer-MSFT 37,656 Reputation points Microsoft Employee

    Hi @sakuraime ,

    Thank you for posting query in Microsoft Q&A Platform.

    When we run Databricks notebook from notebook activity. If there are any errors then they will come under notebook activity output json. So from there we can access and set that in to pipeline variable.

    Please check below example for better idea.

    Step1: My ADB notebook. This notebook has sample code which I going to fail.

    Step2: Create a variable in your ADF pipeline.

    Step3: Notebook activity, to run ADB notebook.

    Step4: On failure of Notebook activity call Set Variable activity to set exception value in to pipeline variable.

    Notebook activity Output json will hold your exception. Please check below screenshot.

    Hope this will help. Please let us know if any further queries.


    Please consider hitting Accept Answer. Accepted answers helps community as well.