Databricks notebook output exception log to Azure data factory

sakuraime 2,321 Reputation points
2022-01-19T14:58:29.6+00:00

Are there anyway that the Databricks notebook output exception log to Azure data factory ? Like set variable and then log to Azure Log analytics ?

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,943 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,628 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. ShaikMaheer-MSFT 37,971 Reputation points Microsoft Employee
    2022-01-20T18:53:22.003+00:00

    Hi @sakuraime ,

    Thank you for posting query in Microsoft Q&A Platform.

    When we run Databricks notebook from notebook activity. If there are any errors then they will come under notebook activity output json. So from there we can access and set that in to pipeline variable.

    Please check below example for better idea.

    Step1: My ADB notebook. This notebook has sample code which I going to fail.
    166888-image.png

    Step2: Create a variable in your ADF pipeline.
    166848-image.png

    Step3: Notebook activity, to run ADB notebook.
    166857-image.png

    Step4: On failure of Notebook activity call Set Variable activity to set exception value in to pipeline variable.
    166923-image.png

    Notebook activity Output json will hold your exception. Please check below screenshot.
    166866-image.png

    Hope this will help. Please let us know if any further queries.

    --------------

    Please consider hitting Accept Answer. Accepted answers helps community as well.