Databricks notebook output exception log to Azure data factory

sakuraime 2,326 Reputation points
2022-01-19T14:58:29.6+00:00

Are there anyway that the Databricks notebook output exception log to Azure data factory ? Like set variable and then log to Azure Log analytics ?

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,218 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,841 questions
{count} votes

1 answer

Sort by: Most helpful
  1. ShaikMaheer-MSFT 38,456 Reputation points Microsoft Employee
    2022-01-20T18:53:22.003+00:00

    Hi @sakuraime ,

    Thank you for posting query in Microsoft Q&A Platform.

    When we run Databricks notebook from notebook activity. If there are any errors then they will come under notebook activity output json. So from there we can access and set that in to pipeline variable.

    Please check below example for better idea.

    Step1: My ADB notebook. This notebook has sample code which I going to fail.
    166888-image.png

    Step2: Create a variable in your ADF pipeline.
    166848-image.png

    Step3: Notebook activity, to run ADB notebook.
    166857-image.png

    Step4: On failure of Notebook activity call Set Variable activity to set exception value in to pipeline variable.
    166923-image.png

    Notebook activity Output json will hold your exception. Please check below screenshot.
    166866-image.png

    Hope this will help. Please let us know if any further queries.

    --------------

    Please consider hitting Accept Answer. Accepted answers helps community as well.


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.