Controlling Notebook Termination on Exception

Lavanya Bhajanthri 200 Reputation points
2023-11-14T20:30:27.72+00:00
I'm attempting to create a query in Data Factory to write to a table. The query and functions are functional, but I want the notebook to abort if an exception occurs. The current code snippet displays an error message in the notebook and halts execution. However, in Data Factory, it still registers as 'Succeeded' rather than 'Failed.' I'm seeking assistance to ensure the status reflects as 'Failed' in Data Factory. Here is the existing code snippet:

```python
try:
    df = spark.sql(query)
    df.write.format(formato).mode(method).saveAsTable(table)
except Exception as e:
    msg = "An error occurred while writing the query to the table: {}".format(e)
    dbutils.notebook.exit(msg)
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,833 questions
0 comments No comments
{count} votes

Accepted answer
  1. phemanth 11,200 Reputation points Microsoft Vendor
    2023-11-15T11:00:39.51+00:00

    @Lavanya Bhajanthri

    Thanks for reaching out to Microsoft Q&A.

    I tried the same with your code and my Notebook is also gave me Success.

    enter image description here

    Here, you are handling the error and passing a message to the notebook using.

    dbutils.notebook.exit(msg).

    Notebook won't fail when it exited with a message but the databricks notebook execution will be aborted.

    If you want to get the reason(Message) causing the abort of the databricks notebook in the ADF pipeline, use the below expression after the Success of the notebook activity.

    @activity('Notebook1').output.runOutput

    User's image

    (OR)

    If you want Notebook activity to fail, you need to generate the error and abort the notebook like below sample.

    try:

    try:
        df = spark.sql(query)
        df.write.format(formato).mode(method).saveAsTable(table)
    except Exception as e:
        msg = "An error occured while writing the query to the table: {}".format(e)
        logging.error(msg)
        raise
    

    Now, to get the error message in ADF, use the below expression after the Failure of the Notebook activity.

    @activity('Notebook1').output.runError

    enter image description here

    Hope this helps. Do let us know if you any further queries.


    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

    0 comments No comments

0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.