Understanding pipeline failure

APPLIES TO: Azure Data Factory Azure Synapse Analytics

Error handling

Azure Data Factory and Synapse Pipeline orchestration allows conditional logic and enables user to take different based upon outcomes of a previous activity. Using different paths allow users to build robust pipelines and incorporates error handling in ETL/ELT logic. In total, we allow four conditional paths,

  • Upon Success (default pass)
  • Upon Failure
  • Upon Completion
  • Upon Skip

Screenshot showing the four branches out of an activity.

Common error handling mechanism

Try Catch block

In this approach, customer defines the business logic, and only defines the Upon Failure path to catch any error from previous activity. This approach renders pipeline succeeds, if Upon Failure path succeeds.

Screenshot showing definition and outcome of a try catch block.

Do If Else block

In this approach, customer defines the business logic, and defines both the Upon Failure and Upon Success paths. This approach renders pipeline fails, even if Upon Failure path succeeds.

Screenshot showing definition and outcome of do if else block.

Do If Skip Else block

In this approach, customer defines the business logic, and defines both the Upon Failure path, and Upon Success path, with a dummy Upon Skipped activity attached. This approach renders pipeline succeeds, if Upon Failure path succeeds.

Screenshot showing definition and outcome of do if skip else block.

Summary table

Approach Defines When activity succeeds, overall pipeline shows When activity fails, overall pipeline shows
Try-Catch Only Upon Failure path Success Success
Do-If-Else Upon Failure path + Upon Success paths Success Failure
Do-If-Skip-Else Upon Failure path + Upon Success path (with a Dummy Upon Skip at the end) Success Success

How pipeline failure are determined

Different error handling mechanisms will lead to different status for the pipeline: while some pipelines fail, others succeed. We determine pipeline success and failures as follows:

  • Evaluate outcome for all leaves activities. If a leaf activity was skipped, we evaluate its parent activity instead
  • Pipeline result is success if and only if all nodes evaluated succeed

Assuming Upon Failure activity and Dummy Upon Failure activity succeed,

  • In Try-Catch approach,

    • When previous activity succeeds: node Upon Failure is skipped and its parent node succeeds; overall pipeline succeeds
    • When previous activity fails: node Upon Failure is enacted; overall pipeline succeeds
  • In Do-If-Else approach,

    • When previous activity succeeds: node Upon Success succeeds and node Upon Failure is skipped (and its parent node succeeds); overall pipeline succeeds
    • When previous activity fails: node Upon Success is skipped and its parent node failed; overall pipeline fails
  • In Do-If-Skip-Else approach,

    • When previous activity succeeds: node Dummy Upon Skip is skipped and its parent node Upon Success succeeds; the other node activity, Upon Failure, is skipped and its parent node succeeds; overall pipeline succeeds
    • When previous activity fails: node Upon Failure succeeds and Dummy Upon Skip succeeds; overall pipeline succeeds

Next steps

Data Factory metrics and alerts

Monitor Visually