What consitutes pipeline failure?

Question

I have this pipeline:

(It's a simplification of a real pipeline). Execution goes like this:

1. The Fail activity fails (I'm casting a non-numeric string to an int, just btw)
2. The OnFailure activity executes and succeeds
3. The pipeline is reported to have succeeded.

I modify it by adding an OnSuccess activity like this:

Nothing else has changed, so I expect the same behavior as last time. But:

1. The Fail activity fails as before
2. The OnFailure activity executes and succeeds as before
3. The pipeline is reported to have failed.

Is this by design? Is there information somewhere about how pipeline success/failure is determined?

Answer 1

Hi @Anonymous ,

Welcome to Microsoft Q&A Platform.

Pipeline status is considered as "Failed" if any activity in the success flow is skipped when it went in failure flow where as if the activities in Failure flow are skipped, the pipeline will still be "Successful" and this is as per design.

Couple of workarounds to avoid this scenario would be as below

• Adding skipped connector between every success path activity and failure path activity to let ADF know that success path activities can be skipped during failures. (Attached the GIF with skip connector)
• If it is activity other than 'set variable', 'If Condition' activity can be used to check the status of the activity with following example for "Copy Activity"
  activity('Copy_Activity1').output.executionDetails[0].status
• To have parent pipeline and trigger the actual pipeline as child pipeline using "Execute Pipeline" activity and handle error logging in parent pipeline, but this has limitation if specific failed activity information is needed from Child Pipeline that caused the error.

There is a request for adding OR condition between success and failure flows in ADF similar to SSIS. Please consider to upvote the idea if you think its good feature to have. All the feedback you share, is closely monitored by the Product team and implemented in future releases.

35616982-allow-choosing-logical-and-or-logical-or-in-activi

)

Hope this helps! Please let us know for more queries on the same.

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

Answer 2

Hi, and thanks for your prompt response.

I'm not sure your logical AND observation applies here. I know that a single activity dependent on multiple predecessors behaves like this, e.g. C only executes if A and B succeed:

but you're suggesting the reverse: that ADF considers only 'AND' operation between 2 dependent activities from a single activity. That can't be true -- as you see in my original pipeline, OnFailure executed when its dependency condition was met (even though the other dependency condition was not met).

My question isn't about the flow of control within the pipeline, it's about why the whole pipeline is considered failed in the second scenario. Can you give me a clear definition of how ADF determines whether a pipeline has succeeded or failed?

Thanks for looking into this for me