Azure data factory pipeline run status "succeeded" when failed activity is followed by other activity (on failure / on completion)

Question

Azure data factory pipeline run status "succeeded" when failed activity is followed by other activity (on failure / on completion)

Gunta Ledina 20

It seems that Azure data factory has changed its behavior on pipeline status (failed/succeeded) based on its activities. Previously, if activity failed (even if followed by other activities "on failure"/"on completion"), pipeline run also would be marked as "failed". Today, I find that pipeline is marked as "succeeded" if failed activity is followed by another activity "on failure"/"on completion" and if the latter activity succeeds. Is this expected? Is there any documentation that describes this change? E.g., this pipeline's run: ![45914-image.png][1] [1]: /api/attachments/45914-image.png?platform=QnA is marked as succeeded.

Accepted answer

0 additional answers

Your answer

Answer 1

MartinJaffer-MSFT 26,236

Hello anonymous user and welcome to Microsoft Q&A. Thank you for your question.

Please allow me to explain the relation between pipeline successs/fail and activity success/fail.

The reason why the failure of an activity does not necessarily force the failure of the pipeline, are the use cases where you might expect something to fail, but want to handle it with a retry or alternative business logic.

The success/failure of a pipeline depends upon the dependencies attached to a failing activity. The shortest way to explain this is "If there is a success path, and that path is not taken, then the pipeline status is failure."

If an activity fails, and there are no dependent activities following it, then the pipeline fails. (In other words, if the last activity in a pipeline fails, the pipeline fails.)

If an activity fails, and it is followed only by another activity connected by a green on-success path, then the pipeline fails.

If an activity fails, and it is followed only by another activity connected by a red on-failure path, then the pipeline succeeds.

If an activity fails, and it is followed only by another activity connected by a blue on-completion path, then the pipeline succeeds.

If an activity fails, and it is followed by two activities, one connected by a red on-failure path, the other connected by a green on-success path, then the pipeline fails.

If an activity fails, and it is followed by two activities, one connected by a red on-failure path, the other connected by a blue on-completion path, then the pipeline succeeds.

Saurabh Sharma 23,846 Reputation points Microsoft Employee Moderator

2020-12-10T22:34:02.57+00:00

anonymous user Please let me know if you find above reply useful. If yes, do click on 'Mark as answer' link in above reply. This will help other community members facing similar query to refer to this solution. Thanks.
Gunta Ledina 20 Reputation points

2020-12-10T23:13:34.427+00:00

Thanks, it makes sense!
Psychotechnopath 51 Reputation points

2023-01-10T09:05:58.413+00:00

Hello, I'm running into issues regarding this. Take a look at the following inner pipeline in a for-each loop:

As you can see, the CheckAccessTokenValidity activity fails in this case, but it is handled gracefully and pipeline succesfully completes because the failure was properly anticipated and handled. However, in my outer pipeline run it still shows the pipeline as failed.

Why is this? How do I circumvent or disable this behavior, as it is unintended (I want my pipelines to show status "success" if they sucessfully executed last activity in it):
Rasmus Mellergaard Pedersen 0 Reputation points

2023-04-28T08:32:13.9433333+00:00

@Psychotechnopath did you manage to have your pipelines showing status "succeeded" if the pipeline successfully executed the last activity?
John Edwards 35 Reputation points

2024-06-05T15:51:46.9066667+00:00

This answer is helpful, but it's describing a stupid decision by the ADF team.

If an error is handled, it's handled. That's intuitive. Your On-Completion option should NEVER be accompanied by an On-Success or On-Failure option, because it overlaps with each. Execution should be On-Success, On-Failure, both On-Success and On-Failure (since they are fully exclusive of each other), or On-Completion on its own.

Unless you want to spawn a second thread, there should not be overlapping options. On-Error and On-Completion should produce two execution paths. I wouldn't provide the option for that, but if you do, it should spawn two threads of execution.

Does it? Or is the presumption that the On-Failure option goes first and trumps the On-Completion option? That it turns off an unconditional execution path?

The logic you describe above sounds like a decision made by an engineer that thought too much and talked themselves into a contorted logic corner.

Failure conditions are sometimes part of standard business process (polling in particular lives on this). If the final activity that is executed ends gracefully then the overall result should be Succeeded. If an error is handled, it's handled. Frankly, ADF should have activity options for End-With-Success and End-With-Error so that the developer can craft the solution as they see fit.

Share via

Azure data factory pipeline run status "succeeded" when failed activity is followed by other activity (on failure / on completion)

0 additional answers

Your answer