How to manually cancle a ADF pipline when an activity fails or completes?

Question

Hi,

I have a pipeline which has two concurrent list of activities running at the same time lets say the first activity has a flag which makes the until loop in the second list of activity to complete if flag is true. Now the problem is the flag is set at the end of the pipeline once all the activities in list one is complete in case if an activity before the set flag fails the set flag is false and the until loop runs for infinite time which is not ideal.

Is there a way that I can manually cancel the activity when any one of the activity fails?

Secondly the until loop has an inner wait activity so once it enters the loop it will stay there for some time until it checks for the condition next time. The wait is important so I cant exclude the wait however I would want the until loop to end as soon as the flag is set to true even though the wait is running. Basically i'm trying to end the Until loop.

I did try the steps in MS Docs : Pipeline cancel run : https://learn.microsoft.com/en-us/rest/api/datafactory/pipelineruns/cancel
But this does not work because even when the RUN ID is correct it says incorrect RUN ID.

Could some one please advise how to achieve this?

Answer 1

Thank you for getting back to me @Stramzik . Sorry for the delay, I think we are in different timezones. Also, Q&A ate my previous attempt to post an answer, and its pictures.
It sounds to me like your loop is used exclusively for refreshing auth tokens. Knowing that, and how it is time dependent, I think there is a way to make it more responsive. However, first I will address how to capture activity failures, and then leverage them.

Below I show two ways to capture the case of a failed activity and stop the loop. The first, "Set X to true when something fails", captures only the failure of the first "Do something" activity. (Technically, validation times out, but it is treated similar to a failure.) If the first "Do something" activity succeeds, but a later activity fails, the "Set X to true when something fails" activity will not catch it, because it is connected to only the first activity.
The second way, the activity named "Set X to true when Last or earlier activity fails" will catch the case of the "Last thing to do" activity, or any of its predecessors failing. This is because, assuming all activities in the chain are connected by success dependencies, when an earlier activity fails, subsequent activities are skipped. The "Set X to true when Last or earlier activity fails" is connected to the "Last thing to do" activity by both a skipped dependency and a failure dependency. The failure dependency catches the case of the last activity failing, and the skipped dependency catches the case of the last one being skipped due to an earlier failure.

The first method allows you to take action specific to each individual activity's failure. The second method collects all in a chain.

We can take this capture-and-handle of failures, and build on it to report on the end of the loop, success or fail. In the below image, I show how to capture the ned of the loop, for both the "good end" where everything is successful, and the "bad end" where either the loop itself fails, or one of the "Do something" activities fail.

The "good end" depends on the success of the last "Set X to true" activity and the success of the loop activity. The bad end depends on the success of the "Set X to true when last or earlier activity fails" and the completion (success or failure) of the loop. There is an important, subtle distinction here. Because I chose the completion dependency, I capture both cases of the loop failing, and the loop succeeding. It is also imporant distinction because by choosing the completion dependency instead of the success dependency, the "bad end" will make the pipeline return failure status.
The pipeline returns fail status when "there is a success path, and it is not taken" or, "the last activity executed returns failure". Because completion dependency is not the same as success dependency, this condition is fulfilled.
The "bad end" and "good end" can be replaced by notification logic, like telling logic apps to send and email, or a stored proc updating your SQL.

I mentioned earlier, I think your loop can be made more responsive. The idea behind this, is to lower the wait time while maintaining the same cadence for updating the token, by checking timestamps. While doing this will reduce time waiting for the loop to end, it will increase cost by more activity executions. The choice is yours. You are charged the same for a 15 minute wait activity as a 50 minute wait activity.
The logic involves checking timestamps. At the beginning of the pipeline, or when you get the token the first time, get the current time and store it in a variable. Then inside the loop, reduce the wait time. Inside the loop, check whether (current time) - ( 50 minutes) > (stored timestamp) If less, do nothing, if more, then update timestamp and update the token. This way the duration of each iteration is shortened, so the loop can exit sooner, but the token update still happens at the same rate.

I found a small flaw in my specific implementation, so I removed the details. Let me know if you are interested, and I will get the responsive loop working.