ADF - need status of actual activity failure in the pipeline status

Question

I am getting the status of the pipeline using the rest API. I get the below status and message.

"status": "Failed",
"message": "Operation on target For every file failed: Activity failed because an inner activity failed"

But when I go into the pipeline and check the activity that failed, I see the below message :
{
"errorCode": "2200",
"message": "ErrorCode=UserErrorFileNotFound,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Cannot find the file specified. Folder path....,
"failureType": "UserError",
"target": "Copy to destination",
"details": []
}

Is it possible to get the error message of the activity directly from the pipeline status, so that it will add value as to what the actual problem is. Currently the pipeline status simply states that "Activity failed because an inner activity failed".

Answer 1

I am unable to change the behavior of the final pipeline error message, but I do have two options for you.

1. Since you are making API calls, I can assume you are capable of implementing custom logic. In the current call you are using, the error message that an error does occur. You can then make a call to the queryActivityRuns as was discussed in the other thread I helped you with today. The Activity runs will get ALL the activities, including the inner ones. Then filter them by "status".

----------

2. Another options is to take advantage of the way success / failure is determined. It is possible to engineer an activity to fail. It is also possible to capture error messages for later use in the pipeline. On the inner loop activities (and wherever else you think it is likely to fail) connect an Append Variable activity via the failure dependency.

If you have a loop, you should use an array type variable and the append variable activity, so if more than one activity fails, you can capture all of them.

Then at the end of the pipeline, place a Set Variable activity. This one we will engineer to fail if our array contains any error messages, and succeed if it is empty. Because of the way we engineer to fail, the input (the array contents) will be displayed in the final error message. This final Set Variable activity should use a boolean or integer to cause a type error.

It is important that the finall Set Variable activity, should it follow a container type activity, be connected by on-completion or on-success dependency. This is because with the way we captured the inner errors, the container activity may no longer report itself as failed. Please let me know if you would like to know the reasoning behind this.

For the final Set Variable activity, I used the following expression:
@if(empty(variables('errors')),true,variables('errors'))

Please let me know if this helps, or I missed anything.
Thank you for your patience and curiosity
Martin

Answer 2

You can capture the error message from the activity object of the specified activity. Normally you look at the output of an activity using activity('my step').output, but you can get much more from it. In particular the error object is in activity('my step').error and the error message is in activity('my step').error.message.

Answer 3

Hello @Kothai Ramanathan and thank you for your question. I hope you do not mind if I provide some explanation before answering.

This particular case only happens for "container" activities. ForEach, Until, Switch ... these are among the activities which "contain" other activities, like a miniature sub-pipeline. The reason the originating "inside" activity's error is not reported at top-level, is because there could be multiple activities failing, each with different error messages. To avoid a 100 iteration loop spamming you with errors at top level, the container reports this error message.

It is also possible for an inner activity to fail, but not cause the container activity (or pipeline) to fail. This is because each inner sequence uses similar criteria as pipeline to determine whether the inner sequence as a whole fails or succeeds. I can talk more about this if you like.

I will provide a solution or work-around in a separate message/post.