How is ForEach activity declaring success in Data Factory?

Question

In the following scenario we have a ForEach activity running in an Azure Data Factory pipeline to copy data from source to destination. The last CopyActivity took 4:10:33 but the ForEach activity declared Succeeded 36 Minutes later: 4:46:12. The question is, why ForEach activity need this 36 Minutes extra? Is it the case that the ForEach needs also to consolidate results from subactivities before declaring success or fail?

Answer 1

Hi @Georgian Pirvu ,

After having conversation with internal team, below are the findings:

ForEach activity does wait for all inner activity runs to complete. In theory, there should not be much delay on marking foreach run success after the last activity run within it succeed. However, ADF rely on partner service to execute the runs and it's possible that the partner service run into failures and could not complete foreach in time. They have build in logic to keep retry and recover but the behavior in ADF activity runs is delay. It's also possible that orchestration service fails and partner service keep retry on calling us. but usually partner service delay is the main cause here.

Are you noticing this behavior consistently or intermittently? If you notice this behavior consistently then we need to file a support ticket for deeper analysis.

Please let me know if this info helps or if you would want to proceed with Support ticket.

----------

Thank you
Please do consider to click on "Accept Answer" and "Upvote" on the post that helps you, as it can be beneficial to other community members.