ADF Foreach Activity- Suggest on settings to exclude successful iterations in rerun

Goutham Kolasani 0 Reputation points Microsoft Employee
2023-01-26T19:52:50.69+00:00

Now, you can see where your pipeline is failed it allows you rerun from that activity, which includes next iteration in a for loop which is successful.

Behavior for re-run is valid in case of synchronization option, but in parallel scenario it's not correct so was curious if I missed something.

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,248 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,334 questions
{count} votes

1 answer

Sort by: Most helpful
  1. AnnuKumari-MSFT 30,101 Reputation points Microsoft Employee
    2023-01-27T13:50:40.6966667+00:00

    Hi Goutham Kolasani ,

    Welcome to Microsoft Q&A platform and thanks for posting your question here.

    As I understand your scenario, you are trying to rerun the ADF pipeline from failure. However , the pipeline contains nested activity like foreach loop and isSequential is not checked which means inner activities are running in parallel and rerun is not helping . Please let me know if that is what your ask is.

    In case of ForEach activity, the rerun would skip the inner activities if it has already been executed based on the rerun rules.

    Suppose there are 100 expected iterations of inner activity , and the pipeline failed for 10 of them , then rerun would call all the 100 iterations again , however, 90 of them will be skipped. Means , it won't execute . You can check the status of those activities coming as skipped.

    Check out the rerun behaviour of activities in the below image:

    User's image

    Instead of just relying on rerun from failure at pipeline level, You can configure a retry policy for inner activities in your pipeline. This will automatically retry the activity if it fails, which can help to mitigate the impact of transient failures.User's image


    Hope it helps. Please do consider clicking Accept Answer and Up-Vote for the same as accepted answers help community as well.

    0 comments No comments