For each activity meant to run for 23 tables but finished only running for 6 tables

Question

I am running a for-each activity, which runs sequentially and is meant to run for all the tables it fetched and post completion of this activity another pipeline activity is set to be triggered. But when the tables are few like 5 it works well but when it gets more than that, it does not run for all the tables and the for each shows succeeded, but the next activity is not triggered because the for-each is not completed for all the tables. How can resolve this?

How can I make it run for all the tables?

Answer 1

I am thinking about 3 major reasons : concurrency limits, timeout issues, or resource constraints.

By default, the ForEach activity in Azure Data Factory runs with a concurrency of 20. If you are running more tables, the remaining ones might be waiting for their turn, leading to timeout or other issues.

You can increase the concurrency in the ForEach activity settings. However, make sure your environment can handle the increased load.

Use the Monitor tab to check the detailed logs of each activity. Look for any failures or timeouts that might be causing the issue.

Also, check if your DF instance or the underlying data sources are facing resource constraints. Scaling the source database can help.