Concurrent copy activity executions fails in ADF

Question

Hi, We have a trigger-based pipeline that contains a foreach activity (Batch count 50) with an execute activity within it. The child pipeline also has a foreach activity (Batch count 50) with a copy activity inside, responsible for copying binary files from one blob to another. However, when we start the pipeline, some of the copy activities fail with the following error: "There are substantial concurrent copy activity executions causing failures due to throttling under subscription xxxxxxxxxxxxx, region aue, and a limitation of 3000. Please reduce the concurrent executions. For limits, refer to https://aka.ms/adflimits." It would be greatly appreciated if someone could provide some suggestions on how to handle this case. Thanks

Answer 1

Hi Shikhil S, Welcome to Microsoft Q&A Forum and thanks for posting your query. As per the error message it looks like you have met the Azure subscription limits for concurrent runs as stated in the public documentation.

As these are hard limits, you will have to adjust the pipelines concurrency configuration that way you limit the number of concurrent pipeline runs which would avoid hitting the Azure subscription limits.

Please try updating the concurrency setting to for example 20 or 30 in both parent and child pipeline that way you will ensure that only limited concurrent pipelines runs at the same time.

Hope this info helps. Let us know how it goes. Thank you

---Please do consider to click on "Accept Answer" and "Upvote" on the post that helps you, as it can be beneficial to other community members.