Azure Data Factory ForEach limit at 5000 items?

Minxiao Zhou 1

Hi,
I am using ForEach activity that loops over more than 7000 items in a Azure Data Factory pipeline.
However, looking at the run history of my pipeline, ForEach only actually covers 5000 of the items.

According to doc: https://learn.microsoft.com/en-us/azure/data-factory/control-flow-for-each-activity, ForEach should have a limit of 100,000 items, not 5000.

Is this a bug?

Thanks.

HarithaMaddi-MSFT 10,136 Reputation points

2020-09-08T08:46:00.107+00:00

Hi @Minxiao Zhou ,

Welcome to Microsoft Q&A Platform. Thanks for posting the query.

There is a support ticket recently opened in June that is based on the same issue where reason is because of limitation in getting list of files using API as in the documentation. Can you please suggest if you are using any other activity prior to foreach activity and share the design of pipeline. Also please share the pipeline run id and region of the data factory for us to investigate further.

Thanks for your patience!
Minxiao Zhou 1 Reputation point

2020-09-08T14:48:21.413+00:00

Thanks.
Here is a run of my pipeline:
https://ms-adf.azure.com/monitoring/pipelineruns/a387f750-5c4b-413b-928d-eee9fc099a65?factory=%2Fsubscriptions%2Ff93c9853-c615-4b55-8244-99e4bfd18257%2FresourceGroups%2FFUN.Klondike%2Fproviders%2FMicrosoft.DataFactory%2Ffactories%2FKlondike-DataFactory

I am using a LookUp activity before the ForEach. That LookUp will have about ~7,600 items total. But the run only had 5010 activities. 10 of the items are outside the ForEach loop, and 5000 is inside.
How do I find out the region of the data factory?

Thanks.

1 answer

HarithaMaddi-MSFT 10,136 Reputation points

2020-09-08T16:30:31.32+00:00

Thanks @Minxiao Zhou for sharing additional details. Lookup activity currently has limitation as mentioned in documentation (snap below), this is the reason why you are seeing only 5000 records processed by foreach activity. Data Factory Region can be found as in second snap below.

Hope this helps! Please let us know for further queries and we will be glad to assist.
Please sign in to rate this answer.

1 person found this answer helpful.
HarithaMaddi-MSFT 10,136 Reputation points

2020-09-10T05:26:04.34+00:00

Hi @Minxiao Zhou ,

We have not heard any response from you. Please let us know if above answer is helpful and do let us know for more queries, we will be glad to assist.

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

HarithaMaddi-MSFT 10,136 Reputation points

2020-09-11T09:42:23.017+00:00

Hi @Minxiao Zhou ,

We still did not hear back from you. Following up to check if above answer is helpful and do let us know any queries that will enable us to assist further.

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Azure Data Factory ForEach limit at 5000 items?

1 answer

Your answer