Foreach activity not always execute at this number (BatchCount)

Cai Zhang 1 Reputation point Microsoft Employee
2022-07-07T07:08:46.447+00:00

I have a foreach activity that 12 items in it. The Batch count is 10.
I find the activity is not always to run 10 parallel jobs.
It means when one job has been finished, anther pending job will not run at once. It seems will wait for several jobs finished.
What is parallel strategy of foreach activity?
If I want always to run full count parallel jobs, how to update?

218466-image.png

218428-image.png

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. AnnuKumari-MSFT 34,566 Reputation points Microsoft Employee Moderator
    2022-07-07T11:21:09.74+00:00

    Hi @Cai Zhang ,

    Welcome to Microsoft Q&A platform and thankyou for posting your question here.

    As I understand your query, you are trying to execute pipeline within ForEach activity which will have 12 iterations. You want to know how the batch count option in ForEach activity works. Please let me know if my understanding is incorrect.

    First of all, if you do not specify any batch count and keep it blank , by default it sets the value to 50. The activity iterates in parallel with a maximum of 50 concurrent iterations. In your case, if you keep it blank , all 12 iterations will start parallelly.

    Now coming to the scenario you mentioned, the expectation is that once any of the instance gets completed out of the batch of 10 , it will take one more new iteration in its place and perform the execution. It should not wait for all 10 instances to complete in order to start the new instance.

    For more details, kindly check Parallel execution

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.
      Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.