ForEaach Activity runs same element multiple times in Aure datafactory

Question

ForEaach Activity runs same element multiple times in Aure datafactory

DR 1

From Lookup activity taking the output and passing to foreach for Iteration. Set the batch count as 10 and trying to process each item.
However, the foreach activity is processing same element 4 times.
The output from Lookup has :1,2,3...10
Have to create separate files for each item as 1.txt,2.txt ,3.txt....10.txt
Have set the batch count as 4.
Howvever, the foreach activity iterating the first item value 1 four times and trying to create the file 1.txt and hence failing with error as trying to access same file.
Why the Foreach activity processing same item multiple times when Batch count is set?

MartinJaffer-MSFT 26,236 Reputation points

2022-03-08T16:47:25.793+00:00

Hello @DR ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet . In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .

Thanks
Martin

1 answer

Your answer

MartinJaffer-MSFT 26,236 Reputation points

2022-03-08T16:47:25.793+00:00

Hello @DR ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet . In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .

Thanks
Martin

Answer 1

MartinJaffer-MSFT 26,236

Hello @DR and welcome to Microsoft Q&A.

Please help me verify I understand you ask correctly.

You have a source data you are doing a lookup on, data is something like

Num, Text  
1, abc  
2, def  
3 ghi  
4 jkl  
...  
8, tuv  
9, wxy  
10, ZZZ

You want to take each row and create a file for each, named as <Num>.txt . This is done somehow inside a ForEach loop, iterating over the output of the lookup. I'm guessing there is SQL involved somewhere.

The problem you are seeing is processing the same item multiple times. Please let me know if there is a Set Variable activity inside the ForEach loop. The behavior you have described matches some of my experiments with what happens to a Set Variable activity inside a non-sequential loop.

From those experiments, I learned there is only 1 "slot" for a variable. All reads and writes access this same "slot". This means when the simultaneous instances of the loop activate, they can interfere with each other. This is called a "race condition". The main solutions is to either make the loop sequential, or to be clever with the Dynamic Expression so you do not need to use the Variable.

MartinJaffer-MSFT 26,236 Reputation points

2022-03-09T17:43:46.113+00:00
Improving explanation,

suppose we have 2 instances in parallel, each instance tries does a lookup, sets a variable, then uses the variable in a copy activity. The following might happen.

Instance 1 does a lookup, which returns value "ABC"

Instance 2 does a lookup, which returns value "XYZ"

Instance 1 Sets Variable "var" to "ABC"

Instance 2 Set Variable "var" to "XYZ"

Instance 1 uses variable "var" in its Copy, the value of var is "XYZ"

Instance 2 uses variable "var" in its Copy, the value of var is "XYZ"

The output of both Copy activities ends up the same, because they both used the same "XYZ"
Xiang ZHU 40 Reputation points

2024-08-12T14:59:02.9966667+00:00

my solution is to run only pipeline activities inside ForEach loop, if I need to access item sent by ForEach in batch mode. And set the item as pipeline parameter.

Share via

ForEaach Activity runs same element multiple times in Aure datafactory

1 answer

Your answer