AzureDF IfCondition in ForEach activity not acting as expected

Question

Hi all,

I am new to Azure and I am testing on data factory.

I have a container which stored three CSV files. I would like to build a pipeline that copy the files to different output container base on the filename.

I added a Get Metadata activity and connect it to a ForEach activity

In the ForEach activity, I first set the filename to a variable and pass the variable to if Condition

The if Condition test on the below condition:

@startswith(variables('filename'), 'emp')

And the true/false activity are setting another variable which display the target copy location

I ran a debug and found that the three output are all "/dep/department.csv" while I am expecting "/emp/emp_1.csv", "/emp/emp_2.csv", "/dep/department.csv"

Please help me out, thank you in advance.

Answer 1

There is a limitation in forEach activity if you use set variable inside of it.
Please make sure you have follow this workaround mentioned in the document by disabling the parallel setting inside of forEach activity.

https://learn.microsoft.com/en-us/azure/data-factory/control-flow-for-each-activity#limitations-and-workarounds

Answer 2

It seems like the issue is with how the IfCondition activity is evaluating the condition within the ForEach loop in your pipeline.

First step you need to verify that within the ForEach activity, you are correctly setting the filename variable. You can do this with a Set Variable activity:

• Set Variable activity:
  • Name: SetFilename
  • Variable: filename
  • Value: @item().name (assuming name is the property that contains the filename)

You can add a Set Variable activity or a logging mechanism to confirm that the filename is being set correctly. For example:

• Set Variable activity:
  • Name: LogFilename
  • Variable: log
  • Value: @variables('filename')

I would go for setting up your IfCondition like below:

• If Condition activity:
  • Condition: @startswith(variables('filename'), 'emp')
  • True activities: Set variable to /emp/ location
  • False activities: Set variable to /dep/ location