Pass value from an array selectively to For Each

Question

Pass value from an array selectively to For Each

Kumar, Arun 336

Hi,

I have a parent pipeline that captures values in an array - ArrayFullLoadData. Currently ArrayFullLoadData have three values [A,B,C] and the same is passed to a ForEach with @pipeline.parameters.ArrayFullLoadData. Inside ForEach, values A, B,C are passed to notebooks and dataflow in a loop.

I need to split the single ForEach run into three ForEach in series, so ForEach1 will take on value A, ForEach2 will take Value B and ForEach3 will take value C. Any suggestions how to do this?

Looking for something like @equals(pipeline.parameters.ArrayFullLoadData, 'A'), but this is not working. Do we have something similar to a where clause like in SQL and pass array value to ForEach?

Thanks,
Arun

AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2024-05-05T07:44:12.99+00:00
Hi Kumar, Arun ,

Thankyou for using Microsoft Q&A platform and thanks for posting your query here.

I understand that you want to split the array into multiple branches based on the value of each element, and then passes each branch to a separate ForEach activity.

You can also use the "Switch" activity in Azure Data Factory to split a single ForEach loop into multiple ForEach loops based on the value of each element in the array.

Here are the steps to split a single ForEach loop into multiple ForEach loops using the "Switch" activity:

Create a "Switch" activity in your pipeline.

Set the "Switch on" property to "@pipeline().parameters.ArrayFullLoadData".

Add a "Case" activity for each value in the array (A, B, and C).

In each "Case" activity, you can add a "ForEach activity" , but ADF won't allow to use foreach inside switch directly, so you can use a child pipeline using "execute pipeline activity" and inside the child pipeline add foreach activity, send the parameter value to child pipeline and set the "Items" property to the corresponding value in the array (A, B, or C).

Add the necessary activities inside each ForEach activity to process the data.

Add a "Default" activity to handle any values in the array that are not A, B, or C.

This allows you to split a single ForEach loop into multiple ForEach loops based on the value of each element in the array.

I hope this helps.
Kumar, Arun 336 Reputation points

2024-05-05T22:14:55.4+00:00

@AnnuKumari-MSFT - Wouldn't the switch activity execute in parallel? I mean all th three child pipelines will execute in parallel correct? I need the ForEach activity to execute in series. For example, execute with array value A first, then array value B, then C.
Kumar, Arun 336 Reputation points

2024-05-05T22:19:12.0933333+00:00

Wouldn't the switch activity execute in parallel. I need ForEach to execute in sequence like ForEach for Array A first, then ForEach for Array B and then ForEach for Array C
Kumar, Arun 336 Reputation points

2024-05-05T23:42:12.9966667+00:00

Switch activity gives error
Kumar, Arun 336 Reputation points

2024-05-07T14:52:56.0333333+00:00

@AnnuKumari-MSFT Ok i fixed the issue. Created three ForEach loops and connected in series. Inside ForEach loop used If Condition to check for array values A, B, C. So first ForEach will run only for A, second for B and third for C
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2024-05-15T11:36:44.3233333+00:00

@Kumar, Arun - I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

Accepted answer

0 additional answers

Your answer

AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2024-05-05T07:44:12.99+00:00

Hi Kumar, Arun ,

Thankyou for using Microsoft Q&A platform and thanks for posting your query here.

I understand that you want to split the array into multiple branches based on the value of each element, and then passes each branch to a separate ForEach activity.

You can also use the "Switch" activity in Azure Data Factory to split a single ForEach loop into multiple ForEach loops based on the value of each element in the array.

Here are the steps to split a single ForEach loop into multiple ForEach loops using the "Switch" activity:

Create a "Switch" activity in your pipeline.

Set the "Switch on" property to "@pipeline().parameters.ArrayFullLoadData".

Add a "Case" activity for each value in the array (A, B, and C).

In each "Case" activity, you can add a "ForEach activity" , but ADF won't allow to use foreach inside switch directly, so you can use a child pipeline using "execute pipeline activity" and inside the child pipeline add foreach activity, send the parameter value to child pipeline and set the "Items" property to the corresponding value in the array (A, B, or C).

Add the necessary activities inside each ForEach activity to process the data.

Add a "Default" activity to handle any values in the array that are not A, B, or C.

This allows you to split a single ForEach loop into multiple ForEach loops based on the value of each element in the array.

I hope this helps.
Kumar, Arun 336 Reputation points

2024-05-05T22:14:55.4+00:00

@AnnuKumari-MSFT - Wouldn't the switch activity execute in parallel? I mean all th three child pipelines will execute in parallel correct? I need the ForEach activity to execute in series. For example, execute with array value A first, then array value B, then C.
Kumar, Arun 336 Reputation points

2024-05-05T22:19:12.0933333+00:00

Wouldn't the switch activity execute in parallel. I need ForEach to execute in sequence like ForEach for Array A first, then ForEach for Array B and then ForEach for Array C
Kumar, Arun 336 Reputation points

2024-05-05T23:42:12.9966667+00:00

Switch activity gives error
Kumar, Arun 336 Reputation points

2024-05-07T14:52:56.0333333+00:00

@AnnuKumari-MSFT Ok i fixed the issue. Created three ForEach loops and connected in series. Inside ForEach loop used If Condition to check for array values A, B, C. So first ForEach will run only for A, second for B and third for C
PRADEEPCHEEKATLA 90,641 Reputation points Moderator

2024-05-15T11:36:44.3233333+00:00

@Kumar, Arun - I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

Answer 1

@Kumar, Arun - I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

Ask: Pass value from an array selectively to For Each

Solution: The issue is resolved. Ok i fixed the issue. Created three ForEach loops and connected in series. Inside ForEach loop used If Condition to check for array values A, B, C. So first ForEach will run only for A, second for B and third for C

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

Share via

Pass value from an array selectively to For Each

0 additional answers

Your answer