ADF: complex parameter reference

Question

ADF: complex parameter reference

Tal Zadok 21

I have 2 pipelines: master and child.
master pipeline contains a foreach loop which calls the child pipeline (execute pipeline activity).
the child pipeline contains a single activity of data explorer command.
foreach item is being passed as parameter to child pipeline.

1 - in master pipeline, how can I define an array of complex objects to iterate in foreach?
using square brackets with complex objects raised an error.
probably something like:
@createArray( json('{"cloudEnv": "ffx", "states": ["a", "b"]}'), json('{"cloudEnv": "mck", "states": ["c"]}')) ?

2 - how can I reference a sub-field (e.g "states") in the child pipeline, within the kusto query?
tried "mytable | where State in [pipeline().parameters.cloudobj].states" but that didn't work.

Thanks

MartinJaffer-MSFT 26,161 Reputation points

2021-03-23T16:52:41.13+00:00

Hello @Tal Zadok and welcome to Microsoft Q&A.

I think I understand ask #1. For ask #2 do you mean, how to reference and place inside a usable query?
MartinJaffer-MSFT 26,161 Reputation points

2021-03-23T17:10:38.223+00:00

I believe I have reproduced the error of #1.

When running the child directly, with array type parameter and value [{"name":"foo","value":[2,3]},{"name":"var","value":[7,8]},{"name":"baz","value":[23,1]}] it runs fine. But then when I try to pass the same value through execute pipeline, I get error

Operation on target ForEach1 failed: The execution of template action 'MainForEach1' failed: the result of the evaluation of 'foreach' expression '@pipeline().parameters.asarray' is of type 'String'. The result must be a valid array.

Is this the same error you get, @Tal Zadok ?
MartinJaffer-MSFT 26,161 Reputation points

2021-03-23T17:16:42.26+00:00
In the execute pipeline, I wrapped the value in @json('stuff_goes_here') and it worked. I will inquire/provide feedback to the product group about this.

@json('[{"name":"foo","value":[2,3]},{"name":"var","value":[7,8]},{"name":"baz","value":[23,1]}]')
Tal Zadok 21 Reputation points

2021-03-29T15:22:18.05+00:00

Thanks

can you please specify how do you reference "value" in the child pipeline within an Azure Data Explorer query?
let's say I want to use "2,3" as part of the query (see below)

my configurations:
master pipeline:

"Items" in foreach activity: @json ('[ {"cloudEnv": "ffx", "states": ["a", "b"]}, {"cloudEnv": "mck", "states": ["c"]}]')"

child pipeline:
pipeline parameter:

ADE command:
myTable | where State in [pipeline().parameters.cloudobj].states

the syntax is following this example: https://learn.microsoft.com/en-us/azure/data-factory/control-flow-expression-language-functions#complex-expression-example

output error:

thanks
Nasreen Akter 10,896 Reputation points Volunteer Moderator

2021-03-29T16:13:30.32+00:00
Hi @Tal Zadok ,

The following steps could help:

in the parent pipeline, make the cloudobj parameter as an Array and put the value [{"cloudEnv": "ffx", "states": ["a", "b"]}, {"cloudEnv": "mck", "states": ["c"]}]

while invoking child pipeline pass @markus.bohland@hotmail.de ('("', join(item().states, '", "'), '")') instead of @item()

in the child pipeline, change the cloudobj parameter as a String type instead of object

In the ADE command use: myTable | where State in @{pipeline().parameters.cloudobj}

Hope this helps! Thanks! :)
Tal Zadok 21 Reputation points

2021-03-29T19:01:29.357+00:00

Thanks! that did the trick.

Just to make sure I got it right: I defined that complex object as a pipeline parameter with default value, then used that parameter under "Items" of ForEach activity. Is that what you meant in #1 above?
Is there a way to define that object in-place under "Items"? as it's not really needed for the whole pipeline..

Thanks
Nasreen Akter 10,896 Reputation points Volunteer Moderator

2021-03-29T19:36:53.17+00:00

Hi @Tal Zadok , yes you got it right. :)

I could not think of any other option with the foreach ITEM section. :(
MartinJaffer-MSFT 26,161 Reputation points

2021-04-12T16:20:29.167+00:00

It is possible to write in the for-each items sections, @Tal Zadok @Nasreen Akter . Whatever you put inside must be iterable, so wrapping in a @json('{"my_stuff":6}') or @array([6,2,34,-19]) may help