How to send an array between a pipeline and a dataflow?

mattssok 1 Reputation point
2021-05-10T18:45:24.593+00:00

I have posted a similar issue here (https://stackoverflow.com/questions/67466310/dynamic-column-masking-in-azure-with-derived-column) but since I have refined my problem a bit I'll try here.

I want to send a an array with column names from my pipeline to be used as a matching array in the in() function within my data flow. In raw, returned from my SQL query, this array looks like:

MyArrayParamPipeline

"value": [  
    {  
        "ColumnName": "a"  
    },  
    {  
        "ColumnName": "b"  
    },  
    {  
        "ColumnName": "c"  
    }  
]  

For some reason I cannot use this in my data flow. When I go from MyArrayParamPipeline to MyArrayParamDF I do like this: MyArrayParamDF = @ hide (MyArrayParamPipeline) that resolves my errors but I still can't use it in my data flow.

When I have searched around for some answer I found https://stackoverflow.com/questions/64929282/how-to-pass-an-array-parameter-into-an-adf-dataflow but I was not able to resolve my issues in any of those ways. When I applied those I got these 'raw' results going in to my data flow:

1) "MyArrayParamDF ": "array({\"ColumnName\":\"a\"},{\"ColumnName\":\"b\"},{\"ColumnName\":\"c\"})"
2) "MyArrayParamDF ": "['{\"ColumnName\":\"a\"}','{\"ColumnName\":\"b\"}','{\"ColumnName\":\"c\"}']" (here I removed the split suggestion since it did not work with array)

None of those worked as intended. So how can I send string-arrays from a pipeline to a data flow and still be able to use them?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,196 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. Kiran-MSFT 691 Reputation points Microsoft Employee
    2021-05-11T00:56:15.53+00:00

    Pass it as a formatted string ['a', 'b', 'c']. This is the dataflow DSL syntax. Then use the expr(<string>) function on the parameter passed to evaluate within the dataflow to make it an array of strings.