How to dynamically use column names from a parameter in ADF Dataflow Derived Column transformation?
I'm working with Azure Data Factory Dataflow and trying to implement a solution where I need to dynamically reference column names based on a pipeline parameter.
I have a parameter called $UpdateColumns
containing a comma-separated list of column names (e.g. 'column1,column2,column3,column4,column5,column6').
In my Dataflow, I'm setting up a Derived Column transformation where I need to dynamically create expressions based on the column names in this parameter.
Let's say I have this 'column1_cnt'. But I want to select 'column1_cnt_avg' as a value if 'column1_cnt' is NULL.
Currently, I have this pattern working in my source query:
iif(isNull($$), toLong(byName(replace('column1_cnt', '_cnt', '_cnt_avg'))), toLong($$))
And I can get the correct data where it's getting me 'column1_cnt' with 'column1_cnt_avg' value.
But then this is hard coded, so I am desperately looking for dynamic solution to this.
I was hoping to do something like below but since under 'Value expression' I think $$ is just the value of that column.
(isNull($$), toLong(byName(replace('$$', '_cnt', '_cnt_avg'))), toLong($$))
So I was wondering and hoping if there is anyway to be able to dynamically access the column name in 'Value expression'
Or I would appreciate it if there could be another way to tackle this problem.
Currently what I have is $UpdateColumns parameter where it contains columns to update 'column1_cnt,column2_cnt,column3_cnt'. But still, I don't know how I would be able to use this to 1) check which column contains null for that row of the data and 2) dynamically get _avg for that column.
Thank you and please let me know if you need a clarification to this.