ADF Dataflow - dervied column, Projection, Schema

Question

Hi,

I have observed that ADF dataflow derived column or any other transformations require projection to be available in previous transformations.

For eg, I have a derived column after source. If I have to add a derived column 'xyz' as 'column1+5', column1 is expected to be in projection in source transformation. I want to develop this dataflow as dynamic, not specific to any specific source file. my source file could be test.1 csv or test2.csv. It should automatically pick up columns from source and I should be able to add transformations dynamically.

Currently it gives an error If we try to add a derived column for the second file, as the projection is not available for this. How can I make my DF transformations dynamic.

Answer 1

I am able to fix this. We should be utilizing byPosition(), columnNames() expression, instead of column names.

And we can pass expressions as parameters.

two sample expressions passed as DF parameters

columnnames = toString(columnNames()[1]) + '|' + toString(columnNames()[2]) + '|' + toString(columnNames()[4]) + '|' + toString(columnNames()[5]) + '|' + toString(columnNames()[6]) + '|' + toString(columnNames()[7])

column1data = '<row>' + toString(byPosition(1) + '|' + 'testcolumn</row>'