Aggregate transformation in Mapping data flow

Question

Hi Team,

I am building an incr load.

Now, to get the high watermark value, I decided to get it from the files (parquet) that will be pulled in the staging datalake location in every load as I do not want to hit source-Oracle to get the max value.

I am trying to get max() of timestamp column from dataflow aggregate transformation . But instead of value , I am getting the column name itself.

Kindly suggest what am I missing to get the max values.

Dataflow parameters:

Answer 1

@Vaibhav ,

Thankyou for using Microsoft Q&A platform and thanks for posting your query here.

I understand that you are trying to get the max value of a timestamp column from a dataflow aggregate transformation .

To pass in a column name as a parameter value, kindly use byname() function wrapped in toString() and pass the parameter containing the columnname within. If the column is defined in the data flow schema, you can reference it directly as a string expression. If the column isn't defined in the schema, use the byName() function. Remember to cast the column to its appropriate type with a casting function such as toString().

For example, if you wanted to map a string column based upon a parameter columnName, you can add a derived column transformation equal to toString(byName($columnName)).

Note

In data flow expressions, string interpolation (substituting variables inside of the string) isn't supported. Instead, concatenate the expression into string values. For example, 'string part 1' + $variable + 'string part 2'

For more details, kindly check the official documentation: https://learn.microsoft.com/en-us/azure/data-factory/parameters-data-flow#passing-in-a-column-name-as-a-parameter

Hope it helps . Kindly accept the answer by clicking on Accept answer button. Thankyou