I have set of parquet files that I need to Upsert (Update else Insert) into "Delta" (sink) using Data Flow with mainly following 3 transformations:
I am trying to make it a generic data flow since for some sources there could be only one key column and for others more than one.
Hence, have defined the parameter for the Data Flow as type String Array = pKColumnsForUpsert (example: ['id', 'name'])
And in the Delta sink, under Key columns - using Custom Expression as:- split(toString($pKColumnsForUpsert), ',') ; where the Update method is selected as Allow Upsert - without any errors.
However, during validation of the data flow it results into error as :- 'Key columns' expression should return string
Have already followed and tried following links for suggestions/workarounds which looks like works for other "database" sink types, but probably not with "Delta" as sink type: