Error: 'Key columns' expression should return string[], when passing array of parameters in DataFlow with "Delta" as Sink

Question

Hello,

I have set of parquet files that I need to Upsert (Update else Insert) into "Delta" (sink) using Data Flow with mainly following 3 transformations:

SourceParquetFile==>AlterRowUpsert:true()==>SinkDelta (AllowUpsert)

I am trying to make it a generic data flow since for some sources there could be only one key column and for others more than one.
Hence, have defined the parameter for the Data Flow as type String Array = pKColumnsForUpsert (example: ['id', 'name'])

And in the Delta sink, under Key columns - using Custom Expression as:- split(toString($pKColumnsForUpsert), ',') ; where the Update method is selected as Allow Upsert - without any errors.

However, during validation of the data flow it results into error as :- 'Key columns' expression should return string[]

Have already followed and tried following links for suggestions/workarounds which looks like works for other "database" sink types, but probably not with "Delta" as sink type:

https://stackoverflow.com/questions/67163563/passing-the-dataflow-parameter-to-sink-key-column-in-azure-data-factory/67202095#67202095

https://learn.microsoft.com/en-us/answers/questions/133996/how-to-pass-a-parameter-with-more-than-one-key-fie.html

Please help/suggest.

Thanks!!

Answer 1

Hi @ManiK ,

Thank you for posting query in Microsoft Q&A Platform.

I could see your dataflow parameter is already array type, hence we no need to perform split operation here.

By any chance if you wish to use your parameter "pKColumnsForUpsert" as string type, then we will need to perform split operation to take column names from string( 'id','name' string passed in to your parameter) as array values.

Hope this will help. Please let us know if any further queries.

----------------

Please consider hitting Accept Answer. Accepted answers helps community as well.