Synape Mapping Dataflow Delta Sink is Failing When Partition Pruning is Applied

Question

Hi everyone,

We are upserting data into delta table using mapping dataflow in Synapse studio.

But when partition pruning is applied the upsert fails with org.apache.spark.sql.AnalysisException

org.apache.spark.sql.AnalysisException: Failed to resolve  
;;  
'DeltaMergeInto (((id#4355 = id#6794) && (rowHash#4366 = rowHash#6805)) && createdDate#6807 IN (19019)), [Update [condition: ((NOT ((source.`rd6c1f498e225442f868a2a54189ffd0a` & 2) = 0)) OR (NOT ((source.`rd6c1f498e225442f868a2a54189ffd0a` & 8) = 0))), actions: `createdOn` = source.`createdOn`, `id2` = source.`id2`, `pr` = source.`pr`, `type` = source.`type`, `version` = source.`version`, `pbcT` = source.`pbcT`, `createdDate` = source.`createdDate`, `modifiedDat` = source.`modifiedDat`, `id4` = source.`id4`, `createdBy` = source.`createdBy`, `createdDate` = source.`createdDate`, `id3` = source.`id3`, `runId` = source.`runId`, `src` = source.`src`, `lastModifiedBy` = source.`lastModifiedBy`], Delete [condition: (NOT ((source.`rd6c1f498e225442f868a2a54189  

Answer 1

Hi @Mujah ,

Thank you for posting query In Microsoft Q&A Platform.

Partition Pruning expects value as array. So could you please try below expression and see if that helps.

array(toDate($runDate,'yyyy-MM-dd'))

Please let us know how it goes. Thank you.