question

arkiboys avatar image
0 Votes"
arkiboys asked ShaikMaheer-MSFT commented

duplicating rows in parquet file (Sink Delta)

Hi,
The pipeline uploads data from source to sink Delta parquet dynamically. One of the parameters passed to the dataflow is the KeyColumn.
If the object to load has only one KeyColumn, all works fine, i.e. update/insert/delete into parquet sink delta.
The problem is when the keyColumn is more than one, then the parquet file is keep being inserted with new rows for each of the Keys.
It looks like the keycolumns are not being handled correctly in the dataflow if they are more than one, i.e. "ProductID, LocationID"

data flow has:
delta sink setting--> update
delta sink setting--> delete
delta sink setting--> insert

It looks like the issue is on the update I think.

Any suggestions?

140190-image.png

140303-image.png

140210-image.png


azure-data-factory
image.png (16.6 KiB)
image.png (20.9 KiB)
image.png (28.7 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @arkiboys,

Would you please try UPSERT instead of UPDATE. Please let us know how it goes. Thanks!


Nasreen

0 Votes 0 ·

Hi @arkiboys ,

Following up to see if your issue resolved. If yes, please consider accepting answer. Thank you.

0 Votes 0 ·

1 Answer

ShaikMaheer-MSFT avatar image
0 Votes"
ShaikMaheer-MSFT answered

Hi @arkiboys ,

your columns names string has space after comma. So I doubt when you use split() function to split based on ',' then in that array second column may come as space in it and creating mess.

Could you please try removing space after comma and see if that helps? Please let us know how it goes.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @arkiboys ,

Just checking is your issue resolved? You tried above commented? Please let us know if any queries or feel free to share your way of implementation if its different.

1 Vote 1 ·