sink delta parquet file rows duplicated on each run

Question

sink delta parquet file rows duplicated on each run

arkiboys 9,711

When I debug step by step each of the transformations in dataflow, I see the correct results for:
sinkDeltaDelete
sinkDeltaInsert
sinkDeltaUpdate

Question:
When I run the pipeline which has this dataflow in it, I get duplicated rows for the existing rows in the sink parquet file.

Any suggestions?

Thank you

Answer accepted by question author

0 additional answers

Your answer

Answer 1

HimanshuSinha 19,567 Microsoft Employee Moderator

Hello @arkiboys ,
Thanks for the ask and using Microsoft Q&A platform .
I see three flows running a parallel , may be thats the reason .

Please do let me know how it goes .
Thanks
Himanshu

-------------------------------------------------------------------------------------------------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification
If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

arkiboys 9,711 Reputation points

2021-10-15T03:05:27.617+00:00

Hi,
So what do you suggest I change the design to?
Thanks
HimanshuSinha 19,567 Reputation points Microsoft Employee Moderator

2021-10-21T18:58:23.78+00:00

Hello @arkiboys ,

Thanks for the ask again , Well since i have not seen the underlining data , i am not be very confident suggesting that , but then you can do the debug and see how the different branches behave and may be tweak or re-design the implementation .

Thanks
Himanshu

Share via

sink delta parquet file rows duplicated on each run

0 additional answers

Your answer