upsert to set a flag for Deleted = 1 and Deleted=0 for upsert- dataflow

arkiboys 9,686 Reputation points
2021-10-07T15:31:11.86+00:00

Hi,
I am trying to set a flag as Deleted = 1 for the rows which are deleted at source and Deleted =0 for the rows which are not deleted.
The sink in Delta parquet file --> WriteDelta2
The source is parquet file --> ParquetFile
...
Join1 --> full outer join on keyColumns
Join2 --> full outer join on keyColumns

Error is on Join2:
Join2
Duplicate columns found in Join
Duplicate columns found in both sides. Consider adding a Select transformation before either the left or right stream. Invalid columns: KeyColumnSource,InputFilePath,KeyColumnTarget from DerivedColumn3

I know I have to add a select but not sure exactly why because as you see one branch is for all rows which are set to have Deleted=1 and the other branch is for rows with Deleted=0
And not sure how to have the settingg for select if that is the answer.

Thanks

138612-image.png
138585-image.png
138574-image.png
138613-image.png
138548-image.png
138549-image.png
138470-image.png
138524-image.png
138621-image.png
138566-image.png
138614-image.png
138567-image.png

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,113 questions
{count} votes

Accepted answer
  1. ShaikMaheer-MSFT 38,326 Reputation points Microsoft Employee
    2021-10-08T09:46:08.79+00:00

    Hi @arkiboys ,

    Thank you posting query here.

    Sorry, your screenshots are not clear. But still what ever I observe seems to okay to me.

    It seems, you are trying to perform Join2 on top of streams one is from first split and another one is from second split of conditional split transformation. So Its obvious that you may have same column on both and they resulted as duplicated columns when you perform join.

    So, as suggested, could you please consider adding select transformation on any one of inputs streams of Join and see how it goes.

    More clear images may help. Please let us know if any further queries.


    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
      • Want a reminder to come back and check responses? Here is how to subscribe to a notification
      • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    0 comments No comments

0 additional answers

Sort by: Most helpful