upsert to set a flag for Deleted = 1 and Deleted=0 for upsert- dataflow

Question

Hi,
I am trying to set a flag as Deleted = 1 for the rows which are deleted at source and Deleted =0 for the rows which are not deleted.
The sink in Delta parquet file --> WriteDelta2
The source is parquet file --> ParquetFile
...
Join1 --> full outer join on keyColumns
Join2 --> full outer join on keyColumns

Error is on Join2:
Join2
Duplicate columns found in Join
Duplicate columns found in both sides. Consider adding a Select transformation before either the left or right stream. Invalid columns: KeyColumnSource,InputFilePath,KeyColumnTarget from DerivedColumn3

I know I have to add a select but not sure exactly why because as you see one branch is for all rows which are set to have Deleted=1 and the other branch is for rows with Deleted=0
And not sure how to have the settingg for select if that is the answer.

Thanks

Answer 1

Hi @arkiboys ,

Thank you posting query here.

Sorry, your screenshots are not clear. But still what ever I observe seems to okay to me.

It seems, you are trying to perform Join2 on top of streams one is from first split and another one is from second split of conditional split transformation. So Its obvious that you may have same column on both and they resulted as duplicated columns when you perform join.

So, as suggested, could you please consider adding select transformation on any one of inputs streams of Join and see how it goes.

More clear images may help. Please let us know if any further queries.

---

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
  • Want a reminder to come back and check responses? Here is how to subscribe to a notification
  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators