Azure data factory:Implementing the SCD2 on flatfile target

Question

Hi,

I have a scenario where in my source and target both are flat files. I want to implement SCD-2 on target flat file using dataflow. I was trying to use alter row then sink but later realized the sink for flat file doesnt give option for update row options. Is there any workaround for this.

Thanks.

Answer 1

Hi @PurushothamChanda-2222,

In my understanding you can not append/update data in the flat-file from the SINK option, either you have to create/replace the existing file. In your scenario, you can read both files i.e., source and target in the DataFlow --> do data manipulation (update data from the target + add new data from the source) --> sink it to the target file. Hope this helps!

---

If the above response is helpful, please Accept as Answer and Upvote it. Thanks!

Answer 2

Not for text flat files. But you can do this by using Delta as the dataset format in the sink if Parquet output is acceptable for your use case.

https://www.youtube.com/watch?v=evqQRwsF_Ps

https://learn.microsoft.com/en-us/azure/data-factory/tutorial-data-flow-delta-lake