Azure data factory:Implementing the SCD2 on flatfile target

PorusDataEngineer 6 Reputation points


I have a scenario where in my source and target both are flat files. I want to implement SCD-2 on target flat file using dataflow. I was trying to use alter row then sink but later realized the sink for flat file doesnt give option for update row options. Is there any workaround for this.


Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
8,489 questions
{count} votes

2 answers

Sort by: Most helpful
  1. Nasreen Akter 10,706 Reputation points

    Hi @PurushothamChanda-2222,

    In my understanding you can not append/update data in the flat-file from the SINK option, either you have to create/replace the existing file. In your scenario, you can read both files i.e., source and target in the DataFlow --> do data manipulation (update data from the target + add new data from the source) --> sink it to the target file. Hope this helps!

    If the above response is helpful, please Accept as Answer and Upvote it. Thanks!

    0 comments No comments

  2. MarkKromer-MSFT 5,176 Reputation points Microsoft Employee

    Not for text flat files. But you can do this by using Delta as the dataset format in the sink if Parquet output is acceptable for your use case.

    0 comments No comments