Is it possible to Append data from SQl Server in same csv file in Azure Data Lake storage from SQL Server

Question

I am trying to copy the data from SQL server where I am getting data in range but I am to store the data in same CSV using append or incremental approach(meant my store file remain same but data stored incrementally line first 0-99 data than 100-199 data like that) is there any way to do that.

Answer 1

Hi @Anonymous ,

Thank you for posting query in Microsoft Q&A Platform.

Files in storage account will get replace or override when you save new data with same name. Hence appending or incrementally loading file with same is not possible.

You can consider storing new data as another name may be with timestamp values in it. So that all data with different timestamp loads will be available in different files.

Once all your data is available in different files then you can consider using Union transformation inside dataflows to merge all data in to single set and load as sink file.

Kindly check below video to know how Union transformation will work.
https://www.youtube.com/watch?v=vFCNbHqWct8

Hope this helps. Please let us know if any further queries.

--------------------

Please consider hitting Accept Answer button. Accepted answers help community as well.