Incremental Changes from Integration Runtime to Data Factory

Question

Hi-

I am attempting to load data from an on-prem sql server to azure data factory (data lake). I will be scheduling the upload once per day.

How do I keep the data together, but still have a way to separate the days?

Basically I do not want to overwrite the data, I just want the tables I am extracting to keep growing daily.

Answer 1

Hi @MSau ,

Thankyou for using Microsoft Q&A platform and posting your query.

My understanding about your query is that you want to perform incremental data load from on-premise SQL server to ADLS using ADF . But the files are getting overwritten with every load. You want to store the incremental data but don't want to overwrite the file.

There is no option to append data in the file . So we need to load the files daily with filename having current date to resolve this .

Try parameterizing the file name of sink dataset to contain date on which file gets loaded. It will lead to ingestion of daily files with dynamic filename that gets auto-generated based on the date on which file gets loaded.

Please refer to this video for more details on how to parameterize the foldername and filename for storing daily transacted data : https://youtu.be/FXw1gPaa2-M?t=1970

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you.
  Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators