I would like to perform incremental load in ADF

Question

I would like to perform incremental load in ADF

Mounika, Durgam 1

I already done creating copy pipeline to get data from servicenow to AZURE SQL DB ... Now i want to do incremental load from Servicenow Input Dataset to Azure SQL DB Output Dataset using ADF. Is it possible or not?

1 answer

Your answer

Answer 1

Vinodh247 34,741 MVP Volunteer Moderator

Hi Mounika,

Thanks for reaching out to Microsoft Q&A.

Yes you can perform incremental load copy from your source to azure sql database using ADF pipelines.

If your source is a database you can go with...

Load from source database by using watermark
Load from source database by using CDC(Change data capture)

https://learn.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-overview

Or if you want to convert your existing pipeline to load incrementally, you have to edit and modify the pipeline to suit your requirement.

If your source is in file format and if you want to copy based on last file creation time, check the below link...

https://vinsdata.wordpress.com/2021/10/25/incremental-file-copy-in-azure-data-factory/

Bhargava-MSFT 31,261 Reputation points Microsoft Employee Moderator

2022-07-22T20:58:45.43+00:00
Thank you @Vinodh247 ,

Hi @Mounika, Durgam ,

I am checking in to see if the above answer from @Vinodh247 helped. And if you have any further queries, do let us know.

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

Want a reminder to come back and check responses? Here is how to subscribe to a notification

If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Share via

I would like to perform incremental load in ADF

1 answer

Your answer