Import Incremental Data from on Prem SQL server using Azure data facotry to Data lake

Question

I am looking for a good tutorial or Article on how to import incremental data ( delta Loads) new records from SQL Server on Prem to Azure Data lake Gen2 in cloud using Azure Data factory V2.

My tables have modified date and I want to use that date do delta imports daily using ADF v2 , right now I a importing whole table using copy activity on daily basis with foreach loop to copy all tables with one step, Any good videos tutorials on how to insert only new records during this COPY ACTIVITY with FOREACH loop/
Thanks

Answer 1

Hello @AzeemK ,

Thanks for the ask and using the forum .

I think you are looking for this tutorial . https://learn.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-portal .

As it sounds you are just missing the watermark logic which is called out in the tuitorial .

Thanks
Himanshu

Answer 2

Thanks for the link

Answer 3

Hi @HimanshuSinha-msft

I have same issue want to load incremental data from on-premises db to azure data lake on daily bases.
Thanks for the link provided

My doubt is where does this watermark table exists in the source or in the azure data lake.

If in Azure data lake , how does it stores?

If it is in the source, I have read only access to read the data from on-premises db. How can I create table and insert/update values into the source system?

Thanks
Bhanu