Import Incremental Data from on Prem SQL server using Azure data facotry to Data lake

AzeemK 511 Reputation points

I am looking for a good tutorial or Article on how to import incremental data ( delta Loads) new records from SQL Server on Prem to Azure Data lake Gen2 in cloud using Azure Data factory V2.

My tables have modified date and I want to use that date do delta imports daily using ADF v2 , right now I a importing whole table using copy activity on daily basis with foreach loop to copy all tables with one step, Any good videos tutorials on how to insert only new records during this COPY ACTIVITY with FOREACH loop/

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,726 questions
No comments
{count} votes

Accepted answer
  1. HimanshuSinha-msft 18,621 Reputation points Microsoft Employee

    Hello @AzeemK ,

    Thanks for the ask and using the forum .

    I think you are looking for this tutorial . .

    As it sounds you are just missing the watermark logic which is called out in the tuitorial .


2 additional answers

Sort by: Most helpful
  1. AzeemK 511 Reputation points

    Thanks for the link

  2. Bhanu Amballa 1 Reputation point

    Hi @HimanshuSinha-msft

    I have same issue want to load incremental data from on-premises db to azure data lake on daily bases.
    Thanks for the link provided

    My doubt is where does this watermark table exists in the source or in the azure data lake.

    If in Azure data lake , how does it stores?

    If it is in the source, I have read only access to read the data from on-premises db. How can I create table and insert/update values into the source system?