Import Incremental Data from on Prem SQL server using Azure data facotry to Data lake

AzeemK 516 Reputation points
2020-12-04T23:27:53.5+00:00

I am looking for a good tutorial or Article on how to import incremental data ( delta Loads) new records from SQL Server on Prem to Azure Data lake Gen2 in cloud using Azure Data factory V2.

My tables have modified date and I want to use that date do delta imports daily using ADF v2 , right now I a importing whole table using copy activity on daily basis with foreach loop to copy all tables with one step, Any good videos tutorials on how to insert only new records during this COPY ACTIVITY with FOREACH loop/
Thanks

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,599 questions
0 comments
Accepted answer
  1. HimanshuSinha-msft 19,381 Reputation points Microsoft Employee
    2020-12-07T23:30:51.757+00:00

    Hello @AzeemK ,

    Thanks for the ask and using the forum .

    I think you are looking for this tutorial . https://learn.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-portal .

    As it sounds you are just missing the watermark logic which is called out in the tuitorial .

    Thanks
    Himanshu

2 additional answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. AzeemK 516 Reputation points
    2020-12-10T23:39:16.82+00:00

    Thanks for the link

    0 comments
  2. Bhanu Amballa 1 Reputation point
    2021-07-13T04:22:25.173+00:00

    Hi @HimanshuSinha-msft

    I have same issue want to load incremental data from on-premises db to azure data lake on daily bases.
    Thanks for the link provided

    My doubt is where does this watermark table exists in the source or in the azure data lake.

    If in Azure data lake , how does it stores?

    If it is in the source, I have read only access to read the data from on-premises db. How can I create table and insert/update values into the source system?

    Thanks
    Bhanu

    0 comments