How to do Incremental load from a Table Storage to SQL DB via ADF?

Manisha Saha 26 Reputation points
2022-06-16T20:43:53.217+00:00

We have table storage. We want to Incremental load to SQL DB via ADF.

Where clause will be >max(Timestamp) of SQL Table

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,225 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. AnnuKumari-MSFT 34,011 Reputation points Microsoft Employee
    2022-06-21T09:45:42.287+00:00

    Hi @Manisha Saha ,

    Thankyou for using Microsoft Q&A platform and thanks for posting your query.

    As I understand your requirement, you want to know how to perform incremental load from Table Storage to SQL DB using ADF pipeline. Please let me know if my understanding is incorrect.

    To implement incremental load concept , we need to have a benchmark column/watermark in the source that has the last updated time stamp or an incrementing key. Kindly check the following article which covers the steps of performing incremental load using watermark value : Incrementally load data from a source data store to a destination data store

    Also, kindly refer to the following video: https://www.youtube.com/watch?v=IPpYDQ0QE_E

    You can change your source and sink dataset accordingly and customize the pipeline according to your requirement , however, the concept would be similar in case of implementation.

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.
      Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.