Share via

Does ADF support incremental load from REST API to Azure SQL Database?

Kriti Bhardwaj 1 Reputation point
2022-04-12T07:35:33.367+00:00

Does ADF support incremental load from REST API to Azure SQL Database? If yes, please provide the documentation/ solution

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
0 comments

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Samy Abdul 3,376 Reputation points
    2022-04-12T08:39:46.187+00:00

    Hi @Kriti Bhardwaj , below thread discusses the scenario in detail, although the thread is bit older but it should give you something to start with:

    https://stackoverflow.com/questions/54812089/bringing-incremental-data-in-from-rest-apis-into-sql-azure Thanks

  2. AnnuKumari-MSFT 34,566 Reputation points Microsoft Employee Moderator
    2022-04-25T09:35:19.66+00:00

    Hi @Kriti Bhardwaj ,

    Thankyou for using Microsoft Q&A platform and posting your query.

    As I understand your query, you want to know whether incremental load is possible in case where your source data set is a REST API and sink is Azure SQL database or not.

    Here are few points that might help you:

    1. REST API supports pagination . You can copy data from REST API which sends response in Pages using Azure data factory. Please refer to the following video for more details: https://www.youtube.com/watch?v=EbURwy2a8dM
    2. You can use copy activity and use REST API dataset as the source and Az SQL table as the sink dataset and load all the data into a dummy table using a FULL LOAD technique and then use that dummy table to join to the main table to perform incremental load.

    If this helps , please do consider clicking Accept Answer as accepted answers help community as well. If you have any further query do let us know.

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.