Does ADF support incremental load from REST API to Azure SQL Database?

Kriti Bhardwaj 1 Reputation point

Does ADF support incremental load from REST API to Azure SQL Database? If yes, please provide the documentation/ solution

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,933 questions
0 comments No comments
{count} votes

2 answers

Sort by: Most helpful
  1. Samy Abdul 3,366 Reputation points

    Hi @Kriti Bhardwaj , below thread discusses the scenario in detail, although the thread is bit older but it should give you something to start with: Thanks

  2. AnnuKumari-MSFT 31,731 Reputation points Microsoft Employee

    Hi @Kriti Bhardwaj ,

    Thankyou for using Microsoft Q&A platform and posting your query.

    As I understand your query, you want to know whether incremental load is possible in case where your source data set is a REST API and sink is Azure SQL database or not.

    Here are few points that might help you:

    1. REST API supports pagination . You can copy data from REST API which sends response in Pages using Azure data factory. Please refer to the following video for more details:
    2. You can use copy activity and use REST API dataset as the source and Az SQL table as the sink dataset and load all the data into a dummy table using a FULL LOAD technique and then use that dummy table to join to the main table to perform incremental load.

    If this helps , please do consider clicking Accept Answer as accepted answers help community as well. If you have any further query do let us know.