Does ADF support incremental load from REST API to Azure SQL Database?

Kriti Bhardwaj 1 Reputation point
2022-04-12T07:35:33.367+00:00

Does ADF support incremental load from REST API to Azure SQL Database? If yes, please provide the documentation/ solution

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,933 questions
0 comments No comments
{count} votes

2 answers

Sort by: Most helpful
  1. Samy Abdul 3,366 Reputation points
    2022-04-12T08:39:46.187+00:00

    Hi @Kriti Bhardwaj , below thread discusses the scenario in detail, although the thread is bit older but it should give you something to start with:

    https://stackoverflow.com/questions/54812089/bringing-incremental-data-in-from-rest-apis-into-sql-azure Thanks


  2. AnnuKumari-MSFT 31,731 Reputation points Microsoft Employee
    2022-04-25T09:35:19.66+00:00

    Hi @Kriti Bhardwaj ,

    Thankyou for using Microsoft Q&A platform and posting your query.

    As I understand your query, you want to know whether incremental load is possible in case where your source data set is a REST API and sink is Azure SQL database or not.

    Here are few points that might help you:

    1. REST API supports pagination . You can copy data from REST API which sends response in Pages using Azure data factory. Please refer to the following video for more details: https://www.youtube.com/watch?v=EbURwy2a8dM
    2. You can use copy activity and use REST API dataset as the source and Az SQL table as the sink dataset and load all the data into a dummy table using a FULL LOAD technique and then use that dummy table to join to the main table to perform incremental load.

    If this helps , please do consider clicking Accept Answer as accepted answers help community as well. If you have any further query do let us know.