How to load Multiple APIS in Single Pipeline.

Ayub shaik 21 Reputation points
2023-01-02T02:41:18.677+00:00

HI All, I am Very New to this Technology. How to load my multiple APIS in single Pipeline in ADF.
Many thanks.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,332 questions
{count} votes

Accepted answer
  1. AnnuKumari-MSFT 19,121 Reputation points Microsoft Employee
    2023-01-03T08:27:47.977+00:00

    Hi @Ayub shaik ,

    Welcome to Microsoft Q&A platform and thanks for posting your query here.

    As I understand your question , you want to know how to load data from multiple REST APIs into destination datastore using single ADF pipeline. Please let me know if my understanding is incorrect.

    Could you please tell what is your destination dataset?

    1. In case of single REST API as source:

    • You can copy data from a source data set using to Azure SQL (sink) using REST API which sends response in Pages using Azure data factory. Please refer to this video.
    • You can use copy activity and use REST API dataset as the source and Az SQL table as the sink dataset and load all the data into a dummy table using a FULL LOAD technique and then use that dummy table to join to the main table to perform incremental load.

    2. In case of multiple REST APIs as source, you can store the APIs in a control table and loop through them using ForEach and use execute pipeline to pass each APIs into the child pipeline (Pipeline for single REST API source i.e. step1 pipeline) as parameter .

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.
      Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

0 additional answers

Sort by: Most helpful