How to load Multiple APIS in Single Pipeline.

Question

How to load Multiple APIS in Single Pipeline.

Ayub shaik 21

HI All, I am Very New to this Technology. How to load my multiple APIS in single Pipeline in ADF.
Many thanks.

AnnuKumari-MSFT 21,401 Reputation points Microsoft Employee

2023-01-05T05:18:01.023+00:00

Hi @Ayub shaik ,
Just checking in to see if the below answer helped. Please do consider clicking Accept Answer and Up-Vote for the same as accepted answers help community as well. If you have any further query do let us know.

AnnuKumari-MSFT 21,401 Reputation points Microsoft Employee

2023-01-05T05:18:01.023+00:00

Hi @Ayub shaik ,
Just checking in to see if the below answer helped. Please do consider clicking Accept Answer and Up-Vote for the same as accepted answers help community as well. If you have any further query do let us know.

Answer 1

Hi @Ayub shaik ,

Welcome to Microsoft Q&A platform and thanks for posting your query here.

As I understand your question , you want to know how to load data from multiple REST APIs into destination datastore using single ADF pipeline. Please let me know if my understanding is incorrect.

Could you please tell what is your destination dataset?

1. In case of single REST API as source:

You can copy data from a source data set using to Azure SQL (sink) using REST API which sends response in Pages using Azure data factory. Please refer to this video.
You can use copy activity and use REST API dataset as the source and Az SQL table as the sink dataset and load all the data into a dummy table using a FULL LOAD technique and then use that dummy table to join to the main table to perform incremental load.

2. In case of multiple REST APIs as source, you can store the APIs in a control table and loop through them using ForEach and use execute pipeline to pass each APIs into the child pipeline (Pipeline for single REST API source i.e. step1 pipeline) as parameter .

Hope this will help. Please let us know if any further queries.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you.
Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification
If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Ayub shaik 21 Reputation points

2023-01-05T05:51:55.293+00:00

HI Thanks for your reply and Help ,

I am Looking for 2nd one , Can you please share the steps how can i do that. if any PDF it will be help full.
AnnuKumari-MSFT 21,401 Reputation points Microsoft Employee

2023-01-12T05:24:11.36+00:00

Hi @Ayubshaik-2899 ,

Create a config / control table in SQL DB to store the APIs in a column say 'colAPI' . If creating a SQLDB is not a feasible solution for you, you can go for storing the APIs in .csv files in ADLS .

Use lookup activity in ADF pipeline to fetch the APIs from the table.

Use Foreach activity to iterate through each of the rows of the table.

You can't use foreach inside foreach . So , Inside Foreach activity , you need to use execute pipeline and create a child pipeline that will have a parameter ,say, paramAPI

Pass the parameter value dynamically from parent pipeline to child pipeline having expression as @item().colAPI

Create the child pipeline in similar way like guided in this video

How to load Multiple APIS in Single Pipeline.

0 additional answers

Activity