Iteration and pagination to call rest api in adf

Question

Iteration and pagination to call rest api in adf

Anjali Maithani 46

Hello Team,

I have created a copy activity which has source for REST API and Sink as Azure SQL DB. It work for the 1000 records.

I need to apply pagination to have the full data from Rest API and also iterate the loop.

From which activity and properties i can achieve this.
Please help ASAP.

Regards,
Anjali Maithani

MartinJaffer-MSFT 26,161 Reputation points

2021-04-05T17:10:12.317+00:00

@Anjali Maithani did my response solve your issue? If so, please mark as answer, otherwise let me know how I may better assist you.
Shreyasi Srivastava 21 Reputation points

2022-03-25T17:33:22.877+00:00

Hi, did your question got resolved. Could you put the solution here as I have same issue. I am not sure how to put the copy activity inside a loop in tis case.

Answer accepted by question author

0 additional answers

Your answer

MartinJaffer-MSFT 26,161 Reputation points

2021-04-05T17:10:12.317+00:00

@Anjali Maithani did my response solve your issue? If so, please mark as answer, otherwise let me know how I may better assist you.
Shreyasi Srivastava 21 Reputation points

2022-03-25T17:33:22.877+00:00

Hi, did your question got resolved. Could you put the solution here as I have same issue. I am not sure how to put the copy activity inside a loop in tis case.

Answer 1

MartinJaffer-MSFT 26,161

Hello @Anjali Maithani and welcome to Microsoft Q&A.

With REST as source, you can add pagination rules in the copy activity source tab. The way to paginate depends upon how the source behaves. Please see this doc for details:

https://learn.microsoft.com/en-us/azure/data-factory/connector-rest#pagination-support

If yours does not behave like that, you can try parameterizing the dataset and put the copy activity inside a loop.

Anjali Maithani 46 Reputation points

2021-03-31T15:50:55.133+00:00

How can i define parameter ? can you please describe the steps.

Regards
MartinJaffer-MSFT 26,161 Reputation points

2021-04-02T15:56:25.707+00:00

@Anjali Maithani
Datasets and Linked services can be parameterized. You only need to do one of them.

For the dataset, open the dataset and go to the parameters tab...

Then go to the connection tab and add a reference to it in the relative url field...

For the linked service you can make it look like this:

Share via

Iteration and pagination to call rest api in adf

0 additional answers

Your answer