Using Until for looping copy activity which has Rest Api call set as source

Question

Hi,
I have created a copy activity which has source for REST API and Sink as Azure ADLS. It work for the 1000 records.

I need to apply custom pagination to track if has more than 1000 records any time. I am trying to parameterize the copy activity dataset but not sure how to use until for looping in this case.

How can I achieve this custom pagination?
Please help ASAP.

Answer 1

Hi @Shreyasi Srivastava ,

Thank you for posting query in Microsoft Q&A Platform.

Sorry your ask is little unclear to me. What is mean when you say API call worked for 1000 records? You mean your API call will give you 1000 records in one call?

You mentioned that you want to apply custom pagination using until loop. Could you please elaborate more with some simple example?

between, could you please check below video, where pagination handled using ForEach loop and copy activity inside of it? may be this will be helpful in your case.
https://www.youtube.com/watch?v=EbURwy2a8dM

Above video contains Foreach loop and copy activity. If in your case you want to use Until activity then yes you can use it. But make sure to end Until loop at some point by using some flag. Please check below video as well to understand how until loop works.
https://www.youtube.com/watch?v=n8e_exWMH5k

Hope this helps. Please let us know it goes. Thank you.

Answer 2

Hi @ShaikMaheer-MSFT , Following the same line of work. Our task is now to copy files from multiple relative urls of a single API instead of just one into same folder but different files.
For the solution I am using For each loop and I have set a parameter at pipeline level as below-

I have kept the until activity inside the for loop and for the settings of for loop I have as below-

Inside for loop it looks like-

Inside until activity I have one copy data which is copying from api and two variables for top and skip.

Inside copy data inside data set I have set three parameters

And passing these values to them

Inside connection there is api and relative url is set

I have set the relative url as

For the sink data set we have set parameter

For connection in sink dataset

With all these settings, still pipeline is failing.