Using Until for looping copy activity which has Rest Api call set as source

Shreyasi Srivastava 21 Reputation points

I have created a copy activity which has source for REST API and Sink as Azure ADLS. It work for the 1000 records.

I need to apply custom pagination to track if has more than 1000 records any time. I am trying to parameterize the copy activity dataset but not sure how to use until for looping in this case.

How can I achieve this custom pagination?
Please help ASAP.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,898 questions
0 comments No comments
{count} votes

Accepted answer
  1. ShaikMaheer-MSFT 38,301 Reputation points Microsoft Employee

    Hi @Shreyasi Srivastava ,

    Thank you for posting query in Microsoft Q&A Platform.

    Sorry your ask is little unclear to me. What is mean when you say API call worked for 1000 records? You mean your API call will give you 1000 records in one call?

    You mentioned that you want to apply custom pagination using until loop. Could you please elaborate more with some simple example?

    between, could you please check below video, where pagination handled using ForEach loop and copy activity inside of it? may be this will be helpful in your case.

    Above video contains Foreach loop and copy activity. If in your case you want to use Until activity then yes you can use it. But make sure to end Until loop at some point by using some flag. Please check below video as well to understand how until loop works.

    Hope this helps. Please let us know it goes. Thank you.

1 additional answer

Sort by: Most helpful
  1. Shreyasi Srivastava 21 Reputation points

    Hi @ShaikMaheer-MSFT , Following the same line of work. Our task is now to copy files from multiple relative urls of a single API instead of just one into same folder but different files.
    For the solution I am using For each loop and I have set a parameter at pipeline level as below-

    I have kept the until activity inside the for loop and for the settings of for loop I have as below-

    Inside for loop it looks like- 199249-image.png

    Inside until activity I have one copy data which is copying from api and two variables for top and skip.

    Inside copy data inside data set I have set three parameters

    And passing these values to them

    Inside connection there is api and relative url is set

    I have set the relative url as 199268-image.png

    For the sink data set we have set parameter 199235-image.png

    For connection in sink dataset

    With all these settings, still pipeline is failing.

    0 comments No comments