How to Fetch 1000 records data from Rest API connect dynamically with Parameters

Kumari, Shweta 41 Reputation points
2021-10-02T11:57:20.267+00:00

Hi,

I am trying to fetch 1000 using the rest API connector link services in ADF and storing in ADLS but I am not able to set pagination for the URL.

Example of my URL :https://api.instantwebtools.net/v1/passenger?page=1&size=10

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,154 questions
0 comments No comments
{count} votes

Accepted answer
  1. svijay-MSFT 5,226 Reputation points Microsoft Employee
    2021-10-04T15:16:26.233+00:00

    Hello @KumariShwetha-1759

    I am assuming that you are making use of the Copy Data activity to move data from REST API to ADLS. If this is not case. Please feel free to provide some context on the activity.

    The copy data by default supports the pagination for the below scenarios :

    137447-image.png

    137370-image.png

    Reference : https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#pagination-support

    Now, coming back to your scenario

    For fetching 1000 records, I am guessing that you would like recursively hit the page like below

    https://api.instantwebtools.net/v1/passenger?page=1&size=10  
    https://api.instantwebtools.net/v1/passenger?page=2&size=10  
    https://api.instantwebtools.net/v1/passenger?page=3&size=10  
    ...  
    ...  
    ...  
    https://api.instantwebtools.net/v1/passenger?page=10&size=10  
       
    

    if the response header/response body have the url for the next page ( for instance, https://api.instantwebtools.net/v1/passenger?page=2&size=10) , you could configure it in the below way.

    137409-image.png

    If the next page link or query parameter doesn't come as response headers/body then you're less likely to be able to use the feature.

    The alternative would be to make use of loop logic & Perform the Copy Activity

    In the Rest Connector create two parameters :

    137464-image.png

    Now populate the relative URL for the RestConnector with the parameters.

    137427-image.png

    @concat(dataset().page,dataset().size)  
    

    This is to form the Relative URL dynamically. In your case "?page=1&size=10"

    Now, In the Copy Activity :

    137482-image.png

    You could set the page number as a Variable.

    @concat('?page=',variables('page'))  
    

    This variable value would be incremented in a loop by making use of the Set Variable activity. for each iteration --> ?page=1, ?page=2,?page=3

    The URL for the Copy Activity is dynamically set for each iteration.

    To Loop / iterate - You could either make use of Until activity

    Condition : Until the variable is less than 100 or so.

    After the page 1 is executed - the variable is updated 2.

    The request is made with the variable value 2 i.e. the page 2 is queried.

    https://reqres.in/api/users?page=2&size=10  
    

    OR

    FOREACH Loop (The implementation is discussed here : https://learn.microsoft.com/en-us/answers/questions/59467/how-to-pass-pagination-rules-during-a-rest-api-cal.html)

    Note : if needed you could then merge the data for all the files (generated for each page)

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful