Azure Data Factory Pagination

Dervishi, Erald 66 Reputation points
2022-02-04T16:29:19.853+00:00

I have a project scenario to get all users from an endpoint URL and insert it to a SQL table, I am using ADF but I'm having some issues with pagination, I have used before pagination but only with nextpage:"someurl" using AbsoluteUrl but now I am having an issue because my response comes as follow and for me to get next pages I need to perform multiple calls as follow. I would appreciate any suggestions or recommandation.
my URL is as follows.

BaseURL: https://xyz.xyz.com/api/v2/users?take=500 which gives me the first 500
Response:

> ],
"total": 3059,
"next": 500
}

In order for me to take the next 500 I need to make a request as follow
https://xyz.xyz.com/api/v2/users?take=500&skip=500
https://xyz.xyz.com/api/v2/users?take=500&skip=1000
and so on..

Could you give me any recommendations on how to approach this?
Thank You

Azure Database Migration service
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,444 questions
Microsoft Partner Center API
Microsoft Partner Center API
Microsoft Partner Center: A Microsoft website for partners that provides access to product support, a partner community, and other partner services.API: A software intermediary that allows two applications to interact with each other.
311 questions
0 comments No comments
{count} votes

Accepted answer
  1. MartinJaffer-MSFT 26,011 Reputation points
    2022-02-07T19:27:28.447+00:00

    Hello and welcome to Microsoft Q&A @Dervishi, Erald .

    I've seen similar requests before. For an older solution, you might want to look into this thread link.

    However, there is a new pagination option: Range!
    The range pagination option is one I haven't had a chance to play with yet. However it appears to be exactly what you need. The following excerpt is taken from pagination support page. I am working now to make a more relatable explanation.

    Example: Pagination rules

    If you want to send multiple sequence requests with one variable in a range, you can define a variable such as {offset}, {id} in AbsoluteUrl, Headers, QueryParameters, and define the range rule in pagination rules. See the following examples of pagination rules:

    Example 1

    You have multiple requests:

    Copy
    baseUrl/api/now/table/incident?sysparm_limit=1000&sysparm_offset=0,

    baseUrl/api/now/table/incident?sysparm_limit=1000&sysparm_offset=1000,

    ......

    baseUrl/api/now/table/incident?sysparm_limit=1000&sysparm_offset=10000
    You need to specify the range pagination:

    AbosoluteUrl = baseUrl/api/now/table/incident?sysparm_limit=1000&sysparm_offset={offset}

    The pagination rule is: QueryParameter.{offset} = RANGE:0:10000:1000

    2 people found this answer helpful.

0 additional answers

Sort by: Most helpful