Azure Data Factory Pagination

Question

I have a project scenario to get all users from an endpoint URL and insert it to a SQL table, I am using ADF but I'm having some issues with pagination, I have used before pagination but only with nextpage:"someurl" using AbsoluteUrl but now I am having an issue because my response comes as follow and for me to get next pages I need to perform multiple calls as follow. I would appreciate any suggestions or recommandation.
my URL is as follows.

BaseURL: https://xyz.xyz.com/api/v2/users?take=500 which gives me the first 500
Response:

> ],
"total": 3059,
"next": 500
}

In order for me to take the next 500 I need to make a request as follow
https://xyz.xyz.com/api/v2/users?take=500&skip=500
https://xyz.xyz.com/api/v2/users?take=500&skip=1000
and so on..

Could you give me any recommendations on how to approach this?
Thank You

Answer 1

Hello and welcome to Microsoft Q&A @Dervishi, Erald .

I've seen similar requests before. For an older solution, you might want to look into this thread link.

However, there is a new pagination option: Range!
The range pagination option is one I haven't had a chance to play with yet. However it appears to be exactly what you need. The following excerpt is taken from pagination support page. I am working now to make a more relatable explanation.

Example: Pagination rules

If you want to send multiple sequence requests with one variable in a range, you can define a variable such as {offset}, {id} in AbsoluteUrl, Headers, QueryParameters, and define the range rule in pagination rules. See the following examples of pagination rules:

Example 1

You have multiple requests:

Copy
baseUrl/api/now/table/incident?sysparm_limit=1000&sysparm_offset=0,

baseUrl/api/now/table/incident?sysparm_limit=1000&sysparm_offset=1000,

......

baseUrl/api/now/table/incident?sysparm_limit=1000&sysparm_offset=10000
You need to specify the range pagination:

AbosoluteUrl = baseUrl/api/now/table/incident?sysparm_limit=1000&sysparm_offset={offset}

The pagination rule is: QueryParameter.{offset} = RANGE:0:10000:1000