ADF Pagination rule: passing data from a response as a request parameter

Question

I am trying to retrieve data, this requires pagination.

According to the app documentation, in the response there will be a pagination key that will describe details on fetching the next page.
I looks like:
{
organizations: [
--{
--id: 1,
--name: 'My organization',
--},
...
],
pagination:
--{
--next_page_start_id: 98
--}
}
So for the first request we use relative url: organization/issues?page_limit=100
The next request should be made with the same parameters as the first, but with a page_start_id of 98.
So for the second request we use relative url: organization/issues?page_limit=100&page_start_id=98 and so on, until "pagination" is not found in the response.

I created a pagination rule:
QueryParameters-----page_start_id-----$.pagination.next_page_start_id
EndCondition---------$.pagination------NonExist

and in relative url I am using organization/issues?page_limit=100&page_start_id={page_start_id}

However I still have an error, page_start_id is not valid.

Could you please suggest how to solve the problem? What am I missing?

Answer 1

Hi @Maryna Paluyanava ,

Thank you for posting query in Microsoft Q&A Platform.

Kindly follow below screenshots and implement same. Please Note, still if issue exists then kindly share your implementation screenshots for linked service, dataset, Source pagination & sample response json. Also, please check below link where pagination examples are documented for better idea.
https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#pagination-rules-examples

REST Dataset configurations:

Copy activity - Pagination settings:

End condition of pagination should be specified based on expected response from API for end. Different kinds of examples for end condition are defined in below link. Kindly follow them for end conditions.
https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#example-4variables-are-in-absoluteurlqueryparametersheaders-the-end-variable-is-not-pre-defined-and-the-end-condition-is-based-on-the-response

Hope this helps. Please let me know if any further queries.

-------------

Please consider hitting Accept Answer button. Accepted answers help community as well.