Pagination Issue With Rest Request

Question

I'm struggling to get pagination to work for a rest request in data factory. Here's a preview of the data that gets returned by the rest call;

For my settings, I've applied 'AbsoluteURL', with Value 'Body' and the query $.paging._links.next.href

I've tried various other tweaks, but always get the same result... only 50 records return. What am I missing?

Answer 1

$.paging… does not work, because the response is json array [], and not object {}.

"The response body should contain only one JSON object. The JSONPath expression should return a single primitive value, which will be used to issue next request."

Answer 2

Hi @Nicholas Mead ,

Thank you for posting query in Microsoft Q&A Platform.

In your case it seems you are getting array and inside array its json object. Hence you may seeing it not working.

If we see official documentation example from below link, here API response was in json object.
https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#example-5set-end-condition-to-avoid-endless-requests-when-range-rule-is-not-defined

Still, kindly try using below syntax and see if that helps.
$.[0].paging._links.next.href or $[0].paging._links.next.href

If not helps, please try to make that API call using web activity and try to take the next URL in to some variable and then frame new URL and make again API call. That way it will work. If possible please also share the output json screenshot of API call from web activity to understand the API response better here.

Kindly consider checking below video, there I explained a different way of implementing pagination and see that fits in your case.
Copy data from REST API which sends response in Pages using Azure data factory

Check this video for understanding web activity.

Hope this helps. Please let me know how it goes.

-----------

Please consider hitting Accept Answer button. Accepted answers help community as well.