How to configure Pagination in REST Source ADF Copy data activity with end cursors in the response?

Question

I have an ADF pipeline with a REST Datasource which pulls data from an API. The way pagination is currently working in this API is, for every page, the response contains a "pagination" field in the body which has 2 fields - hasNextPage (boolean) and endCursor (GUID).

I was able to set the EndCondition parameter correctly, but I am facing issues while setting up the QueryParameters. The URL to get to the next page should contain the "endCursor" value of the previous page. What is the best way to solve this?

Answer 1

@Abhinav Gundapaneni Hello and welcome to Microsoft Q&A.

When the pagination passes you a full or partial url like:

https://MySite.net/api/table/pages/3
or
/table/pages/3

then you should use the "absoluteURL" pagination rule.

If the pagination is meant as a url parameter like

statehash=f4sdh5
in
https://MySite.net/api/queryresults?statehash=f4sdh5

then you should use the "QueryParameters" pagination rule. The Name value would be "statehash" and since you get the cursorEnd from the body, choose "Body" and "pagination.cursorEnd"