Pagination in azure data factory using Rest API

Question

Hello, Team
I have a project scenario that requires me to use REST APIs with pagination, basically I have a resource REST API that I would like to get all the data from it and insert to a table in SQL Server, but the request it includes paging at the end, so basically when I do the request I get one part of the data, and at the end I have pagination as shown below, what I want is the whole data including the paging.

"Paging": {
"Next": "https://ustest60-services.dayforcehcm.com/Api/winn/V1/Reports/Report?cursor=7bCOeZDzU9MuYYaPMapyBnDk%252Ff33ZjHfeCu5bTTl2nGcA233yrTuSFjbKlW%252FjpCm";
}

How can I map in the sink part to retrieve the data from the "Next"

I would appreciate any hints or recommendations, also I can provide my pipeline id to check the runs if need it.

Looking forward for a response.
Thank You.

Answer 1

Hello @DervishiErald-7950 and welcome to Microsoft Q&A.

In the Copy Activity, in the source, try adding an Additional Column to duplicate the paging if it is being removed.

"MyPagingDuplicate" , "$$COLUMN" , "$.Paging.Next"

Please let me know if this helps. I wasn't able to test against your website, but I was able to duplicate in another one.