API pagination & Data Factory

Question

Hi,
I’m working on pulling a data source in via ADF and have the connection all setup and working.
The problem I’ve got is that the API will only return a max of 50 records but does present the following in the JSON.

"paging": {
"offset": 0,
"limit": 50,
"total": 85
}

So I know there are more records there but cannot for the life of me work t the best way to have ADF cycle through and have failed to find a representative example as yet. Any help appreciated in pointing me in the right direction here.

Answer 1

Hello @Mark E and thank you for your question.

The pagination support in the REST connector is rather limited. It does not support that form of pagination out-of-the-box. You can help prioritize this feature improvement by voting for it at the feedback forum. There are several requests to choose from:

• Make ADF pagination functionality more robust to handle more robust pagination rules using variables and comparison operators
• support web linking with rest api pagination
• Support Rest Pagination based on response element count
• Support for Dynamic Content with pagination rule in Rest source while Copy Data activity

I can provide you a convoluted work around if you are interested. The work-around involves first using a web activity to get the paging, offset, limit, and total you mentioned, then set up variables for a loop. Inside the loop there is copy activity and set variable activities. The drawback is each copy activity produces a separate files.
To get around the self-referencing variable limitation, I use 2 variables, X and Y. First I set Y = X + 1. Then I set X = Y. That is how I increment variables.

Let me know if you are interested, and I will provide pictures.

Thank you for your patience,
Martin.

Answer 2

Did the convoluted workaround get posted? Also have this problem...

Answer 3

New features for handling pagination have been added since this question was originally asked.
See https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#pagination-support