Pagination in REST API

Question

Hello,

I am trying to set pagination rule.
We increase page_num from 0 by 1 until in the response we receive last=true. Maximum number of records on the page is set to 200.
There are 620 records in the source.
When I implement the pagination in the following way I get infinite loop, 198 unique records are loading to the table infinitely.
Could you please tell me where I have a mistake?

Thank you!

Answer 1

Hello @Maryna Paluyanava ,
Thanks for the question and using MS Q&A platform.

As I understand, you want help troubleshooting your Copy Activity's REST pagination rules. The symptom is infinite loop.

Given you did not specify an end value in the range, the job of halting the loop falls to the End condition.

Taking a closer look, I see you specified $['last'] and also included a mapping of the screenshot where you also included the $['last']. You did not, however, share an example of the response data to show where last is in the JSON.
I know in mapping there are some odd things to get the property when the body is complex. I haven't tried seeing if those also work in the pagination rules. It just never occured to me. Simply looking with how you specified it in the pagination rule, I am expecting the last property is at the base level, not nested deep. That is, the response would be like

{  
  "size": " 12,  
  "number": 23,  
  "totalElements": 220,  
  "last":false,  
...  
  "empty":true  
}  

If this is the case, could you try changing the pagination from $['last'] to $.last please?

If this is not the case, then I think we need to look and find the exact path to the "last" property.

Another thing to try, is changing from cons true to exists. This is to test whether the rule is looking in the right place for the "last" property. If it isn't looking in the right place, then it will never return a value of true.

Please do let me if you have any queries.

Thanks
Martin

---

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators