ADF REST API Pagination Offset

Question

I'm using the Copy Data activity in Azure Data Factory. My source is REST API. It works correctly but only returns the first 100 records i.e., the first page. I need to implement pagination.

The offset is in the body of the JSON at the very end of the payload:
],
"offset": "itrC8ItybdqNWeqpO/recNz3GXl5KynFXFW"
}

I need to send the offset in the querystring as offset = itrC8ItybdqNWeqpO/recNz3GXl5KynFXFW

The offset changes with every subsequent payload.

My Source looks like this:

It still only returns the first page. Nothing more.

What am I doing wrong? Please help

Answer 1

Hello @Jon Carlisle ,
Thanks for the question and using MS Q&A platform.

As we understand you want help with pagination in copy activity.
You have specified the absoluteUrl pagination rule, but you need to write a URL query parameter. While this may be possible with absoluteURL , it is much easier if you use the "QueryParameters" type pagination rule. See below image and/or pipeline JSON excerpt.

"typeProperties": {  
                    "source": {  
                        "type": "RestSource",  
                        "httpRequestTimeout": "00:01:40",  
                        "requestInterval": "00.00:00:00.010",  
                        "requestMethod": "GET",  
                        "paginationRules": {  
                            "QueryParameters.offset": "$.offset"  
                        }  
                    },  

Please do let me if you have any queries.

Thanks
Martin

---

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators