Help with REST api pagination rules

Question

I am trying to pull data from this Hubspot endpoint. I am using this MS guide.

I want to pull all records of data from the endpoint to an Azure SQL Database table through an ADF pipeline with a copy activity. I want to use the pagination support inside the connector instead of a loop, unless that is the only option. As authentication I use a bearer token in the header. For testing purposes "hapikey=demo" can be added.

What I need:

• Every next call after the first needs to have the parameter "&vidOffset=" set to the value provided in the body response under "$.vid-offset".
• I need the pagination to run till "$.has-more" is set to false in the body response. In my pipeline I temporary used a max of 5 requests.

This is my copy activity:

These is my REST connector settings:

This is the error I get:

Some help would mean much to me!

Answer 1

Hello @Niek ,
Thanks for the question and using MS Q&A platform.

Looks like you want some help with debugging Copy Activity REST API pagination.

Hmm this is tricky... everything looks correct...
The message indicates a literal {offset} was sent instead of the value. Maybe we are not hitting the property path?
Do we need to add $. in the value to make $.vid-offest ... no that made duplicate...

Can we try changing "QueryParameters" to "AbsoluteUrl" , but keeping everything else the same? I think this will fix it.

Please do let me if you have any queries.

Thanks
Martin

---

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Answer 2

@MartinJaffer-MSFT
I've found the solution to my problem.

With the dataset, I left out the variable call of {offset}. This variable did only work when using range settings in the QueryParameters pagination.
With the source settings I listed my endpoint parameter and the value I wanted to parse. The pagination rule adds the given queryparameter to the url.