Help with REST api pagination rules

Niek 6 Reputation points
2022-11-28T13:57:32.8+00:00

I am trying to pull data from this Hubspot endpoint. I am using this MS guide.

I want to pull all records of data from the endpoint to an Azure SQL Database table through an ADF pipeline with a copy activity. I want to use the pagination support inside the connector instead of a loop, unless that is the only option. As authentication I use a bearer token in the header. For testing purposes "hapikey=demo" can be added.

What I need:

  • Every next call after the first needs to have the parameter "&vidOffset=" set to the value provided in the body response under "$.vid-offset".
  • I need the pagination to run till "$.has-more" is set to false in the body response. In my pipeline I temporary used a max of 5 requests.

This is my copy activity:
264768-adf-copy.png
These is my REST connector settings:
264843-adf-dataset.png
This is the error I get:
264844-adf-error.png

Some help would mean much to me!

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,160 questions
0 comments No comments
{count} votes

2 answers

Sort by: Most helpful
  1. MartinJaffer-MSFT 26,061 Reputation points
    2022-11-29T23:05:35.68+00:00

    Hello @Niek ,
    Thanks for the question and using MS Q&A platform.

    Looks like you want some help with debugging Copy Activity REST API pagination.

    Hmm this is tricky... everything looks correct...
    The message indicates a literal {offset} was sent instead of the value. Maybe we are not hitting the property path?
    Do we need to add $. in the value to make $.vid-offest ... no that made duplicate...

    Can we try changing "QueryParameters" to "AbsoluteUrl" , but keeping everything else the same? I think this will fix it.

    Please do let me if you have any queries.

    Thanks
    Martin


    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
      • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

  2. Niek 6 Reputation points
    2022-12-06T13:02:00.173+00:00

    @MartinJaffer-MSFT
    I've found the solution to my problem.

    With the dataset, I left out the variable call of {offset}. This variable did only work when using range settings in the QueryParameters pagination.
    With the source settings I listed my endpoint parameter and the value I wanted to parse. The pagination rule adds the given queryparameter to the url.

    267803-adf-source1.png

    0 comments No comments