Pagination in Azure data Factory(By REST API) using nextpagetoken

teja gudipati 21 Reputation points
2021-04-29T01:29:26.493+00:00

Hi Team,
Can someone help me with it?
I am trying to pull data through Rest API into an azure data lake,
In the copy activity, the API response I am getting has the next page token column, I am using this column in the Pagination rules, Just like below.
But the API is not fetching data from the next pages.
I don't know what I am doing wrong here

92363-image.png
92365-image.png

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,269 questions
{count} votes

2 answers

Sort by: Most helpful
  1. MartinJaffer-MSFT 24,006 Reputation points Microsoft Employee
    2021-04-30T15:31:35.28+00:00

    I think I see where the confusion is coming from now, @teja gudipati .

    So the token comes in the body of the response, and needs to be placed into the header of the next request, if I understand correctly.
    In this case, "AbsoluteURL" is not what you want to use.

    The Key should be Headers['header_name_for_next_page_token'] , renames as appropriate.
    The Value should be $.nextPageToken

    The Key specifies where to insert pagination in the next request.
    The Value specifies where to get the pagination info from the response.


  2. MartinJaffer-MSFT 24,006 Reputation points Microsoft Employee
    2021-05-04T19:12:19.717+00:00

    @teja gudipati
    For when the next page token goes into the URL, you need to use
    QueryParameters['request_query_parameter']

    So for

    `https://xxxxxxxxx/rest/v1/activities.json?access_token=&nextPageToken=DLBUWOITURPGPRQKOPV37TJ7UPOMB4R2APFHLTMADNX4CRADDASQ====&activityTypeIds=1  
    

    it would be

    QueryParameters['nextPageToken']  
    

    This inserts nextPageToken into the URL query parameters.