Pagination in Azure data Factory(By REST API) using nextpagetoken

Question

Pagination in Azure data Factory(By REST API) using nextpagetoken

teja gudipati 21

Hi Team,
Can someone help me with it?
I am trying to pull data through Rest API into an azure data lake,
In the copy activity, the API response I am getting has the next page token column, I am using this column in the Pagination rules, Just like below.
But the API is not fetching data from the next pages.
I don't know what I am doing wrong here

MartinJaffer-MSFT 25,571 Reputation points Microsoft Employee

2021-04-29T16:30:29.213+00:00

Hello @teja gudipati and welcome to Microsoft Q&A.

Looking at your mapping, the Pagination Value should be $.nextPageToken . You have it as $.Headers.nextPageToken. Where is Headers coming from?

Also, is the nextPageToken in the form of a URL, or a piece of a URL, or some other thing?
teja gudipati 21 Reputation points

2021-04-29T20:29:38.213+00:00

Hi @MartinJaffer-MSFT ,

Thanks for your response.

The header is coming from the API response.

It is not in URL format, It is just giving me the key like this
"GVW7W2IHU63RQ36MSKLINOEE3DZWRRW52OPCPQ7MVF65L3TUJFGZKRE2XJYQEFOSZSWOBV5FM3CBE==="

MartinJaffer-MSFT 25,571 Reputation points Microsoft Employee

2021-04-29T16:30:29.213+00:00

Hello @teja gudipati and welcome to Microsoft Q&A.

Looking at your mapping, the Pagination Value should be $.nextPageToken . You have it as $.Headers.nextPageToken. Where is Headers coming from?

Also, is the nextPageToken in the form of a URL, or a piece of a URL, or some other thing?
teja gudipati 21 Reputation points

2021-04-29T20:29:38.213+00:00

Hi @MartinJaffer-MSFT ,

Thanks for your response.

The header is coming from the API response.

It is not in URL format, It is just giving me the key like this
"GVW7W2IHU63RQ36MSKLINOEE3DZWRRW52OPCPQ7MVF65L3TUJFGZKRE2XJYQEFOSZSWOBV5FM3CBE==="

Answer 1

MartinJaffer-MSFT 25,571 Microsoft Employee

I think I see where the confusion is coming from now, @teja gudipati .

So the token comes in the body of the response, and needs to be placed into the header of the next request, if I understand correctly.
In this case, "AbsoluteURL" is not what you want to use.

The Key should be Headers['header_name_for_next_page_token'] , renames as appropriate.
The Value should be $.nextPageToken

The Key specifies where to insert pagination in the next request.
The Value specifies where to get the pagination info from the response.

teja gudipati 21 Reputation points

2021-05-03T00:05:44.513+00:00

Hi @MartinJaffer-MSFT ,
Thanks for the Reply.

now the data set is creating the duplicated data. each record is writing 10 times. when I use the pagination rule.

Any idea why it is happening?
MartinJaffer-MSFT 25,571 Reputation points Microsoft Employee

2021-05-03T16:48:03.2+00:00

Well, maybe we had a miscommunication, @teja gudipati .

To my understanding, "Headers" refers to the metadata part of an http request, like "Content-type", "Authorization", "Accept-Encoding", or "Cookie".

Did you by chance need the next page token to be part of the URL like "https:example.com/query/results?page={token}" ?

If the pagination is set up correctly, it could inifite-loop on the same page. That would duplicate data, but only do one page and not finish.
teja gudipati 21 Reputation points

2021-05-03T21:12:47.793+00:00

Hi @MartinJaffer-MSFT ,

Sorry, I am a beginner to APIs. I understand very little about the API's

Here is the URL, I use
`https://xxxxxxxxx/rest/v1/activities.json?access_token=&nextPageToken=DLBUWOITURPGPRQKOPV37TJ7UPOMB4R2APFHLTMADNX4CRADDASQ====&activityTypeIds=1

I have passed the access_token using Additional Header "Authorization",

I am facing the difficulty to pass the next page token,
I am using the Pagination rule and used
Key : Headers['nextPageToken']
value : $nextPageToken.
Then it is creating the infinite loop on the same page and writing 10 times each record.
Help me to correct this.

Thanks
Teja

Answer 2

MartinJaffer-MSFT 25,571 Microsoft Employee

@teja gudipati
For when the next page token goes into the URL, you need to use
QueryParameters['request_query_parameter']

So for

`https://xxxxxxxxx/rest/v1/activities.json?access_token=&nextPageToken=DLBUWOITURPGPRQKOPV37TJ7UPOMB4R2APFHLTMADNX4CRADDASQ====&activityTypeIds=1

it would be

QueryParameters['nextPageToken']

This inserts nextPageToken into the URL query parameters.

teja gudipati 21 Reputation points

2021-05-04T20:50:57.34+00:00

Hi @MartinJaffer-MSFT
After page 1 , The copy activity is writing constantly like below for 100 times ,but no results
{"requestId":"8d74#17939208384","success":true,"nextPageToken":"DLBUWOITURPGPRQKOPV37TJ7UORZ54DHSC67HJFGTSDKO2HQHBFA====","moreResult":false}
{"requestId":"671#1793920839c","success":true,"nextPageToken":"DLBUWOITURPGPRQKOPV37TJ7UORZ54DHSC67HJFGTSDKO2HQHBFA====","moreResult":false}
{"requestId":"be0c#179392083a9","success":true,"nextPageToken":"DLBUWOITURPGPRQKOPV37TJ7UORZ54DHSC67HJFGTSDKO2HQHBFA====","moreResult":false}

.

And at the end it is writing like
,"errors":[{"code":"606","message":"Max rate limit '100' exceeded with in '20' secs"}]}
Is it something wrong with API or I am doing it wrong?.
.
teja gudipati 21 Reputation points

2021-05-05T23:45:33.323+00:00

Hi @MartinJaffer-MSFT
After page 1 , The copy activity is writing constantly like below for 100 times ,but no results
{"requestId":"8d74#17939208384","success":true,"nextPageToken":"DLBUWOITURPGPRQKOPV37TJ7UORZ54DHSC67HJFGTSDKO2HQHBFA====","moreResult":false}
{"requestId":"671#1793920839c","success":true,"nextPageToken":"DLBUWOITURPGPRQKOPV37TJ7UORZ54DHSC67HJFGTSDKO2HQHBFA====","moreResult":false}
{"requestId":"be0c#179392083a9","success":true,"nextPageToken":"DLBUWOITURPGPRQKOPV37TJ7UORZ54DHSC67HJFGTSDKO2HQHBFA====","moreResult":false}

.

And at the end it is writing like
,"errors":[{"code":"606","message":"Max rate limit '100' exceeded with in '20' secs"}]}
Is it something wrong with API or I am doing it wrong?.

Pagination in Azure data Factory(By REST API) using nextpagetoken

2 answers

Activity