How to accomplish pagination with "rel = next"

sam nick 286

For a REST api endpoint, the client data feed, gives us the pagination value in the header. Please can you advise on how to accomplish this.

The relative URL for the dataset is

https://clientname.com/api/v2/tickets?updated_since=2019-01-19

When the above is executed (for testing) in Postman, i get the next page in the Headers under Link as
<https://clientname.com/api/v2/tickets?updated_since=2019-01-19&page=2>;rel="next"

I currently have my pagination parameters as

Any recommendation on how to resolve this please.

AnnuKumari-MSFT 31,061 Reputation points Microsoft Employee

2022-01-20T07:11:09.45+00:00

Hi @sam nick ,
Following up to see if the below suggestion was helpful, kindly do click Accept Answer and Up-Vote for the same. In case you have any further query please do let us know

1 answer

AnnuKumari-MSFT 31,061 Reputation points Microsoft Employee

2022-01-03T09:23:00.62+00:00
Hello @sam nick ,
Thank you for using Microsoft Q&A platform.

By seeing your query, it seems you are getting a response header under the property called "link" and having value as "https://clientname.com/api/v2/tickets?updated_since=2019-01-19&page=2". If that is true, then kindly do your pagination setting as below.

Kindly check below link to get complete idea of using pagination rules based on different requirements.
https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#pagination-support

Hope this will help. Please let us know if there is any further queries.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

Want a reminder to come back and check responses? Here is how to subscribe to a notification

If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
Please sign in to rate this answer.
sam nick 286 Reputation points

2022-01-03T13:15:39.427+00:00

@AnnuKumari-MSFT
I made the change as per your recommendation, but the pipeline fails with the below error.

Failure happened on 'Source' side. ErrorCode=RestSourceCallFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The HttpStatusCode 404 indicates failure. Request URL: https://clientname.com/%3Chttps://clientname.com/api/v2/tickets?page=2%3E;%20rel=%22next%22 Response payload:<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

sam nick 286 Reputation points

2022-01-03T13:18:36.357+00:00

This is my rest dataset

AnnuKumari-MSFT 31,061 Reputation points Microsoft Employee

2022-01-04T08:14:25.143+00:00

Hi @sam nick ,

From the error message it seems ADF is making API call to "https://clientname.com/%3Chttps://clientname.com/api/v2/tickets?page=2%3E;%20rel=%22next%22" , which is an incorrect URL . I think , In your case it should hit to this URL "https://clientname.com/api/v2/tickets?page=2" (Correct me if my understanding is wrong).

It seems the response header under "Link" was not getting next page URL in proper format. Its coming along with some special chararcers such as <,>," etc. and using Dynamic expressions to modify that string according to our need is not possible at this moment.

Still Could you please share your response header and copy activity source tab details to understand your case better.

sam nick 286 Reputation points

2022-01-05T01:24:21.233+00:00

@AnnuKumari-MSFT

AnnuKumari-MSFT 31,061 Reputation points Microsoft Employee

2022-01-05T07:23:49.23+00:00

Hi @sam nick ,
Thanks for sharing more details along with screenshot . As we assumed , Link for the next page is coming with some symbols like :
'< , >;rel="next"'. Could you please ask your Source team to provide the URL without these extra symbols if possible. So that the Link contains only necessary characters in URL. i.e. "https://clientname.com/api/v2/tickets?per_page=100&page=9" as it's not possible to dynamically transform the link in pagination rule for Header. Meanwhile, I will try to see any other workarounds connecting to internal team.

sam nick 286 Reputation points

2022-01-05T11:44:07.307+00:00

Hi @AnnuKumari-MSFT . Unfortunately, Its not possible to have the client make those changes. Hope to hear any workarounds from your internal team.

AnnuKumari-MSFT 31,061 Reputation points Microsoft Employee

2022-01-12T11:18:30.177+00:00

Hi @sam nick ,
Unfortunately, There is no way to handle this scenario directly using pagination option in copy activity.

You might try to use a web activity to get the content of the API and then using a copy activity inside a Foreach block will be helpful if the next page URLs are consistent.

For example: Let's say this is the first landing URL 'https://api.instantwebtools.net/v1/passenger?page=0&size=10' and changing 'page=1' will give next API URL. In order to go with this approach, you should have the information of total number of pages your API contains.

For more details , kindly check this video: https://www.youtube.com/watch?v=EbURwy2a8dM&t=5s

Appreciate if you could share the details issue you are facing because of not having the capability of writing dynamic expressions under value section of header attribute in pagination rule on our Data Factory feedback channel, which would be open for the user community to upvote & comment on. This allows our product team to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

AnnuKumari-MSFT 31,061 Reputation points Microsoft Employee

2022-01-19T11:40:16.367+00:00

Hi @sam nick ,
Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.
Sign in to comment