Azure Data Factory pagination using QueryParameters cursor

Question

Hi, I'm trying to paginate a REST request using QueryParameters replacement feature.
Different from most examples that I'd found, I need to use an attribute from the first request, as a starting point for the next request.
For example:
1st (representing Page 1)
request: xpto.com/transactions
output:

[  
{"data": "has_more": true,   
"items": [{"index": 1, "key": "tran_a"},{"index": 2, "key": "tran_b"}]   
}]  

2nd (representing Page 2)
request: xpto.com/transactions?starting_after=tran_b
output:

[  
{"data": "has_more": false,   
"items": [{"index": 3, "key": "tran_c"},{"index": 4, "key": "tran_e"}]   
}]  

I tried to implement this logic using pagination on ADF, but since I can't do a first request passing the "starting_after" URL parameter, I thought using a pipeline expression would be a solution. I realized (and presume) the pagination replacement runs after expression evaluation, so I can't implement any logic to avoid sending this URL parameter.
I'm attaching a screenshot from my ADF source and the expression that I tried to use.

Thanks for any help.

@concat(if(equals(length('{next}'),6),'','&starting_after={next}'))  

Answer 1

Hello @Fluna ,
Thanks for the question and using MS Q&A platform.

As I understand you are having trouble in your pagination because your dataset is parameterized to need a value, but you can't get that value until the first response.

I believe we can work around that. We will change how you pass the parameters.
From the dataset, remove the {next} from the concat instead we will leverage the native queryParameters behavior.
In the queryparameter rule, the { } around {next} tell it to look in dataset or linked service for a place to substitute. If we exclude the { } and use next , it should, to my understanding, add next to the query parameter string once the value can be filled. Since you are using body, it won't be filled until the second call.

Please do let me if you have any queries.

Thanks
Martin

---

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators