Rest API pagination help in Azure Data Factory.

Question

Hi Team,

We have a use case where we need to fetch the data from an API using pagination and write it to ADLS.
I went through multiple blogs and did some poc's but pagination didn't work. Can someone please help me here

Below is the Poc that am trying

1. Call token URL with ClientID and Secret to fetch TOKEN
2. Call session URL with TOKEN as Bearer and fetch the Session ID
3. Call the API with Session ID as authorization in header to fetch the data.
4. If the API response body has nextPage details, then it needs to append the nextPage URI in the response body with base URL and call the API for next page response.
5. This should repeat till the nextPage flag is empty in the response body.
6. Once the nextPage is empty all the response should be merged and written to a single file in ADLS

Thanks in advance

Answer 1

Hi @Sandeep ,

Thank you for posting query in Microsoft Q&A Platform.

I would encourage you to check below link where different ways of pagination explained with detailed steps. Kindly choose any approach from there which better fits for you.

• Azure Data Factory pagination rule (for Rest API) - Workaround
• Copy data from REST API which sends response in Pages using Azure data factory - It's a video which explains complete steps detailly.
• Pagination support in Copy activity in Azure data factory - It's an official documentation where different ways of pagination explained with examples.

Hope this helps. If not, feel free to share your implementation details along with screenshots and configurations so that we can spot issue better and help.

-------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification