Data Factory Pagination for RFC 5988 complian REST API (Cisco Meraki)

Question

I'm trying to implement Azure Data Factory's Copy Activity to copy data from an API to our SQL Data Warehouse. I have set up the source & sinks correctly so that when I trigger the pipeline it pulls and loads the first load of data but I am struggling with pagination.

The API I am trying to pull data from is Cisco Meraki (https://developer.cisco.com/meraki/api-v1/#!pagination/how-does-pagination-work-in-the-dashboard-api) which is a REST API that uses the RFC 5988 pagination method. This means it passes the pagination rules for the next page in a response header.

I run the following API call

{<!-- -->{baseUrl}}/networks/:networkId/events?productType=wireless&perPage=1000  

which passes the following as a link header response to dictate the next page

<{<!-- -->{baseUrl}}/networks/:networkId/events?endingBefore=2021-10-19T13%3A03%3A06.769680Z&perPage=1000&productType=wireless>; rel=prev, <{<!-- -->{baseUrl}}/networks/:networkId/events?perPage=1000&productType=wireless&startingAfter=2021-10-19T13%3A44%3A01.586776Z>; rel=next  

How can I implement the above in the pagination rules in ADF to get the next page of info? I've currently defined the rules as here but it's not accessing any new information. No error message just still only the first page. Is this format even supported by ADF? I can't find anything online to state either way,

Thanks for any pointers,

Answer 1

Hi @Cameron Smith ,

Thank you for posting query on Microsoft Q&A Platform.

REST connector in ADF supports the following pagination patterns:

• Next request’s absolute or relative URL = property value in current response body
• Next request’s absolute or relative URL = header value in current response headers
• Next request’s query parameter = property value in current response body
• Next request’s query parameter = header value in current response headers
• Next request’s header = property value in current response body
• Next request’s header = header value in current response headers

Click here for reference document.

I have gone through below link you shared. https://developer.cisco.com/meraki/api-v1/#!pagination/how-does-pagination-work-in-the-dashboard-api

It seems you need to compose your URL with parameters.

• perPage: The number of entries to be returned in the page (the current request)
• startingAfter: A token used by our server to indicate the starting "identifier" of the page (i.e. the data we return in this request will start immediately after the entry with this "identifier")
• endingBefore: A token used by our server to indicate the ending "identifier" of the page (i.e. the data we return in this request will end immediately before the entry with this "identifier")

You can compose your Pagination URL in loop fashion and use Copy activity inside loop to make API call to that particular page accordingly. Please refer below video where similar example defined. Copy data from REST API which sends response in Pages using Azure data factory

If your next page URLs are coming as part of Response Body then you need to have a key "AbsoluteUrl" and pass your URL from response body. An example is discussed at below link. https://learn.microsoft.com/en-us/answers/questions/236150/how-to-perform-pagination-in-azure-data-factory-us.html

REST connector pagination supports totally three Key types. Details of Keys which you can supply and Value you can supply are available in below screenshot.

Hope this will help. Please let us know if any further queries.

---

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators