Data Factory Pagination for RFC 5988 complian REST API (Cisco Meraki)

Cameron Smith 6 Reputation points
2021-10-21T10:53:50.267+00:00

I'm trying to implement Azure Data Factory's Copy Activity to copy data from an API to our SQL Data Warehouse. I have set up the source & sinks correctly so that when I trigger the pipeline it pulls and loads the first load of data but I am struggling with pagination.

The API I am trying to pull data from is Cisco Meraki (https://developer.cisco.com/meraki/api-v1/#!pagination/how-does-pagination-work-in-the-dashboard-api) which is a REST API that uses the RFC 5988 pagination method. This means it passes the pagination rules for the next page in a response header.

I run the following API call 

{<!-- -->{baseUrl}}/networks/:networkId/events?productType=wireless&perPage=1000

which passes the following as a link header response to dictate the next page 

<{<!-- -->{baseUrl}}/networks/:networkId/events?endingBefore=2021-10-19T13%3A03%3A06.769680Z&perPage=1000&productType=wireless>; rel=prev, <{<!-- -->{baseUrl}}/networks/:networkId/events?perPage=1000&productType=wireless&startingAfter=2021-10-19T13%3A44%3A01.586776Z>; rel=next

How can I implement the above in the pagination rules in ADF to get the next page of info? I've currently defined the rules as here 142466-image.png but it's not accessing any new information. No error message just still only the first page. Is this format even supported by ADF? I can't find anything online to state either way,

Thanks for any pointers,

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,487 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. ShaikMaheer-MSFT 37,896 Reputation points Microsoft Employee
    2021-10-25T06:45:04.053+00:00

    Hi @Cameron Smith ,

    Thank you for posting query on Microsoft Q&A Platform.

    REST connector in ADF supports the following pagination patterns:

    • Next request’s absolute or relative URL = property value in current response body
    • Next request’s absolute or relative URL = header value in current response headers
    • Next request’s query parameter = property value in current response body
    • Next request’s query parameter = header value in current response headers
    • Next request’s header = property value in current response body
    • Next request’s header = header value in current response headers

    Click here for reference document.

    I have gone through below link you shared. https://developer.cisco.com/meraki/api-v1/#!pagination/how-does-pagination-work-in-the-dashboard-api

    It seems you need to compose your URL with parameters.

    • perPage: The number of entries to be returned in the page (the current request)
    • startingAfter: A token used by our server to indicate the starting "identifier" of the page (i.e. the data we return in this request will start immediately after the entry with this "identifier")
    • endingBefore: A token used by our server to indicate the ending "identifier" of the page (i.e. the data we return in this request will end immediately before the entry with this "identifier")

    You can compose your Pagination URL in loop fashion and use Copy activity inside loop to make API call to that particular page accordingly. Please refer below video where similar example defined. Copy data from REST API which sends response in Pages using Azure data factory

    If your next page URLs are coming as part of Response Body then you need to have a key "AbsoluteUrl" and pass your URL from response body. An example is discussed at below link. https://learn.microsoft.com/en-us/answers/questions/236150/how-to-perform-pagination-in-azure-data-factory-us.html

    REST connector pagination supports totally three Key types. Details of Keys which you can supply and Value you can supply are available in below screenshot. 143381-image.png

    Hope this will help. Please let us know if any further queries.

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    0 comments