question

CameronSmith-5163 avatar image
0 Votes"
CameronSmith-5163 asked ShaikMaheer-MSFT commented

Data Factory Pagination for RFC 5988 complian REST API (Cisco Meraki)

I'm trying to implement Azure Data Factory's Copy Activity to copy data from an API to our SQL Data Warehouse. I have set up the source & sinks correctly so that when I trigger the pipeline it pulls and loads the first load of data but I am struggling with pagination.

The API I am trying to pull data from is Cisco Meraki (https://developer.cisco.com/meraki/api-v1/#!pagination/how-does-pagination-work-in-the-dashboard-api) which is a REST API that uses the RFC 5988 pagination method. This means it passes the pagination rules for the next page in a response header.

I run the following API call

 {<!-- -->{baseUrl}}/networks/:networkId/events?productType=wireless&perPage=1000

which passes the following as a link header response to dictate the next page

 <{<!-- -->{baseUrl}}/networks/:networkId/events?endingBefore=2021-10-19T13%3A03%3A06.769680Z&perPage=1000&productType=wireless>; rel=prev, <{<!-- -->{baseUrl}}/networks/:networkId/events?perPage=1000&productType=wireless&startingAfter=2021-10-19T13%3A44%3A01.586776Z>; rel=next

How can I implement the above in the pagination rules in ADF to get the next page of info? I've currently defined the rules as here 142466-image.png but it's not accessing any new information. No error message just still only the first page. Is this format even supported by ADF? I can't find anything online to state either way,

Thanks for any pointers,


azure-data-factory
image.png (29.6 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @CameronSmith-5163 ,

Just checking in to see if the below answer helped. If this answers your query, do click 130616-image.png and upvote 130671-image.png for the same. And, if you have any further query do let us know.

0 Votes 0 ·

Hi @CameronSmith-5163 ,

Following up to see if the below answer helped. If this answers your query, do click 130616-image.png and upvote 130671-image.png for the same. And, if you have any further query do let us know.

0 Votes 0 ·

1 Answer

ShaikMaheer-MSFT avatar image
0 Votes"
ShaikMaheer-MSFT answered

Hi @CameronSmith-5163 ,

Thank you for posting query on Microsoft Q&A Platform.

REST connector in ADF supports the following pagination patterns:

  • Next request’s absolute or relative URL = property value in current response body

  • Next request’s absolute or relative URL = header value in current response headers

  • Next request’s query parameter = property value in current response body

  • Next request’s query parameter = header value in current response headers

  • Next request’s header = property value in current response body

  • Next request’s header = header value in current response headers


Click here for reference document.

I have gone through below link you shared.
https://developer.cisco.com/meraki/api-v1/#!pagination/how-does-pagination-work-in-the-dashboard-api


It seems you need to compose your URL with parameters.
- perPage: The number of entries to be returned in the page (the current request)
- startingAfter: A token used by our server to indicate the starting "identifier" of the page (i.e. the data we return in this request will start immediately after the entry with this "identifier")
- endingBefore: A token used by our server to indicate the ending "identifier" of the page (i.e. the data we return in this request will end immediately before the entry with this "identifier")


You can compose your Pagination URL in loop fashion and use Copy activity inside loop to make API call to that particular page accordingly. Please refer below video where similar example defined.
Copy data from REST API which sends response in Pages using Azure data factory

If your next page URLs are coming as part of Response Body then you need to have a key "AbsoluteUrl" and pass your URL from response body. An example is discussed at below link.
https://docs.microsoft.com/en-us/answers/questions/236150/how-to-perform-pagination-in-azure-data-factory-us.html


REST connector pagination supports totally three Key types. Details of Keys which you can supply and Value you can supply are available in below screenshot.
143381-image.png

Hope this will help. Please let us know if any further queries.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators


image.png (69.2 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.