Azure Data Factory - Retrieve next pagination link (decoded) from response headers in a copy data activity of Azure Data Factory

Rubin Shah 0 Reputation points
2023-02-02T10:54:40.6666667+00:00

I have created a copy data activity in azure data factory and this data pipeline pulls the data from an API (via REST activity source) and writes the response body (json) on a file kept in the azure blob storage. The API which I am fetching the response from, is paginated and the link to next page is sent in the response headers in response->headers->link.

This URL to next page is in the following general format: <https%3A%2F%2FsomeAPI.com%2Fv2%2FgetRequest%3FperPage%3D80%26sortOrder%3DDESCENDING%26nextPageToken%3DVAdjkjklfjjgkl>; rel="next"

I want to fetch the next page token present in the above URL and use it in the pagination rule.

I have tried using some pagination rules:

> AbsoluteURL = Headers.link But, this did not work as the entire encoded link shown above, is getting appended directly and hence, the pipeline throws an error.

> Query Parameters I have also tried to use the query parameters but could not get any result.

I have followed questions over stackoverflow and have read the documentations:

https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#pagination-support

Please help me with how can I access this next page token or what can be the pagination rule to support the scenario.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,266 questions
{count} votes

1 answer

Sort by: Most helpful
  1. MartinJaffer-MSFT 26,226 Reputation points
    2023-02-06T05:31:01.4766667+00:00

    @Rubin Shah Hello and welcome to Microsoft Q&A.

    I understand you are having difficulty with copy activity rest pagination.

    <https%3A%2F%2FsomeAPI.com%2Fv2%2FgetRequest%3FperPage%3D80%26sortOrder%3DDESCENDING%26nextPageToken%3DVAdjkjklfjjgkl>; rel="next"

    This form seemed familiar. I think it is the RFC 5988 form. See Example 7 of pagination. This has its own rule, and I don't think you need to do the other pagination rules. Adding other rules might overwrite the RFC 5988 rule.

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.