How to pass Pagination rules during a rest api calls in copy data activity?

Question

Hi,

I have a pipeline which basically firsts gets a access token by making a http request and once I have the token I'm doing a get request to an API with the access token as a header.

As the REST API o/p is limited I need to make multiple requests for pagination so I did follow the steps mentioned here : https://learn.microsoft.com/en-us/azure/data-factory/connector-rest#pagination-support

There are few problems with the NextPageUrl which i receive from the REST get request, The NextPageUrl starts with http and based on my experience if the Linked_service Baseurl does not start with HTTPS it does not accept the headers passed so I need to change the url from this "http://xyz.com/api/v1/?page=1" to "https://xyz.com/api/v1/?page=1" so i tried the replace expression but It says I cant write it this way "@replace('$.ConnectResponse.Metadata.Paging.NextPageURL', 'http', 'https')" or "@replace($.ConnectResponse.Metadata.Paging.NextPageURL, 'http', 'https')"

Please advise how can I change the string?

Secondly I though okay let me pass the page number manually but how do I add a number to the current page number and stop once it reaches the max?

from my API O/p I get example TotalPages=3 and CurrentPage=1 number

so I though I could pass the queryparameter in pagination like below

key = QueryParameters.page and value = @ABDULLLAH ($.ConnectResponse.Metadata.Paging.NextPageURL, 1) ----- I've tried with quotes and without quotes --- and it does work as expected.

Please advise how can I fix this?

Lastly

Is this how i pass a header authorization token in pagination?

key = Headers['Authorization']

value = Bearer @{activity('Login').output.access_token}

I mean this is how I pass the header parameters for the rest source so I'm assuming I just need to pass it the same way correct me if I'm wrong.

Thank you in advance.

Answer 1

Here is one way to iterate over the pages, @stramzik .

Step 1, make a web call to find out how many total pages.

Step 2, using an array type variable, enumerate (make a list of numbers) the page numbers

Step 3, use a for-each loop to copy the pages in parallel.

and the inner activity is a copy activity. We pass the iteration's page number to the parameterized dataset using @{items()}

Because each page is now a separate copy activity, we need to parameterize the sink dataset so each page goes to a separate file, instead of overwriting each other.

Answer 2

No answer replied by mistake

Answer 3

Not sure here how web activity counts the total number of pages

Answer 4

anonymous user , @Anjali Maithani
Follow up information:

The example I used, was a web activit GET call to https://reqres.in/api/users

The page metadata was included in the output, see below.

Output  
{  
    "page": 1,  
    "per_page": 6,  
    "total": 12,  
    "total_pages": 2,  
    "data": [  
        {  
            "id": 1,  
            "email": "******@reqres.in",  
            "first_name": "George",  
            "last_name": "Bluth",  
            "avatar": "https://reqres.in/img/faces/1-image.jpg"  
        },  
        {  
            "id": 2,  
            "email": "******@reqres.in",  
            "first_name": "Janet",  
            "last_name": "Weaver",  
            "avatar": "https://reqres.in/img/faces/2-image.jpg"  
        },  
        ...  

Then I just extracted the value with

@{activity('Web1').output.total_pages}  

Answer 5

In case your API already uses filters such as page and page_size, you may give @item() in the value for page in the copy activity as shown below and you may not need to use pagenumber parameter in API source dataset.