Azure ADF - Copy Activity - Next page url issue

Question

Azure ADF - Copy Activity - Next page url issue

Sreenivas Angara 0

Hello,

Please help with this issue.

I am trying to implement pagination in Copy Activity.

The response has the following data

    "meta": {
        "success": true,
        "pagination": {
            "per_page": 120,
            "next_page_uri": "/api/v2/activity_logs.json?apaKxevTVh19A=Mg%3D%3D%0A&min_last_activity_at=%222023-10-09T013%3A48%3A00%22&per_page=120&search%5Bfeature_type%5D%5B%5D=appointment&search%5Bfeature_type%5D%5B%5D=messenger&search%5Bfeature_type%5D%5B%5D=outreach"
        }
    }

Meta is part of the Body of the JSON response

Copy Activity, under source -> Pagination Rules, I am trying to build this expression, so that I can build the Absolute Url. Please provide guidance, on how I can build the absolute url, using relative url from the response message.

User's image

1 answer

Your answer

Answer 1

ShaikMaheer-MSFT 38,631 Microsoft Employee Moderator

Hi

Thank you for posting query in Microsoft Q&A Platform.

You need to consider having variables in Absolute URL and then pass values to that variables from Pagination rules.

Below is the link of official documentation which explains how to do that.

https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#example-2variables-in-absoluteurl

In your above screenshot, you are not using any variable in the Absolute URL part under pagination. I believe you might have created variables for Absolute URL as part of linked service and dataset. Kindly recheck from above link.

Hope this helps. Please try same and let me know how it goes.

Please consider hitting Accept Answer button. Accepted answers help community as well.

Sreenivas Angara 0 Reputation points

2023-10-10T11:36:10.7433333+00:00

ShaikMaheer,

The documentation you pointed, shows using variables when there are page numbers in the response or part of the url

In my case, there are no page numbers.

Sreenivas Angara
Sreenivas Angara 0 Reputation points

2023-10-10T13:09:14.9833333+00:00
ShaikMaheer,

I saw your YouTube videos and they are good. Keep it up and they are helping a number of souls.

I finally got it working with this setting. However, I am uncomfortable to move forward.

Behavior noticed by me, accidentally of course,

If 'next_page_uri' has a relative url value then ADF builds complete url.

If 'next_page_uri' has a complete url value then ADF uses it as is.

Please confirm.

If this is the case, please document it. It will make many folks happy.
ShaikMaheer-MSFT 38,631 Reputation points Microsoft Employee Moderator

2023-10-16T09:12:32.9833333+00:00

Hi Sreenivas Angara,

Glad to know you are able to solve it. Thank you for kind words. Kindly consider marking it as accepted answer. Accepted answers help community as well.

API should be consistent of sending next_page_uri in response json. Usually APIs will be having consistent way to sending properties. So once you configure it should work for all pages.

Above documentation provided will have other scenarios also converted. That also helps you if you explore. Kindly check them.
Sreenivas Angara 0 Reputation points

2023-10-16T13:24:16.4333333+00:00

Shaik,

I cannot make this an 'Accept Answer'.

I just hacked something that is not documented. Reach out to internal dev teams and confirm that this is the right approach.

If you confirm that this is the right way to go about it, then I will. Take a big-picture approach. There are many professionals who posted the same question on different forums. Microsoft providing a stamp of approval will help the whole community.

Please be a Big Brother and take this approach.

Sreenivas Angara
ShaikMaheer-MSFT 38,631 Reputation points Microsoft Employee Moderator

2023-10-20T17:59:43.3333333+00:00

Hi Sreenivas Angara, Sure. Thank you for suggesting and sharing your implementation details. I will forward this to internal team and see if I can influence to include this as part of documentation.

Consider hitting Accept Answer button. Accepted answers help community as well. Thank you.

Share via

Azure ADF - Copy Activity - Next page url issue

1 answer

Your answer