Pagination in Azure ADF

Venkat 6 Reputation points
2022-02-11T18:03:06.917+00:00

I am running into issues not being able to fetch data from pages beyond 1.

The REST API returns data like this.
173630-image.png

I have setup the Mappings section as follows:

173674-image.png

I have setup the Pagination as follows:
173691-image.png

This does not work and I do not get beyond the 1st page. There are 7 pages of data to be iterated through.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,473 questions
{count} votes

2 answers

Sort by: Most helpful
  1. Venkat 6 Reputation points
    2022-02-14T20:27:03.29+00:00

    Figured it out. The AbsoluteUrl needed some tweak.

    1 person found this answer helpful.

  2. MarkKromer-MSFT 5,186 Reputation points Microsoft Employee
    2022-02-14T07:29:29.26+00:00

    We have to update the REST API docs for this, which we are working on currently.

    Basically, the rules in Copy can support both "." and [''], e.g., AbsoluteUrl: $.{@odata.nextLink} and AbsoluteUrl: $['@odata.nextLink']

    However, rules in dataflow only support "." for json path. e.g., AbsoluteUrl: $.{@odata.nextLink}

    0 comments No comments