Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,196 questions
Figured it out. The AbsoluteUrl needed some tweak.
Pagination in Azure ADF
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 74%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
Venkat
6
Reputation points
I am running into issues not being able to fetch data from pages beyond 1.
The REST API returns data like this.
I have setup the Mappings section as follows:
I have setup the Pagination as follows:
This does not work and I do not get beyond the 1st page. There are 7 pages of data to be iterated through.
{count} votes
-
Venkat 6 Reputation points
2022-02-11T18:15:14.697+00:00 additional information: if I use the AbsoluteURL as $.response.links.next, I get an error
Operation on target Generate data into Lease file per Property failed: ErrorCode=UserErrorTypeInSchemaTableNotSupported,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to get the type from schema table. This could be caused by missing Sql Server System CLR Types.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.InvalidCastException,Message=Unable to cast object of type 'System.DBNull' to type 'System.Type'.,Source=Microsoft.DataTransfer.ClientLibrary,'
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 78%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDM%3C/text%3E%3C/svg%3E)
David Meulenbeld 6 Reputation points2022-06-16T12:38:26.343+00:00 If the absoluteurl is in the response body and $.xxx as value does not work. Try without $.
For me $.metadata.next didn't work while metadata.next did.
Sign in to comment
2 answers
Sort by: Most helpful
-
Venkat 6 Reputation points
2022-02-14T20:27:03.29+00:00 -
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(198.4, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
Mohammed Mukhtar Alam 1 Reputation point2022-10-16T19:06:22.933+00:00 Please share the tweak as I am facing the same issue
Sign in to comment -
-
MarkKromer-MSFT 5,206 Reputation points Microsoft Employee2022-02-14T07:29:29.26+00:00 We have to update the REST API docs for this, which we are working on currently.
Basically, the rules in Copy can support both "." and [''], e.g., AbsoluteUrl: $.{@odata.nextLink} and AbsoluteUrl: $['@odata.nextLink']
However, rules in dataflow only support "." for json path. e.g., AbsoluteUrl: $.{@odata.nextLink}