Azure Data Factory / Synapse: paging method for copy activity and data flow source work diffrently

Question

Hi everyone,

me again fighting with the REST sources in synapse.

I finally discovered how paging methods work in a copy activity. This is the projection of the payload coming from the graph api:

If more than 100 records in the value collection are returned, then a @.odata.nextLink attribute is included with the value of the next page.

Then I just used:

And this is working properly returning in total 128 records.

If I try to do the same in a data flow with a source using a REST dataset is not working and only 100 records are returned:

I also tried with different combinations like

$.@odata.nextLink  

Or setting the value dropdown to None, changing the Document form, etc. with the same result.

Any help will be appreciated.

Best regards,
Paul

Answer 1

Hello @Paul Hernandez and welcome back. Our expert just responded to an extremely similar question at:
https://learn.microsoft.com/en-us/answers/questions/732455/index.html

We have to update the REST API docs for this, which we are working on currently.

Basically, the rules in Copy can support both "." and [''], e.g., AbsoluteUrl: $.{@odata.nextLink} and AbsoluteUrl: $['@odata.nextLink']

However, rules in dataflow only support "." for json path. e.g., AbsoluteUrl: $.{@odata.nextLink}

Answer 2

Hi everyone,

this is not a complete answer but I least I can tell you how I solved it.

I think the major problem here is the "@" sign and the "." in the attribute name:

@odata.nextLine  

So in order to make the dataflow source work I used this notation:

Please note I have selected "None" and then entered:

body.{@odata.nextLinkl}  

That should be equivalent to this:

And also to this:

but it is NOT!

JSONPath supports both dot and brackets notation but the UI is failing when you have special characters in an attribute name. At least those are my findings at the moment.

I hope that helps someone.

Best regards,
Paul

Answer 3

@ Paul Hernandez
You're Certified OG for this.

Thank you.