Azure Data Factory add logical operators with JSONPath expression values in dynamic content field

Question

To realize pagination in my pipeline where I execute a REST GET Request I need to pass the next_offset value from the output to the PaginationRules.

I can do this by typing $.next_offset to the dynamic content field but I get an error when the pagination is finished and returns -1 as next_offset value.

Unfortunately I get an exception when the value is lower then zero. Instead it only succeed when the HTTP status code is 204 (No Content), or any of the JSONPath expressions in "paginationRules" returns null.

Because of that I wanted to add an If condition to the dynamic content field which looks like this now:

@if(equals('$.next_offset','-1'),'$.next_offset',null)
Unfortunately it is not possible to use the JSON path expression like this inside of a logical function I guess.

I would be glad if you can tell me how a JSON Path expression needs to be entered in logical functions.

Many Thanks in advance!

Answer 1

Hello @Bryan Tuescher ,
Thanks for the ask and using the Microsoft Q&A platform .
As I understand the ths issue is happens when you are in the last page . I think the $.next_offset should return null , is this not happening ?

I think what you have called out is very much inline as to what is called out here ,
"The connector will stop iterating when it gets HTTP status code 204 (No Content), or any of the JSONPath expressions in "paginationRules" returns null."

Is this an API which is on the internet , just wanted to know if we can try to repro the use case here .

Thanks
Himanshu

Answer 2

Hello Himanshu,
thank you for your help.

It would be great to have more opportunities to customize the pagination fuction, because there are REST APIs (like the one I trigger) which return -1 in the last page. There is no opportunity to catch this because the function is limited to HTTP code 204 or null.

I have created a workaround by querying the total number of records before the GET call and using the number of records per page to calculate the number of passes required.

Best regards,
Bryan