Azure Data Factory Copy Data Pagination Rule EndCondition Doesn't work for empty body

Question

I am trying to set up pagination for a rest api data source.

The only way to know that there is no more data is by incrementing the page count until no data is returned. I cannot use any headers or metadata to determine the end of pagination.

an empty response simply looks like this:

I have set the pagination rules to the following:

the QueryParameter condition works in isolation. The EndCondition fails in isolation and when paired with the QueryParameter condition. in both cases, the error is the following:

Error code

23363

More than one object returned.\r\n'System.Collections.Generic.List`1[Newtonsoft.Json.Linq.JObject]'

I have no way to investigate further. What is wrong and how do I fix this?

Answer 1

@Aaron Carter Welcome to Microsoft Q&A forum and thanks for reaching out here.

I just did a quick test using this REST API - https://pokeapi.co/api/v2/pokemon

And noticed that if we configure the end condition correctly, the pagination is terminated when no data is returned.

The is the response from the API which has the next page details and below is how I configured the pagination settings which worked for me.

First page response:

Last page response where no data is returned except few fields:

Below is pagination configuration:

Have you tried configuring end condition as NonExist ? If not please give a try and let me know how it goes.
I am pretty sure this is something to do with the configuration versus your API response at for the last call.

It would help me understand better if you can share a sample first page and last page response of your API (Please mask any sensitive data in this public forum) to better understand the structure of it so that we can assist in configuring your pagination settings.

Please refer to below image to better understand what end condition to be used for the last response types:

I see that you only configured $ sign in end condition. Isn't there a name for your next page field? or is your last response only contains [] ? instead of an object {} ?

Thanks