Azure Synapse/DF Copy Activity Error - REST Pagination End Condition

Question

I am trying to get REST pagination working for a case where I have an undetermined amount of records. I have my limit/offset working, but the end-condition logic does not seem to work. My data is returned in an array like so:

[
    {
        "_id": "64cbe649d15d25f03d7182a4"
    },
    {
        "_id": "64cbe89ed15d25f03d721a22"
    }
]

When an offset is reached with no data, an empty array is returned in the body like so:

[]

The following error is returned in any of the following Rules:
Error:

Error code: 23363 Activity ID <> Details: More than one object returned.\r\n'System.Collections.Generic.List`1[Newtonsoft.Json.Linq.JObject]'

Rules:
$ = Empty, $ = NonExist, $[0] =Empty, $[0] = NonExist, $[_id] =Empty, $[id] = NonExist

What end condition do I need to utilize to stop when an empty array is detected?

Answer 1

@Tom Anderson Welcome to Microsoft Q&A forum and thanks for reaching out here.

I'm not sure if the response you have shared is complete JSON payload or just a part of it. But in general, below will be settings. For more information about the end condition, the ADF public document has few examples called out, please review them and let us know if your scenario didn't cover in that.

Ref doc: ADF REST Connector - Variables are in AbsoluteUrl/QueryParameters/Headers, the end variable is not pre-defined and the end condition is based on the response

Hope this info helps. In case if you are still blocked, please help us with a sample free API that can provide a response JSON payload as your so that we can try to reproduce and assist accordingly. Let us know how it goes.

---

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.