Azure Data Factory V2 Copy Data Pipeline Using REST API and Pagination - Error Code 2200, Details: ErrorCode = User Error More Than One Objects Returned

Question

I am trying to set up a copy data pipeline to pull data from a REST API and have a lot of problems.

I have the copy data job set up and it can run but will only pull in 20 records, which I assume is the first page from the API.

I have tried looking around and some guides say to add the AbsoluteUrl with the value of $.d._next

When I don't put in a value for the AbsoluteUrl I get this error with what looks to be all, or most, of the data written into the Detail section:

When I fill in the AbsoluteUrl section with the BaseUrl from the API I get the following error:

Any help would greatly be appreciated!!!!

Answer 1

Hi @Steven Garis ,

Thanks for posting query in Microsoft Q&A platform.

As per my understanding, You are trying to use pagination in Copy activity source but ending up with above issues. Please correct me if my understanding is wrong.

Pagination can be used when you REST API will send response data along with next page URL details.

REST connector supports the following pagination patterns.

Below thread has detailed discussion about pagination and how to use it along with steps and screenshots. Kindly check it once.
https://learn.microsoft.com/en-us/answers/questions/606235/azure-data-factory-pagination-rule-for-rest-api-wo.html

It seems your response may not sending next URL details at key $d.__next. kindly check your response once after seeing above thread discussion.

Hope this helps. Please let us know if any further queries.

------------

Please consider hitting Accept Answer. Accepted answers helps community as well.