Azure synapse Copy data with REST API Source failing with

Question

Azure synapse Copy data with REST API Source failing with

asked

I was trying to use the copy data activity in azure synapse where the source system is REST API with multiple pages. I used the pagination settings like the image attached, where the end condition is not known, so we put it as and when no data is returned by the API.

Below is the error I'm getting. Please help me resolve this issue.
{
"errorCode": "2200",
"message": "Failure happened on 'Source' side. ErrorCode=UserErrorMoreThanOneObjectsReturned,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=More than one objects returned\r\n{\r\n \"Id\": \"a8f4221f-7250-453f-80f7-207deaeaa0ea\",\r\n \"ChangeVersion\": 105422\r\n}\r\n{\r\n \"Id\": \"cfbaa3bf-0c90-477e-aacf-974ef425fa8d\",\r\n \"ChangeVersion\": 105423\r\n}\r\n{\r\n \"Id\": \"6c60adc7-461b-4f0c-994d-35ac95d12fb2\",\r\n \"ChangeVersion\": 105773\r\n}\r\n{\r\n \"Id\": \"20cbec8b-3d7f-4e42-bd19-c6e86f370d3c\",\r\n \"ChangeVersion\": 105779\r\n},Source=Microsoft.DataTransfer.ClientLibrary,'",
"failureType": "UserError",
"target": "Copy Deletes to stage1",
"details": []
}

Answer 1

answered

ShaikMaheer-MSFT 28,096 Microsoft Employee

Hi @Abhinav Gundapaneni ,

Thank you for posting query in Microsoft Q&A Platform.

Please Note, "end_condition" is user-defined, which indicates the condition that will end the pagination loop in the next HTTP request.

In your case if there is no end condition, then may be simply ignore it and see if that helps. But having an end condition based on response we get helps.

Currently now, the end condition which mentions actually see the property called data with values exists or not.

Always make sure the property which we are pointing in end condition should be present in our response json. For example, As per below end condition, The pagination ends when the value of the specific node in response is empty.

Hope this helps. Please let us know how it goes.

----------

Please consider hitting Accept Answer button. Accepted answers help community as well.

Abhinav Gundapaneni 1 Reputation point

2022-06-15T20:40:39.637+00:00

Hello @ShaikMaheer-MSFT ,

I'm using a 3rd party API that doesn't support end conditions as you mentioned. The only end condition is when the API returns no records (i.e. when offset crosses the number of records). Is there any way to use this end condition to use pagination?

Thanks,
Abhinav
ShaikMaheer-MSFT 28,096 Reputation points Microsoft Employee

2022-06-17T06:38:18.527+00:00

When API returns no record then I guess above mentioned screenshot condition which validates empty may work. Kindly a give a try and see.

Please let us know how it goes.

------------

Please consider hitting Accept Answer. Accepted answers help community as well.
Abhinav Gundapaneni 1 Reputation point

2022-06-27T13:03:50.203+00:00

@ShaikMaheer-MSFT I am still getting the same error when trying to use pagination using REST Dataset as source. My API is returning multiple JSON objects instead of a single primitive object as shown in the Documentation. Is there any way where we can set an end condition on the count of records returned or when the response does not return any records at all.

Azure synapse Copy data with REST API Source failing with

1 answer

Activity