Dealing with Throttling in Copy activity with source restapi

Question

Hi Guys - i have a copy activity that has source as a restapi and sink as a synapse DB table that fails with

{ "errorCode": "2200", "message": "Failure happened on 'Source' side. ErrorCode=RestCallFailedWithClientError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Rest call failed with client error, status code 429 429, please check your activity settings

so as per the source side documentation (https://csod-external.force.com/supportcentral/s/article/Data-Exporter-API-DEAPI-General-Information?language=en_US)the throttling seems to be set to 120 requests per minute.

I manage to get the pipeline to finish fine if I set the MaxRequestNumber Parameter to 119.

is there a way I could feed the odata.nextdata value into a variable and then resume for another 119 requests after using a wait for 1min. do I have to do that by looping through variables or has anyone used a combination of parameters on the source side between pagination and throttling to do this within copy stage itself ?

i also see the throttling values left in that session on postman - how can i expose them in my pipeline perhaps to compute how much is left ?

any suggestion would be appreciated. thanks in advance.

Answer 1

Hi Vamsi Madhav ,

Thankyou for using Microsoft Q&A platform and thanks for posting your question here.

As I understand your issue, you are getting an error: "Rest call failed with client error, status code 429" when you are copying the data from REST API to Synapse DB using copy activity.

To achieve this , your API should return the response in such a way that it contains some properties which describes the next range of request you want to fetch.

For example : If you take a look at this sample URL https://reqres.in/api/users?Page=1 , it contains the page number value and the API response gives the properties like total_pages which describes the total number of pages in the API , using which we can loop until the last page i.e. 2 in this case. Checkout this video to see implementation of the same: Copy data from REST API which sends response in Pages using Azure data factory

Similary , you need to check with your source API client/ admin team to help you get the properties that provides the range to iterate through the set of rows and read the data.

Additonal resources: https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/request-limits-and-throttling#error-code

As stated in the above document , properties like 'Retry-After' can specify the number of seconds your application should wait , or 'x-ratelimit-remaining' can determine the remaining number of requests allowed. However, you need to check if it is supported in the API you are using or not.

---

Hope it helps.