How to fix the request throttles, quto exceeded in the Azure Data pipeline built through data flows

Rohan Dussa 0

I have created a data flow using the source as a REST connector and sink as Azure SQL database. The linked service has pages in the URL for which I have applied the pagination rule. However, I am getting the below error when I run the pipeline for the data flow: Operation on target KeapContactData failed: {"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: Failure to read most recent page request:","Details":"com.microsoft.dataflow.Issues: DF-Rest_015 - Failure to read most recent page request: DF-REST_001 - Error response from server: Some({\n "code": "429",\n "message": "Quota Exceeded",\n "status": "Request Throttled",\n "details": null\n }), Status code: 429. Please check your request url and body. (url:https://api.infusionsoft.com/crm/rest/v1/contacts/,request body: None, request method: GET)\n\tat com.microsoft.dataflow.Utils$.failure(Utils.scala:76)\n\tat org.apache.spark.sql.execution.datasources.rest.RestClient.$anonfun$readResourcesWithDynamicPaging$1(RestClient.scala:88)\n\tat scala.util.Try$.apply(Try.scala:213)\n\tat org.apache.spark.sql.execution.datasources.rest.RestClient.readResourcesWithDynamicPaging(RestClient.scala:55)\n\tat org.apache.spark.sql.execution.datasources.rest.RestClient.readResources(RestClient.scala:27)\n\tat org.apache.spark.sql.execution.datasources.rest.RestRDD.compute(RestRDD.scala:20)\n\tat org.apache.spark.rdd.R"} From the description, I understand that the error is regarding the quota and request limit. However I am not able to resolve the issue. I have set the request interval time to 16000 ms to allow the service to navigate through pages.

I appreciate the help

Bhargava-MSFT 29,266 Reputation points Microsoft Employee

2024-01-31T20:41:34.68+00:00

Hello Rohan Dussa, Welcome to the Microsoft Q&A forum.

From the error message, REST API you are using has returned a 429 status code. This means that the request has been throttled due to exceeding the API's quotas. This can happen if you are making too many requests too quickly, or if you are requesting too much data at once.

If you are requesting a large amount of data at once, try reducing the amount of data requested by using filters or pagination to limit the number of records returned in each request.

also, please check the API documentation to see if there are any specific guidelines or recommendations for quotas that you need to adhere.

I hope this helps.
Rohan Dussa 0 Reputation points

2024-01-31T20:45:18.7933333+00:00

thank you for your response. The data needs to be brought at once but using pagination as there is a record limit of 1000 per page. How can I leverage "request timeout" and "request interval" to get maximum data?
Bhargava-MSFT 29,266 Reputation points Microsoft Employee

2024-02-01T00:26:01.7933333+00:00

You can consider increasing the value of request interval to a reasonable value so that the next request will wait for (request interval - time elapsed for getting & processing the response from previous request) before sending.

Request interval should be long enough to avoid triggering rate limiting or throttling, but short enough to retrieve the data in a reasonable amount of time. You can start with a value of 16000 ms, as you mentioned in your previous message, and adjust it as needed based on the API's response.

Request timeout- should be long enough to allow the API to retrieve all the data, but short enough to avoid timeouts or other issues. You can start with a value of 300000 ms(5 mins) and adjust basedon the size of the data and APIs response time.

Share via

How to fix the request throttles, quto exceeded in the Azure Data pipeline built through data flows