ADF Rest API - How To Handle "too many requests" error (429)

Tomasz Swinarski 0 Reputation points
2023-02-27T14:54:11.87+00:00

Hello,

I am struggling with integrating API source into Azure SQL DB using Azure Data Factory. Problem is that I am getting "too many requests" error (429) randomly when trying to get data from API source. It looks like error is caused by some other API callers I have no control over.

That's why I would like to now how I can handle this error. According to documentation I should get response from API with Retry-After header specifying how many seconds i should wait before I make another call:

https://developer.zendesk.com/documentation/ticketing/using-the-zendesk-api/best-practices-for-avoiding-rate-limiting/

How can I implement it in below:

User's image

In general this flow works fine for other zendesk instances. But for this particular instance I am getting 429 error very often. I have added maxrequestnumber as a workaround since it reduces probability of getting this error. My goal is to be able to catch this error and wait with requesting next page for specified number of seconds (from retry-after header) and then continue with requests.

I would be grateful for any help you can provide.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,539 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. HimanshuSinha-msft 19,376 Reputation points Microsoft Employee
    2023-02-28T20:39:42.8433333+00:00

    Hello @Tomasz Swinarski , Thanks for the question and using MS Q&A platform. As I understand that the ask here is if we can catch the exception and the put a wait and then try to get started from the page where we get the exception. The challenge here you can only read the exception once you are out of the activity and so let's say the issue happened while calling the 10th page, then you will have to also catch the 10th page, so that you can start the start the pull from the 10th page only next time. I highly doubt if you can do this.

    I do suggest checking as to what is the Compute size are you using , please try to use the small option and see if that helps .

    User's image

    Also

    Try to increase the Request interval and see if that helps .

    User's image

    Thanks
    Himanshu

    Please accept as "Yes" if the answer provided is useful , so that you can help others in the community looking for remediation for similar issues.