'm noticing rate limits and throttling issues during high usage.

Question

I'm working on integrating Azure OpenAI GPT-4o into my chatbot, but I'm noticing rate limits and throttling issues during high usage.

Even though I’ve checked my quota limits in the Azure portal, I still get 429 errors (Too Many Requests) when multiple users interact with the bot simultaneously. Would increasing my SKU tier help, or is there a way to optimize requests for better performance?

Answer 1

Hi there

Try upgrading to a higher SKU tier can help, but first, check your quota limits in the Azure portal and request an increase if needed. try batching requests, reducing unnecessary API calls, and implementing caching for frequently used responses. Also, use Azure OpenAI Rate Limit headers to monitor usage patterns and adjust accordingly. If traffic is unpredictable, implementing a queueing mechanism can help distribute requests more efficiently.

If this helps kindly accpt the answer thanks.

Answer 2

Hello adan ameen,

When a deployment is created, the assigned TPM will directly map to the tokens-per-minute rate limit enforced on its inferencing requests. A Requests-Per-Minute (RPM) rate limit will also be enforced whose value is set proportionally to the TPM assignment using the following ratio:

6 RPM per 1000 TPM.

The flexibility to distribute TPM globally within a subscription and region has allowed Azure OpenAI Service to loosen other restrictions:  

1. Increase TPM from model deployment to avail higher RPM and rate limit failure threshold.
2. Create a multiple regions to deal regional outages, you can create outage alert from Azure status and perform remedial steps accordingly.
3. Lessen your input query size and reduce max_token and
4. Adopt retry in code

Reference - https://cookbook.openai.com/examples/how_to_handle_rate_limits

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

Thank you!