Assistant stuck at "rate_limit_exceeded"

Romanowski, Tomasz 60

I've created an assistant with function calls. The first thread run poll succeeds with the correction function call. I execute the function, feed it back to the thread run. I get an error: rate_limit_eceeded "Please try again in XX seconds" . I parse out the timeout, sleep, then poll again only to get the exact same error.

YutongTie-MSFT 52,866 Reputation points

2024-08-07T14:20:19.9966667+00:00
Hello Romanowski,

Thanks for reaching out to us, generally, Azure OpenAI computes an estimated max processed-token count that includes the following:

Prompt text and count

The max_tokens parameter setting

The best_of parameter setting

As requests come into the deployment endpoint, the estimated max-processed-token count is added to a running token count of all requests that is reset each minute. If at any time during that minute, the TPM rate limit value is reached, then further requests will receive a 429 response code until the counter resets. For more details, see Understanding rate limits.

To minimize issues related to rate limits, it's a good idea to use the following techniques:

Set max_tokens and best_of to the minimum values that serve the needs of your scenario. For example, don’t set a large max-tokens value if you expect your responses to be small.

Use quota management to increase TPM on deployments with high traffic, and to reduce TPM on deployments with limited needs.

Implement retry logic in your application.

Avoid sharp changes in the workload. Increase the workload gradually.

Test different load increase patterns.

Also, see A Guide to Limits, Quotas, and Best Practices for more details. As each request is received, Azure OpenAI computes an estimated max processed-token count that includes the following:

Prompt text and count

The max_tokens parameter setting

The best_of parameter setting

As requests come into the deployment endpoint, the estimated max-processed-token count is added to a running token count of all requests that is reset each minute. If at any time during that minute, the TPM rate limit value is reached, then further requests will receive a 429 response code until the counter resets. For more details, see Understanding rate limits.

To minimize issues related to rate limits, it's a good idea to use the following techniques:

Set max_tokens and best_of to the minimum values that serve the needs of your scenario. For example, don’t set a large max-tokens value if you expect your responses to be small.

Use quota management to increase TPM on deployments with high traffic, and to reduce TPM on deployments with limited needs.

Implement retry logic in your application.

Avoid sharp changes in the workload. Increase the workload gradually.

Test different load increase patterns.

Also, see A Guide to Limits, Quotas, and Best Practices for more details.

Please let us know if your issue has been resolved, if not, I would highly recommend you raise a support ticket.

If you have no support plan, we are happy to enable you one.

Regards,

Yutong

Share via

Assistant stuck at "rate_limit_exceeded"

Your answer