Azure OpenAI Service
An Azure service that provides access to OpenAI’s GPT-3 models with enterprise capabilities.
4,092 questions
Assistant stuck at "rate_limit_exceeded"
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 28.000000000000004%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERT%3C/text%3E%3C/svg%3E)
Romanowski, Tomasz
65
Reputation points
I've created an assistant with function calls. The first thread run poll succeeds with the correction function call. I execute the function, feed it back to the thread run. I get an error: rate_limit_eceeded "Please try again in XX seconds" . I parse out the timeout, sleep, then poll again only to get the exact same error.
Azure OpenAI Service
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 1%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EYM%3C/text%3E%3C/svg%3E)
YutongTie-MSFT 53,971 Reputation points Moderator2024-08-07T14:20:19.9966667+00:00 Hello Romanowski,
Thanks for reaching out to us, generally, Azure OpenAI computes an estimated max processed-token count that includes the following:
- Prompt text and count
- The max_tokens parameter setting
- The best_of parameter setting
As requests come into the deployment endpoint, the estimated max-processed-token count is added to a running token count of all requests that is reset each minute. If at any time during that minute, the TPM rate limit value is reached, then further requests will receive a 429 response code until the counter resets. For more details, see Understanding rate limits.
To minimize issues related to rate limits, it's a good idea to use the following techniques:
- Set max_tokens and best_of to the minimum values that serve the needs of your scenario. For example, don’t set a large max-tokens value if you expect your responses to be small.
- Use quota management to increase TPM on deployments with high traffic, and to reduce TPM on deployments with limited needs.
- Implement retry logic in your application.
- Avoid sharp changes in the workload. Increase the workload gradually.
- Test different load increase patterns.
Also, see A Guide to Limits, Quotas, and Best Practices for more details. As each request is received, Azure OpenAI computes an estimated max processed-token count that includes the following:
- Prompt text and count
- The max_tokens parameter setting
- The best_of parameter setting
As requests come into the deployment endpoint, the estimated max-processed-token count is added to a running token count of all requests that is reset each minute. If at any time during that minute, the TPM rate limit value is reached, then further requests will receive a 429 response code until the counter resets. For more details, see Understanding rate limits.
To minimize issues related to rate limits, it's a good idea to use the following techniques:
- Set max_tokens and best_of to the minimum values that serve the needs of your scenario. For example, don’t set a large max-tokens value if you expect your responses to be small.
- Use quota management to increase TPM on deployments with high traffic, and to reduce TPM on deployments with limited needs.
- Implement retry logic in your application.
- Avoid sharp changes in the workload. Increase the workload gradually.
- Test different load increase patterns.
Also, see A Guide to Limits, Quotas, and Best Practices for more details.
Please let us know if your issue has been resolved, if not, I would highly recommend you raise a support ticket.
If you have no support plan, we are happy to enable you one.
Regards,
Yutong
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 0%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ3%3C/text%3E%3C/svg%3E)
JoeS-3935 21 Reputation points2024-11-19T02:26:56.5533333+00:00 Hi,
Im working with the Assistants API and I am getting a similar error message. I am well under the token count. The Tokens per Minute limit I have is 30,000 tokens but when I do a tools call and get back the results and run a thread against it I hit the limit even though it is well below the limit. Sometimes it is even at 15,000 tokens and still not working. I have tried various things, but nothing seems to fix the issue.
Is there some special difference that the Assistants api is counting or using tokens in the threads and runs? Even when it is a completely new thread and first message, I still hit the same error. None of the answers above have helped with the error.
Please let me know if there is anything else that can help.
-
JoeS-3935 21 Reputation points
2024-11-19T04:35:46.21+00:00 Hi,
Im working with the Assistants API and I am getting a similar error message. I am well under the token count. The Tokens per Minute limit I have is 30,000 tokens but when I do a tools call and get back the results and run a thread against it I hit the limit even though it is well below the limit. Sometimes it is even at 15,000 tokens and still not working. I have tried various things, but nothing seems to fix the issue.
Is there some special difference that the Assistants api is counting or using tokens in the threads and runs? Even when it is a completely new thread and first message, I still hit the same error. None of the answers above have helped with the error.
Please let me know if there is anything else that can help.
-
YutongTie-MSFT 53,971 Reputation points Moderator
2024-11-19T18:44:21.48+00:00 Hello @JoeS-3935 I would suggest you open a new thread for your case since this issue is outdated. Thanks a lot.
Sign in to comment
Sign in to answer