Why doesn't Azure OpenAI support same token input and output limits as OpenAI?

Sam Cohan 5

I am trying to fine-tune a gpt-35-turbo model (version 613) using a training and validation file that were working fine on the same version of the model on OpenAI but it is failing due to the Azure version having an input token limit of 4096. Any idea why the Azure version has lower limits? Does Microsoft plan to bring capabilities on par with OpenAI or do I need to use another model type?

Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
AshokPeddakotla-MSFT 30,511 Reputation points

2024-07-25T09:03:35.0966667+00:00
Sam Cohan Greetings & Welcome to Microsoft Q&A forum!

Does Microsoft plan to bring capabilities on par with OpenAI or do I need to use another model type?

The reason for the input token limit of 4096 is due to the current quota and limits set by Azure OpenAI Service. We are committed to providing the best possible experience for our customers and we are constantly working to improve our services. We are always looking for ways to improve our services and bring capabilities on par with OpenAI.

The quota and limits are subject to change, and we anticipate that you will need higher limits as you move toward production and your solution scales. When you know your solution requirements, Quota increase requests can be submitted from the Quotas page of Azure OpenAI Studio. For other rate limits, submit a service request.

You can use the other available models as well with higher token limit if the specific limits is not sufficient.

To minimize issues related to rate limits, it's a good idea to use the following techniques:

Implement retry logic in your application.

Avoid sharp changes in the workload. Increase the workload gradually.

Test different load increase patterns.

Increase the quota assigned to your deployment. Move quota from another deployment, if necessary.

I hope this information helps.

Do let me know if you have any further queries.
Sam Cohan 5 Reputation points

2024-07-25T18:20:14.39+00:00

The answer above is cookie-cutter (likely AI-generated) and unhelpful. All I wanted to know is if there are concrete steps to take to have increased context window for fine-tuning gpt-35-turbo models (on par with what OpenAI provides).
AshokPeddakotla-MSFT 30,511 Reputation points

2024-07-26T03:48:42.34+00:00

Apologies if my response is not clear.

I have checked further and these are the limits set by default and no plans on modifying the default limits. As suggested earlier, You can use the other available models as well with higher token limit if the specific limits is not sufficient or increase the limits by submitting a request.

Let me know if that clarifies.

Share via

Why doesn't Azure OpenAI support same token input and output limits as OpenAI?