What is the exact maximum input tokens of Azure GPT-3.5-turbo?

Tali.AI 20

We have an endpoint for Azure GPT-3.5-turbo version 0301, which is said to support 4096 max input tokens. But when we send an input with 10K tokens, it actually works and generates the output without giving any errors about exceeding max tokens. The question I have is, what is the actual maximum number of input tokens for GPT-3.5-turbo? Does Azure automatically send long inputs to a larger model?

AshokPeddakotla-MSFT 30,511 Reputation points

2023-09-01T04:24:33.0833333+00:00

Tali.AI Greetings & Welcome to Microsoft Q&A forum!

what is the actual maximum number of input tokens for GPT-3.5-turbo? Does Azure automatically send long inputs to a larger model?

The gpt-35-turbo model supports 4096 max input tokens and the gpt-35-turbo-16k model supports up to 16,384 tokens.

Azure does not automatically send long inputs to a larger model. If you send a prompt that exceeds this limit, you will receive an error.

Could you check again and update if you are still seeing this?
Tali.AI 20 Reputation points

2023-09-01T14:32:53.2633333+00:00

I checked it again, and when input has more than 4K tokens GPT-3.5-turbo version 0301 returns the answer without any errors. But when I send input with more than 16K tokens it returns Response [400]. These are properties of the model we are using:

Model name: gpt-35-turbo

Model version: 0301

Deployment type: Standard

Content Filter: Default

Tokens per Minute Rate Limit (thousands): 120

Rate limit (Tokens per minute): 120000

Rate limit (Requests per minute): 720
AshokPeddakotla-MSFT 30,511 Reputation points

2023-09-05T15:51:59.75+00:00

Tali.AI Strange. But as mentioned, this is not the intended behavior.
I will check this internally and update you if any known issues. Appreciate your time and patience.
leomeet _ 5 Reputation points

2023-11-06T11:13:25.4066667+00:00

Its strange to see this behaviour

in my app when i pass an input with context length of ~21k tokens to Azure-3.5-turbo model it gives me error of "max token limit 8049 exceeded please check your ....."

while it being a 4k context window model
and yes i checked in my callback metadata to check if the model was correct
ITSSA-ITS 0 Reputation points

2024-01-04T13:40:22.32+00:00

I have the same doubt. In my case the prompt accept 8192 tokens (8k) using gpt-35-turbo.

What's happening?

Share via

What is the exact maximum input tokens of Azure GPT-3.5-turbo?