Azure OpenAI GPT Turbo 3.5 Instruct Model - Max Tokens (Request)

Test Admin 176

We are using Azure OpenAI gpt-3.5-turbo-instruct (0914) model with Standard tier.

We have a query regarding the maximum tokens(context length) of gpt-3.5-turbo-instruct (0914).

As it is mentioned in the Microsoft documentation that gpt-3.5-turbo-instruct (0914) only accepts the maximum tokens - 4097 tokens
https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models#gpt-35-models

But when we sent request using Java Application and Postman is accepts maximum context length - 5000 tokens
Also in Azure OpenAI Studio(PlayGround) it accepts maximum tokens - 4000 tokens

Question:
What is actual context length (max tokens) of gpt-3.5-turbo-instruct (0914) model?

Thanks in advance.

romungi-MSFT 43,696 Reputation points Microsoft Employee

2023-10-13T14:25:45.14+00:00

@Test Admin Which region are you using in this case where you are seeing this discrepancy?
Test Admin 176 Reputation points

2023-10-16T05:25:59.4533333+00:00

@romungi-MSFT We are using East US, Sweden Central.
Test Admin 176 Reputation points

2023-10-16T05:30:11.8033333+00:00

@romungi-MSFT gpt-35-turbo-instruct (0914) only available in East US, Sweden Central
Guillermo Castro (external) 5 Reputation points

2023-10-16T21:43:54.9833333+00:00

Is there an ETA to have it available in West Europe? Thanks
romungi-MSFT 43,696 Reputation points Microsoft Employee

2023-10-17T06:17:54.6233333+00:00

@Test Admin It looks like the lesser token limit from studio is expected. This seems to be a design choice as per the AOI team.

With respect to higher limit you have observed, it is supposed to be 4097 tokens when you use the REST API. If there is any discrepancy you have pass the request details through support for the team to take a look at it.

Currently, there is no ETA for adding other regions. for this model. Thanks!!

Test Admin 176

@romungi-MSFT
Thanks for the reply,

Kindly check the below,

Dependencies used

<dependency>
	<groupId>com.azure</groupId>
	<artifactId>azure-sdk-bom</artifactId>
	<version>1.2.15</version>
	<type>pom</type>
	<scope>import</scope>
</dependency>

<dependency> 
  <groupId>com.azure</groupId> 
  <artifactId>azure-ai-openai</artifactId> 
  <version>1.0.0-beta.2</version> 
</dependency>

String azureKey = "t#######";
String deploymentOrModelId = "deploymentID"; //deploymentid created in Azure portal(gpt-35-turbo-instruct) under Azure open AI services

String endpoint = "https://wyz.openai.azure.com/";

OpenAIClient client = new OpenAIClientBuilder().endpoint(endpoint)
		.credential(new AzureKeyCredential(azureKey)).buildClient();

List<String> prompt = new ArrayList<>();
prompt.add("what is tree");

CompletionsOptions options = new CompletionsOptions(prompt);
options.setMaxTokens(800);
options.setPresencePenalty(0.0);
options.setFrequencyPenalty(0.0);
options.setTemperature(1.0);
options.setTopP(0.5);

Completions completions = client.getCompletions(deploymentOrModelId, options);

When we add 5000+ tokens in prompt it were accepted and response is received, when using library as well as REST call

Could you verify it once and share the sample snippet for using GPT 35 turbo model.

Share via

Azure OpenAI GPT Turbo 3.5 Instruct Model - Max Tokens (Request)