Azure OpenAI Service gpt-35-turbo model returns over 4096 tokens.

Question

Azure OpenAI Service gpt-35-turbo model returns over 4096 tokens.

Yuta Kuroda 10

I am using the gpt-35-turbo model with Azure OpenAI Service. The documentation(1) states that the max request tokens are 4096, but it seems to be returning more than 4096 tokens. Can anyone explain why this is happening?

(1) https://learn.microsoft.com/en-us/azure/cognitive-services/openai/concepts/models

the api response is below. The finish_reason is "stop". I expect it is "length".
I delete the id and the content.

{
  "id": "{it's deleted}",
  "object": "chat.completion",
  "created": 1682590906,
  "model": "gpt-35-turbo",
  "usage": {
    "prompt_tokens": 4378,
    "completion_tokens": 2359,
    "total_tokens": 6737
  },
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "{it's so long... deleted}"
      },
      "finish_reason": "stop",
      "index": 0
    }
  ]
}

Ramr-msft 17,826 Reputation points

2023-04-27T12:24:54.11+00:00
@Yuta Kuroda Thanks for the question, In this notebook, There are two strategies for managing the token limit below 4096 when working with the ChatGPT model.

Option 1: Keep the conversation within a given token limit

Option 2: Keep the conversation within a given number of turns
Yuta Kuroda 10 Reputation points

2023-05-01T00:29:47.4033333+00:00

@Ramr-msft Thank you for your response. However, it seems that my concern has not been addressed. In the example mentioned above, the total_tokens are shown to be 6737, which exceeds the limit of 4096, and that is the point of my inquiry.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Ramr-msft 17,826 Reputation points

2023-05-02T12:58:30.6666667+00:00

@Yuta Kuroda Thanks for the details. I am checking with the team on this internally, will update on the same.
Xing Yuan Chen 5 Reputation points

2023-05-03T01:34:15.4533333+00:00

Same here. Discovered that I could use more than even 6000 tokens in multiple turns of conversation as a request, and could still get a complete answer.
Philip Chung 65 Reputation points

2023-05-06T21:18:19.2433333+00:00
@Ramr-msft I am using GPT-3.5.-turbo and have the same question. I am also getting total tokens > 4000. The "finish_reason" is "stop", and I don't have truncation of the output completion tokens/text returned by the model.

I want to know whether earlier parts of my prompt or system message is getting discarded in this situation since the total tokens exceeds the max token limit. If so, I would reduce the length of my prompt; otherwise, I would not make any modifications.

Below is a redacted version of the chat completion object I get back as a response.

<OpenAIObject chat.completion id=redacted> JSON: { "choices": [ { "finish_reason": "stop", "index": 0, "message": { "content": "redacted content", "role": "assistant" } } ], "created": 1000000000, "id": "redacted", "model": "gpt-35-turbo", "object": "chat.completion", "usage": { "completion_tokens": 516, "prompt_tokens": 4071, "total_tokens": 4587 } }
ImranTech 0 Reputation points

2023-05-09T15:30:31.1433333+00:00

100% agree with this statement: "I want to know whether earlier parts of my prompt or system message is getting discarded in this situation since the total tokens exceeds the max token limit. If so, I would reduce the length of my prompt" Please let us know if earlier tokes beyond 4096 are being discarded.
Deip Kumar 15 Reputation points

2023-05-09T21:27:30.9033333+00:00
Running into the same thing - to demonstrate I tried a large prompt with max_tokens = 6000 and I get the following:

InvalidRequestError: This model's maximum context length is 8192 tokens. However, you requested 10819 tokens (4819 in the messages, 6000 in the completion). Please reduce the length of the messages or completion.

Is this even running gpt-3.5-turbo like it should be? It's also possible this is going to gpt4 which would also be bad

Deip Kumar 15

I think I confirmed that it is actually running gpt-3.5-turbo based on the difference in output, so I'm mainly concerned that it is cutting out part of the prompt. Although when I ask the model to return the first and last sentence of a (> 4096 token) prompt I give it, it is successfully able to do so:

import time

start_prompt = "You are a mathematician. "

middle_prompt = (
    "This is a random sentence repeated a lot. "
) * 800

end_prompt = "The poem is about a mathematician."

import tiktoken
encoding = tiktoken.encoding_for_model("gpt-3.5-turbo")

print(len(encoding.encode(start_prompt)))
print(len(encoding.encode(middle_prompt)))
print(len(encoding.encode(end_prompt)))

tokens = len(encoding.encode(start_prompt)) + len(encoding.encode(middle_prompt)) + len(encoding.encode(end_prompt))

start_time = time.time()

response = openai.ChatCompletion.create(
    engine="gpt35",
    messages=[
        {"role": "system", "content": start_prompt + middle_prompt + end_prompt },
        {"role": "user", "content": "Repeat back ONLY the first and last sentence of your current system prompt." },
    ],
    temperature=0,
    max_tokens=(8192 - tokens - 100)
)

# calculate the time it took to receive the response
response_time = time.time() - start_time

# print the time delay
print(f"Full response received {response_time:.2f} seconds after request")
print(f"Full response received:\n{response}")

7
7201
8
Full response received 1.02 seconds after request
Full response received:
{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "You are a mathematician. The poem is about a mathematician.",
        "role": "assistant"
      }
    }
  ],
  "created": 1683669923,
  "id": "chatcmpl-7EPkRbRaSS1zEWF2umCIrXlWIhJM0",
  "model": "gpt-35-turbo",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 14,
    "prompt_tokens": 7241,
    "total_tokens": 7255
  }
}

Deip Kumar 15 Reputation points

2023-05-11T00:21:48.24+00:00

@Ramr-msft any updates on this?

1 answer

Your answer

Ramr-msft 17,826 Reputation points

2023-04-27T12:24:54.11+00:00

@Yuta Kuroda Thanks for the question, In this notebook, There are two strategies for managing the token limit below 4096 when working with the ChatGPT model.

Option 1: Keep the conversation within a given token limit

Option 2: Keep the conversation within a given number of turns
Yuta Kuroda 10 Reputation points

2023-05-01T00:29:47.4033333+00:00

@Ramr-msft Thank you for your response. However, it seems that my concern has not been addressed. In the example mentioned above, the total_tokens are shown to be 6737, which exceeds the limit of 4096, and that is the point of my inquiry.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Ramr-msft 17,826 Reputation points

2023-05-02T12:58:30.6666667+00:00

@Yuta Kuroda Thanks for the details. I am checking with the team on this internally, will update on the same.
Xing Yuan Chen 5 Reputation points

2023-05-03T01:34:15.4533333+00:00

Same here. Discovered that I could use more than even 6000 tokens in multiple turns of conversation as a request, and could still get a complete answer.
Philip Chung 65 Reputation points

2023-05-06T21:18:19.2433333+00:00

@Ramr-msft I am using GPT-3.5.-turbo and have the same question. I am also getting total tokens > 4000. The "finish_reason" is "stop", and I don't have truncation of the output completion tokens/text returned by the model.

I want to know whether earlier parts of my prompt or system message is getting discarded in this situation since the total tokens exceeds the max token limit. If so, I would reduce the length of my prompt; otherwise, I would not make any modifications.

Below is a redacted version of the chat completion object I get back as a response.

<OpenAIObject chat.completion id=redacted> JSON: { "choices": [ { "finish_reason": "stop", "index": 0, "message": { "content": "redacted content", "role": "assistant" } } ], "created": 1000000000, "id": "redacted", "model": "gpt-35-turbo", "object": "chat.completion", "usage": { "completion_tokens": 516, "prompt_tokens": 4071, "total_tokens": 4587 } }
ImranTech 0 Reputation points

2023-05-09T15:30:31.1433333+00:00

100% agree with this statement: "I want to know whether earlier parts of my prompt or system message is getting discarded in this situation since the total tokens exceeds the max token limit. If so, I would reduce the length of my prompt" Please let us know if earlier tokes beyond 4096 are being discarded.
Deip Kumar 15 Reputation points

2023-05-09T21:27:30.9033333+00:00

Running into the same thing - to demonstrate I tried a large prompt with max_tokens = 6000 and I get the following:

InvalidRequestError: This model's maximum context length is 8192 tokens. However, you requested 10819 tokens (4819 in the messages, 6000 in the completion). Please reduce the length of the messages or completion.

Is this even running gpt-3.5-turbo like it should be? It's also possible this is going to gpt4 which would also be bad
Deip Kumar 15 Reputation points

2023-05-09T22:05:48.6833333+00:00

I think I confirmed that it is actually running gpt-3.5-turbo based on the difference in output, so I'm mainly concerned that it is cutting out part of the prompt. Although when I ask the model to return the first and last sentence of a (> 4096 token) prompt I give it, it is successfully able to do so:

import time start_prompt = "You are a mathematician. " middle_prompt = ( "This is a random sentence repeated a lot. " ) * 800 end_prompt = "The poem is about a mathematician." import tiktoken encoding = tiktoken.encoding_for_model("gpt-3.5-turbo") print(len(encoding.encode(start_prompt))) print(len(encoding.encode(middle_prompt))) print(len(encoding.encode(end_prompt))) tokens = len(encoding.encode(start_prompt)) + len(encoding.encode(middle_prompt)) + len(encoding.encode(end_prompt)) start_time = time.time() response = openai.ChatCompletion.create( engine="gpt35", messages=[ {"role": "system", "content": start_prompt + middle_prompt + end_prompt }, {"role": "user", "content": "Repeat back ONLY the first and last sentence of your current system prompt." }, ], temperature=0, max_tokens=(8192 - tokens - 100) ) # calculate the time it took to receive the response response_time = time.time() - start_time # print the time delay print(f"Full response received {response_time:.2f} seconds after request") print(f"Full response received:\n{response}")

7 7201 8 Full response received 1.02 seconds after request Full response received: { "choices": [ { "finish_reason": "stop", "index": 0, "message": { "content": "You are a mathematician. The poem is about a mathematician.", "role": "assistant" } } ], "created": 1683669923, "id": "chatcmpl-7EPkRbRaSS1zEWF2umCIrXlWIhJM0", "model": "gpt-35-turbo", "object": "chat.completion", "usage": { "completion_tokens": 14, "prompt_tokens": 7241, "total_tokens": 7255 } }
Deip Kumar 15 Reputation points

2023-05-11T00:21:48.24+00:00

@Ramr-msft any updates on this?

Answer 1

John Sanders 176 Microsoft Employee

We always recommend staying within the documented token limit
While we don't intend to change the behavior of version 0301 of the model, all future versions will only support 4k tokens.

Share via

Azure OpenAI Service gpt-35-turbo model returns over 4096 tokens.

1 answer

Your answer