Token Length Error with Azure's 'text-embedding-ada-002' Model

Mohan, Tejaswi (Fairfield, OH) 30 Reputation points

I'm utilizing the Azure embedding model 'text-embedding-ada-002' to obtain vectors for my input texts. However, I've encountered an error with the message:

'This model's maximum context length is 8191 tokens, but you requested 8193 tokens (8193 in your prompt; 0 for the completion).

I am using, 'tiktoken', to determine the number of tokens in my input. However, each input has varying token counts. How can I ensure that my input doesn't exceed the 8191-token limit, and What is the character length of a single token?

tokenizer = tiktoken.get_encoding("cl100k_base")
def count_tokens(text):
    tokens = tokenizer.encode(text)
    return len(tokens)
count_tokens('input text')
Azure OpenAI Service
Azure OpenAI Service
An Azure service that provides access to OpenAI’s GPT-3 models with enterprise capabilities.
1,435 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Pramod Valavala 19,356 Reputation points Microsoft Employee

    @Mohan, Tejaswi (Fairfield, OH) When your input is longer than the limit, you will need to chunk and fetch separate encodings for each chunk. The size of the chunk is up to you and should be decided based on your scenario.

    Here is some sample code on how you can chunk your input

    tokenizer = tiktoken.get_encoding("cl100k_base")
    def split_text(text, max_tokens):
        tokens = tokenizer.encode(text)
        chunks = []
        startIndex = 0
        while startIndex < len(tokens):
            endIndex = startIndex + max_tokens
            startIndex = endIndex
        return [tokenizer.decode(chunk) for chunk in chunks]
    input_text = "input text"
    max_tokens = 4000
    chunks = split_text(input_text, max_tokens)