Token Length Error with Azure's 'text-embedding-ada-002' Model

Question

I'm utilizing the Azure embedding model 'text-embedding-ada-002' to obtain vectors for my input texts. However, I've encountered an error with the message:

'This model's maximum context length is 8191 tokens, but you requested 8193 tokens (8193 in your prompt; 0 for the completion).

I am using, 'tiktoken', to determine the number of tokens in my input. However, each input has varying token counts. How can I ensure that my input doesn't exceed the 8191-token limit, and What is the character length of a single token?

tokenizer = tiktoken.get_encoding("cl100k_base")
def count_tokens(text):
    tokens = tokenizer.encode(text)
    return len(tokens)
count_tokens('input text')

Answer 1

@Mohan, Tejaswi (Fairfield, OH) When your input is longer than the limit, you will need to chunk and fetch separate encodings for each chunk. The size of the chunk is up to you and should be decided based on your scenario.

Here is some sample code on how you can chunk your input

tokenizer = tiktoken.get_encoding("cl100k_base")


def split_text(text, max_tokens):
    tokens = tokenizer.encode(text)
    chunks = []
    startIndex = 0
    while startIndex < len(tokens):
        endIndex = startIndex + max_tokens
        chunks.append(tokens[startIndex:endIndex])
        startIndex = endIndex
    return [tokenizer.decode(chunk) for chunk in chunks]


input_text = "input text"
max_tokens = 4000
chunks = split_text(input_text, max_tokens)