Azure OpenAI Streamed Responses Are Missing Characters or Words

Marwan 20 Reputation points
2023-08-17T20:30:49.6166667+00:00

I am trying to get a very simple streamed response using my Azure OpenAI deployment. Unfortunately, one out of every 3 or so responses are missing characters in their response, or outright missing entire words. For example, I will send the following:

(Also I apologize, for marking this as YAML, couldn't change it for whatever reason)

const response = await openai.azureChat({
        messages: [
            {
                role: "system", content: `You are a helpful assistant. Respond in less than 25 words. Use the context to help:
        
            #### Product 1 ####
            Sample product 1 : $43,232.13
            ####

            #### Product 2 ####
            Sample product 2  : $38,329.34
            ####

            #### Product 3 ####
            Sample product 3 : $39,295.23
            ####	
            ` },
            { role: "user", content: "Hello" },
            { role: "assistant", content: "Hi, how can I help?" },
            { role: "user", content: "Tell me about the pricing of your products" },
        ],
        max_tokens: 150,
        temperature: 1.2,
        stream: true,
    })

(openai.azureChat just calls https://${region}.openai.azure.com/openai/deployments/${deployment}/chat/completions?api-version=${apiVersion} (i've tried both 2023-07-01-preview and 2023-08-01-preview api versions) and returns a stream using Axios)

The response I get from gpt is:

"""

We have three products available, here are their prices:

  • Product 1 is priced at $43,.13
  • Product 2 is priced at $38,329.
  • Product 3 is priced at $39,295.23

How can I assist you further?

"""

Notice how product 1 is $43,.13 instead of the expected $43,232.13, and product 2 is $38,329. instead of $38,329.34

I even printed each chunk from the stream, manually combed through them, and its clearly missing some completion tokens.

Note that if I send the exact same body to the Open AI API (not azure), I get the expected response every time with no completion tokens missing.

Is anyone else facing the same issue? This seems to happen with both 3.5 and 4, but only happens on Azure. I don't think I have run into this issue when stream is false.

Azure OpenAI Service
Azure OpenAI Service
An Azure service that provides access to OpenAI’s GPT-3 models with enterprise capabilities.
2,920 questions
{count} votes

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.