Azure OpenAI Streamed Responses Are Missing Characters or Words
I am trying to get a very simple streamed response using my Azure OpenAI deployment. Unfortunately, one out of every 3 or so responses are missing characters in their response, or outright missing entire words. For example, I will send the following:
(Also I apologize, for marking this as YAML, couldn't change it for whatever reason)
const response = await openai.azureChat({
messages: [
{
role: "system", content: `You are a helpful assistant. Respond in less than 25 words. Use the context to help:
#### Product 1 ####
Sample product 1 : $43,232.13
####
#### Product 2 ####
Sample product 2 : $38,329.34
####
#### Product 3 ####
Sample product 3 : $39,295.23
####
` },
{ role: "user", content: "Hello" },
{ role: "assistant", content: "Hi, how can I help?" },
{ role: "user", content: "Tell me about the pricing of your products" },
],
max_tokens: 150,
temperature: 1.2,
stream: true,
})
(openai.azureChat just calls https://${region}.openai.azure.com/openai/deployments/${deployment}/chat/completions?api-version=${apiVersion}
(i've tried both 2023-07-01-preview
and 2023-08-01-preview
api versions) and returns a stream using Axios)
The response I get from gpt is:
"""
We have three products available, here are their prices:
-
Product 1 is priced at $43,.13
-
Product 2 is priced at $38,329.
-
Product 3 is priced at $39,295.23
How can I assist you further?
"""
Notice how product 1 is $43,.13 instead of the expected $43,232.13, and product 2 is $38,329. instead of $38,329.34
I even printed each chunk from the stream, manually combed through them, and its clearly missing some completion tokens.
Note that if I send the exact same body to the Open AI API (not azure), I get the expected response every time with no completion tokens missing.
Is anyone else facing the same issue? This seems to happen with both 3.5 and 4, but only happens on Azure. I don't think I have run into this issue when stream is false.