problem with streaming chat api

Question

problem with streaming chat api

中秋陆 50

I found that the output is not streaming, look like output with lines when I use chatgpt-35-turbo stream api.

I compared to origin openai streaming api that looks like right

romungi-MSFT 48,906 Reputation points Microsoft Employee Moderator

2023-05-05T12:58:16.16+00:00

@中秋陆 Could you add an example of the request? Did you pass "stream": true in the request body?

ymshenyu 75

@romungi-MSFT Here is an example in CURL.

Original OpenAI API:


curl --location 'https://api.openai.com/v1/chat/completions' \
                     --header 'Authorization: Bearer YOUR-KEY' \
                     --header 'Content-Type: application/json' \
                     --data '{
                                    "messages": [{"role":"system","content":"You are an AI assistant that helps people find information."},{"role":"user","content":"Write about yourself in 200 words."}],
                                    "max_tokens": 1000,
                                    "temperature": 0.7,
                                    "frequency_penalty": 0,
                                    "presence_penalty": 0,
                                    "top_p": 0.95,
                                    "stop": null,
									"stream": true,
									"model":"gpt-3.5-turbo"
                                  }'

Azure OpenAI API:


curl "https://RESOURCE.openai.azure.com/openai/deployments/DEPLOYMENT_ID/chat/completions?api-version=2023-03-15-preview" \
                       -H "Content-Type: application/json" \
                       -H "api-key: YOUR_KEY" \
                       -d '{
                   "messages": [{"role":"system","content":"You are an AI assistant that helps people find information."},{"role":"user","content":"Write about yourself in 200 words."}],
                   "max_tokens": 1000,
                   "temperature": 0.7,
                   "frequency_penalty": 0,
                   "presence_penalty": 0,
                   "top_p": 0.95,
                   "stop": null,
				   "stream": true
                 }'

Compared to the Azure OpenAI Service, you can see faster initial response times with the original OpenAI API.

ymshenyu 75 Reputation points

2023-05-09T14:51:22.7366667+00:00

In my experience, this is likely related to proxy buffering settings. Please refer to: https://serverfault.com/questions/801628/for-server-sent-events-sse-what-nginx-proxy-configuration-is-appropriate.
ymshenyu 75 Reputation points

2023-05-09T14:52:27.5233333+00:00

I hope we can get the same experience as the original one.
Sriram.c 20 Reputation points

2023-09-05T11:48:54.55+00:00

@ymshenyu Have we got the soultion to this problem?
FAN LI 0 Reputation points

2023-12-06T12:05:48.3+00:00

Have we got the soultion to this problem?
Mattias Stahre 0 Reputation points

2023-12-22T12:25:14.85+00:00

Any solution to this?

1 answer

Your answer

romungi-MSFT 48,906 Reputation points Microsoft Employee Moderator

2023-05-05T12:58:16.16+00:00

@中秋陆 Could you add an example of the request? Did you pass "stream": true in the request body?
ymshenyu 75 Reputation points

2023-05-09T14:44:33.75+00:00

@romungi-MSFT Here is an example in CURL.

Original OpenAI API:

curl --location 'https://api.openai.com/v1/chat/completions' \ --header 'Authorization: Bearer YOUR-KEY' \ --header 'Content-Type: application/json' \ --data '{ "messages": [{"role":"system","content":"You are an AI assistant that helps people find information."},{"role":"user","content":"Write about yourself in 200 words."}], "max_tokens": 1000, "temperature": 0.7, "frequency_penalty": 0, "presence_penalty": 0, "top_p": 0.95, "stop": null, "stream": true, "model":"gpt-3.5-turbo" }'

Azure OpenAI API:

curl "https://RESOURCE.openai.azure.com/openai/deployments/DEPLOYMENT_ID/chat/completions?api-version=2023-03-15-preview" \ -H "Content-Type: application/json" \ -H "api-key: YOUR_KEY" \ -d '{ "messages": [{"role":"system","content":"You are an AI assistant that helps people find information."},{"role":"user","content":"Write about yourself in 200 words."}], "max_tokens": 1000, "temperature": 0.7, "frequency_penalty": 0, "presence_penalty": 0, "top_p": 0.95, "stop": null, "stream": true }'

Compared to the Azure OpenAI Service, you can see faster initial response times with the original OpenAI API.
ymshenyu 75 Reputation points

2023-05-09T14:51:22.7366667+00:00

In my experience, this is likely related to proxy buffering settings. Please refer to: https://serverfault.com/questions/801628/for-server-sent-events-sse-what-nginx-proxy-configuration-is-appropriate.
ymshenyu 75 Reputation points

2023-05-09T14:52:27.5233333+00:00

I hope we can get the same experience as the original one.
Sriram.c 20 Reputation points

2023-09-05T11:48:54.55+00:00

@ymshenyu Have we got the soultion to this problem?
FAN LI 0 Reputation points

2023-12-06T12:05:48.3+00:00

Have we got the soultion to this problem?
Mattias Stahre 0 Reputation points

2023-12-22T12:25:14.85+00:00

Any solution to this?

Answer 1

ymshenyu 75

We figured out that the content filter may cause this issue. Maybe you could apply to disable it, but I hope there will be an option to do this in the future.

云天黎 5 Reputation points

2023-05-23T04:00:18.4433333+00:00

It is indeed possible that this issue is caused by this problem. Can I disable content filter ？.
建唐 5 Reputation points

2023-05-31T16:11:06.0366667+00:00

Here is the link to apply disable content filter: https://customervoice.microsoft.com/Pages/ResponsePage.aspx?id=v4j5cvGGr0GRqy180BHbR7en2Ais5pxKtso_Pz4b1_xURE01NDY1OUhBRzQ3MkQxMUhZSE1ZUlJKTiQlQCN0PWcu

Share via

problem with streaming chat api

1 answer

Your answer