OpenAI gpt-3.5-turbo streaming responds slowly

Question

OpenAI gpt-3.5-turbo streaming responds slowly

Niu Sang 40

When I request the API of Azure OpenAI gpt-3.5-turbo and set stream to true to get a streaming response, the result will be returned very slowly, and all the data will be returned suddenly after waiting for a few seconds. This is completely different from OpenAi's official API streaming return experience. Azure does not have the effect of OpenAI's official typewriter at all, and it is more like a non-streaming experience. Please how can I solve this problem? I think it may be that the nginx gateway of Azure OpenAI has not enabled SSE?

Igor Tytyk 0 Reputation points

2023-08-02T13:14:54.7666667+00:00

Are the non-streaming response any faster?

Accepted answer

0 additional answers

Your answer

Igor Tytyk 0 Reputation points

2023-08-02T13:14:54.7666667+00:00

Are the non-streaming response any faster?

Answer 1

AshokPeddakotla-MSFT 35,971 Moderator

Niu Sang

Could you share more details about the API request you're making? Specifically, what is the size of the payload you're sending?

Can you try accessing the API from a different network or device to confirm that the issue is not related to the network or device?

I suggest you, check Interacting with the model for Use the following practices for best results when chatting with the model.

Also, Consider setting the following parameters even if they are optional for using the API.

User's image

Please note that content filtering and abuse monitoring features of Azure OpenAI still apply to the data.

See similar thread which addressed the same issue here : https://learn.microsoft.com/en-us/answers/questions/1276363/problem-with-streaming-chat-api

Hope this helps. Do let us know if that helps or have any further queries,

Niu Sang 40 Reputation points

2023-07-06T05:25:09.9566667+00:00

Thank you. I have checked the case you shared with me, and I guess it is caused by content filtering. Thank you very much
AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2023-07-06T05:43:48.2+00:00

Niu Sang Glad to hear it was helpful.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Niu Sang 40 Reputation points

2023-07-06T06:13:16.44+00:00

In addition, I would like to know if there is any other faster way to remove content filtering besides submitting a form? After all, Azure is always slow to respond to forms.
AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2023-07-06T08:37:23.7033333+00:00

Niu Sang Unfortunately, there is only one way.

Approval is required for full content filtering control, including (i) configuring content filters at severity level high only (ii) or turning the content filters off. Managed customers only may apply for full content filtering control via this form: Azure OpenAI Limited Access Review: Modified Content Filters and Abuse Monitoring (microsoft.com).

Share via

OpenAI gpt-3.5-turbo streaming responds slowly

0 additional answers

Your answer