Azure OpenAI "There was an error generating a response." or "An error occurred."

Question

I have deployed an Azure OpenAI chat using both the models GPT-35-Turbo and GPT-4. The chat seems to be working for a while, but at a certain point, guaranteed, I get the following errors in the chat:

Error

There was an error generating a response. Chat history can't be saved at this time. If the problem persists, please contact the site administrator.
Error

An error occurred. Answers can't be saved at this time. If the problem persists, please contact the site administrator.

---

After I get this error, the chat is broken and I can no longer proceed with the conversation. As soon as I start a new chat/conversation, there are no errors until that conversation also starts to get errors.

This has been very frustrating as a history of a conversation can no longer be used and I have to start over with a chat. As well, the new chat will give errors after a certain time as well.

I have used West Europe and Sweden as the region, but both give the errors.

Please help as we are paying for Azure OpenAI but without it actually working correctly.

Answer 1

Finally found the issue and solve it, it is working perfectly fine for me now, it is an issue with answer streaming.

If the answer text streaming is not important for you, you can solve it by adding :

"SHOULD_STREAM = False "

to your app environment variables, or you can directly edit the app.py and comment the existing condition set to read from the environment variables :

AZURE_COSMOSDB_MONGO_VCORE_VECTOR_COLUMNS = os.environ.get("AZURE_COSMOSDB_MONGO_VCORE_VECTOR_COLUMNS")

SHOULD_STREAM = False 
## if AZURE_OPENAI_STREAM.lower() == "true" else False ##