Performance issue in generating the results using Open AI.

Question

I am leveraging Azure OpenAI to build the interactive chatbot for a client, by using GPT-4 model.

The token limit has already increased to 80K at the subscription level. When I am asking any questions, the model searches in different data sources, i.e set of PDF documents stored in various directories, list of tables in Database, using respective agent and comes up with the response.

However, the performance is very poor. For a simple question, the model is taking 1 minute 40 seconds to come up with answer and the client is not going to like it.

Please suggest what is the best possible solution to this.

Even if I have to raise a ticket for this, where to do that, since I am not able to see the option to create ticket in Azure Portal.

Answer 1

Sudhindra Kulkarni Greetings & Welcome to Microsoft Q&A forum!

However, the performance is very poor. For a simple question, the model is taking 1 minute 40 seconds to come up with answer and the client is not going to like it. Please suggest what is the best possible solution to this.

If you are using GPT4 model then latency is expected considering that gpt-4 has more capacity than the gpt-3.5 version.

As of now, we do not offer Service Level Agreements (SLAs) for response times from the Azure OpenAI service. .

This article talks about Azure OpenAI service about improving the latency performance.

Here are some of the best practices to lower latency:

• Model latency: If model latency is important to you we recommend trying out our latest models in the GPT-3.5 Turbo model series.
• Lower max tokens: OpenAI has found that even in cases where the total number of tokens generated is similar the request with the higher value set for the max token parameter will have more latency.
• Lower total tokens generated: The fewer tokens generated the faster the overall response will be. Remember this is like having a for loop with n tokens = n iterations. Lower the number of tokens generated and overall response time will improve accordingly.
• Streaming: Enabling streaming can be useful in managing user expectations in certain situations by allowing the user to see the model response as it is being generated rather than having to wait until the last token is ready.
• Content Filtering improves safety, but it also impacts latency. Evaluate if any of your workloads would benefit from modified content filtering policies.

Even if I have to raise a ticket for this, where to do that, since I am not able to see the option to create ticket in Azure Portal.

Please see how to Create an Azure support request or Create a support ticket directly from Azure portal.

Please let me know if you have any follow-up questions. I would be happy to answer it.

---

If the response helped, please do click Accept Answer and Yes for was this answer helpful.

Doing so would help other community members with similar issue identify the solution. I highly appreciate your contribution to the community.