Why is the OpenAI service provided by Microsoft so slow?

Question

I am from China and currently using the Azure OpenAI API. My selected server region is Central Sweden, but the speed is extremely slow, to an unacceptable extent. I am not sure if this is due to the improper choice of region or because of the payment level. Is there any way to improve this situation and make it faster?

Answer 1

vedeng Greetings!

Could you please provide prompt size, max token set which you are using?

Please check the documentation Performance and latency about improving the latency performance.

Here are some of the best practices to lower latency:

• Model latency: If model latency is important to you we recommend trying out our latest models in the GPT-3.5 Turbo model series.
• Lower max tokens: OpenAI has found that even in cases where the total number of tokens generated is similar the request with the higher value set for the max token parameter will have more latency.
• Lower total tokens generated: The fewer tokens generated the faster the overall response will be. Remember this is like having a for loop with n tokens = n iterations. Lower the number of tokens generated and overall response time will improve accordingly.
• Streaming: Enabling streaming can be useful in managing user expectations in certain situations by allowing the user to see the model response as it is being generated rather than having to wait until the last token is ready.
• Content Filtering improves safety, but it also impacts latency. Evaluate if any of your workloads would benefit from modified content filtering policies.

Please let me know if you have any further queries.