Azure Open AI vs Open AI API performance

Question

I'm using the Azure OpenAI SDK for .NET, and there's an endpoint in my app that communicates with Azure OpenAI. The model is GPT-4o (Omni), and I am experiencing two issues:

1. The response from the model is significantly slower when using Azure OpenAI, taking twice the time compared to the OpenAI direct API.
2. Despite the API requests being well below the token limit and the request per minute usage, the requests seem to queue up, causing subsequent requests to become even slower.

When I use the same .NET SDK and switch to the OpenAI direct API (not azure but api.openai.com), the responses are much faster, typically taking 6-7 seconds. In contrast, responses from Azure OpenAI take 15-18 seconds, and I don't encounter the second issue at all.

Here is a picture of the Azure app insights, where the arrows in red indicate Azure OpenAI, and the green indicates OpenAI direct.

If Microsoft is aware and plan to fix it , it's good, otherwise though mentioning it .

Answer 1

Azure OpenAI Service and OpenAI service have different response times that depend on several factors such as server load, and network latency. Also in Azure OpenAI, you have the option for a pay-as-you-go model or provisioned throughput Units options. You can improve Azure OpenAI performance by

1. Make sure you deploy the OpenAI service to a region near your location
2. You may use provisioned throughput to achieve minimal latency

Refer: https://azure.microsoft.com/en-us/pricing/details/cognitive-services/openai-service/

https://learn.microsoft.com/en-us/azure/ai-services/openai/overview

Hope this helps