Increase of latency in GPT4 deployments

Question

Increase of latency in GPT4 deployments

Rubén Fernández Isla 0

My organization has several Azure AI services which contain GPT4 deployments, located in France Central (we are in Spain)

Since late November we have experimented a large increase in latency in all of GPT4 deployments.

Before this, the time response for a 2000 tokens prompt (request + completion) was about 20 seconds at most. Now, with the same prompt (and same deployment) the average time response is about 50-60 seconds. Max tokens variable is set to 500.

We assume this has to do with the increase in users and requests (which led to the temporary cancellation of new GPT4 subscriptions).

But we would like to ask:

Is there some way we can minimize this latency problem?
How long this problem is expected to persist?

Thanks in advance

AshokPeddakotla-MSFT 35,961 Reputation points Moderator

2023-12-21T13:56:46.9633333+00:00
Rubén Fernández Isla Greetings & Welcome to Microsoft Q&A forum!

Since late November we have experimented a large increase in latency in all of GPT4 deployments. Before this, the time response for a 2000 tokens prompt (request + completion) was about 20 seconds at most. Now, with the same prompt (and same deployment) the average time response is about 50-60 seconds.

I understand that you are experiencing an increase in latency in your GPT4 deployments located in France Central. Can you also share the models details?

Is there some way we can minimize this latency problem?

There are a few things you can do to minimize latency. One option is to consider deploying your GPT4 models in a region that is closer to your users. This can help reduce the network latency between your users and the GPT4 models. Additionally, you can consider optimizing your code and data to reduce the amount of data that needs to be transferred between your users and the GPT4 models.

Please check Performance and latency to know more about Azure OpenAI service improving the latency performance.

Here are some of the best practices to lower latency:

Model latency: If model latency is important to you we recommend trying out our latest models in the GPT-3.5 Turbo model series.

Lower max tokens: OpenAI has found that even in cases where the total number of tokens generated is similar the request with the higher value set for the max token parameter will have more latency.

Lower total tokens generated: The fewer tokens generated the faster the overall response will be. Remember this is like having a for loop with n tokens = n iterations. Lower the number of tokens generated and overall response time will improve accordingly.

Streaming: Enabling streaming can be useful in managing user expectations in certain situations by allowing the user to see the model response as it is being generated rather than having to wait until the last token is ready.

Content Filtering improves safety, but it also impacts latency. Evaluate if any of your workloads would benefit from modified content filtering policies.

How long this problem is expected to persist?

I would recommend you keep an eye on the Azure status page and Azure Service Health for any updates on this issue.

I hope this helps! Let me know if you have any other questions.
Rubén Fernández Isla 0 Reputation points

2023-12-21T15:53:55.4333333+00:00

Thanks for your answer AshokPeddakotla-MSFT :)

We have already taken into account those recommendations but the problem still persists.

These are the properties of one of the gpt4 deployments we have implemented, with them maximum TPM assigned:

With that capacity, shouldn't be able to generate a much quicker response in a single request to the model (2000 tokens for request, 400 for completion)?

The average latency now is between 40 and 60 seconds...

Do you see something wrong?

Thanks in advance
Mauricio Larenas 10 Reputation points

2023-12-21T16:28:05.6266667+00:00

I have the same issue. These last days ChaGPT from Azure has been working really slow compared to consuming ChatGPT from OpenAI API.

In our case we have it configured at Sweeden Central, but I don't think that's the problem at all.

Sometimes a simple instruction takes 5 minutes to receive an answer.
AshokPeddakotla-MSFT 35,961 Reputation points Moderator

2023-12-29T03:28:32.94+00:00

Rubén Fernández Isla & Mauricio Larenas Please note that this latency issue is an expected behavior. Latency varies based on what model you are using. For an identical request, it is expected that different models will have a different latency.

As mentioned earlier, Please check Performance and latency to know more about Azure OpenAI service improving the latency performance.

If you are still facing the same issue, For a deeper investigation and immediate assistance on this issue, please file a support request @ https://aka.ms/azsupt?
Nikita Shilenok 10 Reputation points

2024-01-09T22:39:55.28+00:00

Your latency was steadily tranforming for GPT-4 from stable 4-5 seconds towards now regular spikes within 5-15+ sec. range just within past months. Do you recommend us to expect a further increase in latency upcoming months?
Aimee Peterson 0 Reputation points

2024-01-10T16:08:08.24+00:00

i realy thank for Developer to give me valueable information

Your answer

Rubén Fernández Isla 0 Reputation points

2023-12-21T15:53:55.4333333+00:00

Thanks for your answer AshokPeddakotla-MSFT :)

We have already taken into account those recommendations but the problem still persists.

These are the properties of one of the gpt4 deployments we have implemented, with them maximum TPM assigned:

With that capacity, shouldn't be able to generate a much quicker response in a single request to the model (2000 tokens for request, 400 for completion)?

The average latency now is between 40 and 60 seconds...

Do you see something wrong?

Thanks in advance
Mauricio Larenas 10 Reputation points

2023-12-21T16:28:05.6266667+00:00

I have the same issue. These last days ChaGPT from Azure has been working really slow compared to consuming ChatGPT from OpenAI API.

In our case we have it configured at Sweeden Central, but I don't think that's the problem at all.

Sometimes a simple instruction takes 5 minutes to receive an answer.
AshokPeddakotla-MSFT 35,961 Reputation points Moderator

2023-12-29T03:28:32.94+00:00

Rubén Fernández Isla & Mauricio Larenas Please note that this latency issue is an expected behavior. Latency varies based on what model you are using. For an identical request, it is expected that different models will have a different latency.

As mentioned earlier, Please check Performance and latency to know more about Azure OpenAI service improving the latency performance.

If you are still facing the same issue, For a deeper investigation and immediate assistance on this issue, please file a support request @ https://aka.ms/azsupt?
Nikita Shilenok 10 Reputation points

2024-01-09T22:39:55.28+00:00

Your latency was steadily tranforming for GPT-4 from stable 4-5 seconds towards now regular spikes within 5-15+ sec. range just within past months. Do you recommend us to expect a further increase in latency upcoming months?
Aimee Peterson 0 Reputation points

2024-01-10T16:08:08.24+00:00

i realy thank for Developer to give me valueable information

Share via

Increase of latency in GPT4 deployments

Your answer