Azure OpenAI service GPT-4-turbo slower than GPT-4

Question

Azure OpenAI service GPT-4-turbo slower than GPT-4

Jonas 15

About one month ago, GPT-4-turbo was a little faster than GPT-4.

On friday, GPT-4-turbo was a lot slower than GPT-4, so I needed to go back to GPT-4 to prevent timeouts on complex questions.

Today, the difference is smaller, but GPT-4 still answers faster than GPT-4-turbo. Same experience when using the azure playground without own implementation.

Does the release of GPT-4o causes all the GPT-4-turbo resources go to GPT-4o causing GPT-4-turbo to become slow?

Due to legal reasons I need a hosting inside Europe, so I can't use GPT-4o at this moment yet. GPT-4 costs more and feels like going a step back at a moment our solution was almost ready to go to production. It is now hosted in Sweden Central region.

Anyone else that has the same experience? Is there a way to solve this?

santoshkc 15,330 Reputation points Microsoft External Staff Moderator

2024-06-03T14:28:35.12+00:00

Hi @Jonas,

Thank you for reaching out to us regarding your experience with GPT-4-turbo and GPT-4 on Azure.

To clarify, GPT-4-turbo and GPT-4 are both language models, GPT-4 is the predecessor to GPT-4 Turbo, and both models have a base model name of gpt-4.

There could be several reasons why you are experiencing long latencies, including the model, the number of tokens in the prompt, the number of tokens generated, and the overall load on the deployment and system. You can find more information on Performance and latency documentation.

Regarding your concern about the release of GPT-4o affecting the performance of GPT-4-turbo, we can confirm that this should not be the case.

We understand that you need a hosting solution inside Europe due to legal reasons and cannot use GPT-4o at the moment. We also understand your concern about the cost and potential setback of switching to GPT-4. To help you with this, you can check the available regions of GPT-4: Standard deployment model availability and you can choose the closest available region to minimize latency and improve performance.

I hope this information helps. Thank you.
Jonas 15 Reputation points

2024-06-03T15:29:48.4933333+00:00

Hi, is there a page where you can see the load on your systems for a specific region? Is it possible to automatic change region based on system load?
Daniel Nicusor Naicu 0 Reputation points

2024-06-03T16:33:07.33+00:00

@santoshkc here you can find a table of data about duration and token changes for 3 different tests:

As you can see we currently have 4 different operation in the same Routine and you can find the duration of each execution including prompt tokens total tokens and completion tokens.

Currently we use a normal Azure open ai Resource deployed on Sweden central(Not PTU) and the service that call the resource is hosted on west europe
santoshkc 15,330 Reputation points Microsoft External Staff Moderator

2024-06-04T06:42:08.2266667+00:00

Hi @Jonas,

Thank you for your follow-up question.

Azure provides several tools to monitor and manage the performance of your resources. One such tool is Azure Monitor, which allows you to collect and analyze telemetry data from your resources and applications. You can use Azure Monitor to monitor the performance of your language models and identify any issues that may be affecting their performance.
For more info: Azure Monitor overview.

Regarding your question about automatically changing regions based on system load, Azure provides several tools for load balancing and scaling your resources. For example, you can use Azure Traffic Manager to distribute traffic across multiple regions based on performance and availability. You can also use Azure Autoscale to automatically scale your resources up or down based on demand.

See: Overview of autoscale.

I hope this information helps. Thank you.
santoshkc 15,330 Reputation points Microsoft External Staff Moderator

2024-06-05T05:09:18.2066667+00:00

Hi @Jonas,

Following up to see if the above response was helpful. Thank you.
Jonas 15 Reputation points

2024-06-05T14:15:18.71+00:00

Hi @santoshkc

I looked at azure traffic manager and tried to add an endpoint, at the azure app list, I can't see the OpenAI service. And to add it as an external url as an alternative, you can only use FQDN / IP and custom header settings. It says not to put api keys in the custom headers. So there is also no way to configure it with a deployment name and api credentials. Is it possible and if so, how?

To give an example about the performance, gpt-4 answers a short question in 4 seconds, gpt-4-turbo takes 16 seconds.
santoshkc 15,330 Reputation points Microsoft External Staff Moderator

2024-06-05T15:28:43.41+00:00
Hi @Jonas,

Sorry for the trouble you are facing. I recommend reporting this issue to the Azure support team. They will be able to investigate the issue further and provide a more targeted solution. You can report the issue by following these steps:

Go to the Azure portal and navigate to your Speech Services resource.

Click on the "Support + troubleshooting" tab.

Fill out the required information, including a detailed description of the issue and any steps you have taken to troubleshoot it.

Submit the support request.

The Azure Speech Services support team will review your request and provide assistance as soon as possible. Thank you.
Jonas 15 Reputation points

2024-06-06T08:28:31.44+00:00

I see that it requires a paid support subscription to contact the azure support team. My company will first need to subscribe to it.
If someone else has already asked a similar question, answers are welcome :)

2 answers

Your answer

santoshkc 15,330 Reputation points Microsoft External Staff Moderator

2024-06-03T14:28:35.12+00:00

Hi @Jonas,

Thank you for reaching out to us regarding your experience with GPT-4-turbo and GPT-4 on Azure.

To clarify, GPT-4-turbo and GPT-4 are both language models, GPT-4 is the predecessor to GPT-4 Turbo, and both models have a base model name of gpt-4.

There could be several reasons why you are experiencing long latencies, including the model, the number of tokens in the prompt, the number of tokens generated, and the overall load on the deployment and system. You can find more information on Performance and latency documentation.

Regarding your concern about the release of GPT-4o affecting the performance of GPT-4-turbo, we can confirm that this should not be the case.

We understand that you need a hosting solution inside Europe due to legal reasons and cannot use GPT-4o at the moment. We also understand your concern about the cost and potential setback of switching to GPT-4. To help you with this, you can check the available regions of GPT-4: Standard deployment model availability and you can choose the closest available region to minimize latency and improve performance.

I hope this information helps. Thank you.
Jonas 15 Reputation points

2024-06-03T15:29:48.4933333+00:00

Hi, is there a page where you can see the load on your systems for a specific region? Is it possible to automatic change region based on system load?
Daniel Nicusor Naicu 0 Reputation points

2024-06-03T16:33:07.33+00:00

@santoshkc here you can find a table of data about duration and token changes for 3 different tests:

As you can see we currently have 4 different operation in the same Routine and you can find the duration of each execution including prompt tokens total tokens and completion tokens.

Currently we use a normal Azure open ai Resource deployed on Sweden central(Not PTU) and the service that call the resource is hosted on west europe
santoshkc 15,330 Reputation points Microsoft External Staff Moderator

2024-06-04T06:42:08.2266667+00:00

Hi @Jonas,

Thank you for your follow-up question.

Azure provides several tools to monitor and manage the performance of your resources. One such tool is Azure Monitor, which allows you to collect and analyze telemetry data from your resources and applications. You can use Azure Monitor to monitor the performance of your language models and identify any issues that may be affecting their performance.
For more info: Azure Monitor overview.

Regarding your question about automatically changing regions based on system load, Azure provides several tools for load balancing and scaling your resources. For example, you can use Azure Traffic Manager to distribute traffic across multiple regions based on performance and availability. You can also use Azure Autoscale to automatically scale your resources up or down based on demand.

See: Overview of autoscale.

I hope this information helps. Thank you.
santoshkc 15,330 Reputation points Microsoft External Staff Moderator

2024-06-05T05:09:18.2066667+00:00

Hi @Jonas,

Following up to see if the above response was helpful. Thank you.
Jonas 15 Reputation points

2024-06-05T14:15:18.71+00:00

Hi @santoshkc

I looked at azure traffic manager and tried to add an endpoint, at the azure app list, I can't see the OpenAI service. And to add it as an external url as an alternative, you can only use FQDN / IP and custom header settings. It says not to put api keys in the custom headers. So there is also no way to configure it with a deployment name and api credentials. Is it possible and if so, how?

To give an example about the performance, gpt-4 answers a short question in 4 seconds, gpt-4-turbo takes 16 seconds.
santoshkc 15,330 Reputation points Microsoft External Staff Moderator

2024-06-05T15:28:43.41+00:00

Hi @Jonas,

Sorry for the trouble you are facing. I recommend reporting this issue to the Azure support team. They will be able to investigate the issue further and provide a more targeted solution. You can report the issue by following these steps:

Go to the Azure portal and navigate to your Speech Services resource.

Click on the "Support + troubleshooting" tab.

Fill out the required information, including a detailed description of the issue and any steps you have taken to troubleshoot it.

Submit the support request.

The Azure Speech Services support team will review your request and provide assistance as soon as possible. Thank you.
Jonas 15 Reputation points

2024-06-06T08:28:31.44+00:00

I see that it requires a paid support subscription to contact the azure support team. My company will first need to subscribe to it.
If someone else has already asked a similar question, answers are welcome :)

Answer 1

Daniel Nicusor Naicu 0

Hi We Have the same problem on the same region

Answer 2

Jonas 15

I received an answer from Azure support:

Unfortunately, the observed behavior is currently expected due to Capacity constraints in the deployable regions due to the high demand in OpenAI. While we are unable to provide a specific timeline for improvement, our dedicated product team is actively engaged in refining the service.

Share via

Azure OpenAI service GPT-4-turbo slower than GPT-4

2 answers

Your answer