GPT-4o via Azure OpenAI timing out constantly

Matthew Hertz (London) 25

Hi,

GPT-4o via Azure API seems almost trivially broken in a way that really surprises me. Basically when streaming is enabled, once the number of input tokens exceeds around 15K, it'll time out before producing a single token. The timeout cannot be increased.

Basically means GPT-4o, a model that should support 128k only supports around 10k. Maybe not even that!

I'm happy to share my full code used for testing this but it's not very fancy at all and does only what you'd expect it to. This is when testing US WEST and US EAST 2. Both exhibit the same issue.

Has anyone else seen this?

I'd go as far as to suggest that GPT-4o is completely broken and mostly unusable to be honest - am I missing something or is this actually genuinely the case?

Gleb Krivosheev 0 Reputation points

2024-06-18T19:33:48.0066667+00:00

US East2 has improved, but Sweden-central (EU's only location for this model) is unbearably slow (>100s to respond) vs 5-10s for the same request for the same model in USEAST2
Leon H 0 Reputation points

2025-01-08T22:02:01.0933333+00:00

It is January 2025 and gpt-4o models are still not working at all in 'eastus'.
Oh, Microsoft, why are you wasting our time? Back to direct OpenAI calls....

1 answer

VasaviLankipalle-MSFT 18,386 Reputation points

2024-06-06T16:52:26+00:00

Hello @Matthew Hertz (London) , Thanks for using Microsoft Q&A Platform.

Sorry for the inconveniences that has caused. We have seen a similar issue related to Latency or Timeout issues for GPT-4o model.

If you are facing similar issue, then please note that the product team is already aware of this and working on the fix for all the regions. Currently the ETA is June 12th.

I hope this helps.

Regards,

Vasavi

-Please kindly accept the answer and vote 'yes' if you feel helpful to support the community, thanks.
Please sign in to rate this answer.

2 people found this answer helpful.
Matthew Hertz (London) 25 Reputation points

2024-06-07T08:11:48.8066667+00:00

Thanks.

For reference, this already seems much improved in US WEST. Previously my test was timing out when generating the first token after ~10k tokens of input. That reliably takes no longer than 5s now.

VasaviLankipalle-MSFT 18,386 Reputation points

2024-06-07T17:09:41.86+00:00

Hello @Matthew Hertz (London) , glad to know that.

Please remember to "Accept Answer" if any answer/reply helped, so that others in the community facing similar issues can easily find the solution.

Johan Yman 5 Reputation points

2024-06-18T07:29:18.6733333+00:00

We still have this problem for Sweden-central. Do you know when it will be fixed there?

Matthew Hertz (London) 25 Reputation points

2024-06-18T07:36:04.9133333+00:00

US WEST is OK.

US East 2 is still broken.

Matthew Hertz (London) 25 Reputation points

2024-06-18T14:15:49.88+00:00

Now US WEST is broken...

Matthew Hertz (London) 25 Reputation points

2024-06-18T14:17:18.5366667+00:00

Hi @VasaviLankipalle-MSFT

It's June 18th - US West, East and East 2 are still borderline broken for the flagship GenAI model.

Matthew Hertz (London) 25 Reputation points

2024-06-18T14:46:47.98+00:00

Hi @VasaviLankipalle-MSFT

I've an update on this - we were previously using a custom content filter to reduce the false-positive rate of safe inputs being rejected (e.g., asking about financial futures contracts was considered "violent"!)

Using a custom content filter seemingly massively increases this latency. Reverting to "default" has reduced the latency back to normal.

Is there a way to use a custom content filter without incurring such a large latency cost?

Gleb Krivosheev 0 Reputation points

2024-06-18T19:33:00.61+00:00

Please fix Central Sweden! This is the only European deployment of this model and it is terribly slow. In my test it takes on the order of 100-300 seconds to complete a simple prompt of 5k tokens. Same request takes 5-10 seconds in USEAST2 now. This is with content filtering disabled.

Leon H 0 Reputation points

2025-01-08T21:55:39.46+00:00

Yes, all gpt-4o models exhibit the same timeout behavior. I tried it with Target URI which is different for every model as well and getting either timeout exception or resource not found exception.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

GPT-4o via Azure OpenAI timing out constantly

1 answer

Your answer