Azure OpenAI: Intermittent HTTP 500 Errors During "High" Concurrency

Question

Azure OpenAI: Intermittent HTTP 500 Errors During "High" Concurrency

AbigailIreland-5813 0

Problem description

I am experiencing intermittent but persistent HTTP 500 errors when using my Azure OpenAI resource with the gpt-5.1 model. These errors started around Thu, Sep 17, 2026, and occur primarily under higher concurrency levels, specifically when I have 8 or more concurrent requests. Despite the resource metrics showing only successful responses (HTTP 200), my probe requests often fail with 500 errors, and the Azure metrics do not reflect these failures - the failing requests are not reported at all. I have already implemented retries with backoff and jitter, but the failures continue, and the frequency of errors makes the process unreliable.

Environment

Service: Azure OpenAI; Resource: qadvisor-openai-dev; Region: Global Standard; Model: gpt-5.1.

What I've already tried

I have tested with minimal payloads and production-like payloads, observing failures with both, mostly when there are 8 in-flight requests. I compared probe results with Azure metrics and noted the requests which received 500 responses are not being reported at all. I have confirmed that there are no reported issues in Azure Service Health, and I am testing directly against the endpoint. Despite retries with backoff, the errors persist. Based on the missing metrics, I believe this may be related to the Azure gateway receiving my requests rather than an issue with the OpenAI service, but Azure support removed my ability to continue responding in the support case thread and left me only with the option to create a forum post here.

Current status

I am seeking assistance to identify the root cause of these 500 errors. Additionally, any suggestions on how to get in touch with an actual support human at Azure would be appreciated. I tried calling the Microsoft Customer Support numbers I could find but they all take you to the same system, which tells me their representatives will not be able to assist me with this case and directs me back to the Azure portal.

Qdeck 0 Reputation points

2026-09-21T23:32:00.22+00:00

Hi Karnam,

Thank you for looking into this for me!

I did want to add that these errors I'm experiencing are while trying to access the resource from a functionapp running on my local machine for testing purposes. I have not seen the same frequency of 500 errors occurring when hitting the live endpoint in the Azure-deployed functionapp (there may be some errors, but we have retry with backoff and jitter in the functionapp code so I wouldn't necessarily see them; the errors when running locally are frequent enough that sometimes the retries are completely exhausted, failing the whole execution). The Azure-deployed functionapp reaches the Azure OpenAI resource via private endpoint whereas my local runtime accesses it directly using a firewall rule.

Please let me know if there is any additional information I can provide or tests I can perform to help diagnose the issue.
Karnam Venkata Rajeswari 5,340 Reputation points Microsoft External Staff Moderator

2026-09-22T20:06:51.02+00:00

Hello @AbigailIreland-5813 ,

Checking in to see if the above response was helpful

Thank you
Karnam Venkata Rajeswari 5,340 Reputation points Microsoft External Staff Moderator

2026-09-23T19:07:17.0533333+00:00

Hello @AbigailIreland-5813 ,

Following up to see if you had any chance to review the above response

Thank you

1 answer

Your answer

Qdeck 0 Reputation points

2026-09-21T23:32:00.22+00:00

Hi Karnam,

Thank you for looking into this for me!

I did want to add that these errors I'm experiencing are while trying to access the resource from a functionapp running on my local machine for testing purposes. I have not seen the same frequency of 500 errors occurring when hitting the live endpoint in the Azure-deployed functionapp (there may be some errors, but we have retry with backoff and jitter in the functionapp code so I wouldn't necessarily see them; the errors when running locally are frequent enough that sometimes the retries are completely exhausted, failing the whole execution). The Azure-deployed functionapp reaches the Azure OpenAI resource via private endpoint whereas my local runtime accesses it directly using a firewall rule.

Please let me know if there is any additional information I can provide or tests I can perform to help diagnose the issue.
Karnam Venkata Rajeswari 5,340 Reputation points Microsoft External Staff Moderator

2026-09-22T20:06:51.02+00:00

Hello @AbigailIreland-5813 ,

Checking in to see if the above response was helpful

Thank you
Karnam Venkata Rajeswari 5,340 Reputation points Microsoft External Staff Moderator

2026-09-23T19:07:17.0533333+00:00

Hello @AbigailIreland-5813 ,

Following up to see if you had any chance to review the above response

Thank you

Answer 1

Hello @AbigailIreland-5813 ,

Welcome to Microsoft Q&A .Thank you for reaching out to us.

The behaviour described is consistent with failures occurring before the requests are admitted by the service, which is why the resource metrics show only successful responses while the probes record 500 errors.

Metrics capture only those requests that the service accepts and processes, so errors that never appear in telemetry usually originate along the path between the calling application and Azure. The concurrency threshold around eight in-flight requests, combined with identical behaviour across minimal and production-like payloads, points in the same direction, since a model-side failure would normally vary with payload shape rather than with the number of open connections.

Please note that

If apim-request-id or x-ms-request-id is present, the request reached the Azure OpenAI gateway and is traceable in backend telemetry.
If the header is absent, or the body is not the standard Azure OpenAI JSON error format, the 500 was generated locally by the Functions host, an SDK wrapper, a proxy, or a TLS-inspecting agent.

In a locally hosted Function App, the most common contributors are an unhandled exception, timeout or cancellation inside the function that the local runtime surfaces as HTTP 500; connection pool exhaustion caused by creating a new client instance per request; interference from a corporate proxy, VPN or security agent during simultaneous TLS handshakes; and a short client-side timeout that aborts calls as latency rises under load. Transient service-side 500 and 503 responses are documented and are addressed by retry with exponential backoff, though these would normally be visible in the metrics, so that remains a secondary possibility until the request IDs confirm otherwise.

Recommended steps, in order:

Capture apim-request-id, x-ms-request-id, full response headers, raw body and exact UTC timestamp for three to five failures. These identifiers are required for any backend trace.
Enable diagnostic settings on the resource (Audit Logs, Request and Response Logs, AllMetrics) to a Log Analytics workspace, then query the failure window.
Under Monitoring, chart Requests split by StatusCode and ModelDeploymentName at one-minute granularity, and cross-check Resource Health for the same period.
Apply the following client adjustments:
- Reuse a single HTTP or Azure OpenAI client instance across requests.
- Raise the connection limit.
- Confirm HTTP/2 is in use.
- Increase the client timeout to at least 120 seconds during testing.
Reproduce the test in three configurations for comparison: a plain script at eight concurrency from the same machine, the same script from an Azure VM, and the Function App deployed to Azure rather than run locally.
Confirm the deployment has sufficient TPM and RPM headroom, and check whether any 429 responses are being masked by retry logic.
Retain retries with backoff and jitter, and log each retry outcome to separate transient failures from structural ones.

The following references might be helpful , please check them out

Please let us know if the response was helpful

Thank you