AzureOPEN AI Latency increased to 2 min

Question

My azure openai deployed has increase latency of 2 min for single request why is that so how to fix it ?

Answer 1

High latency (around 2 minutes per request) in Azure OpenAI is usually caused by one of the following:

Common Reasons:

• Large prompt size or high max_tokens value

Using GPT-4 instead of faster models like GPT-4o or GPT-35-turbo

Low TPM (Tokens Per Minute) quota causing throttling

Regional capacity issues (requests getting queued)

Network distance between your app and Azure region

What You Can Do:

Reduce max_tokens to the minimum required

Enable streaming responses to improve perceived speed

Check Azure Metrics for latency and throttled requests

Increase TPM quota or scale your deployment

Test with GPT-4o or GPT-35-turbo if performance is critical

If possible, please share your model name, region, and token usage so the exact bottleneck can be identified.High latency (around 2 minutes per request) in Azure OpenAI is usually caused by one of the following:

Common Reasons:

Large prompt size or high max_tokens value

Using GPT-4 instead of faster models like GPT-4o or GPT-35-turbo

Low TPM (Tokens Per Minute) quota causing throttling

Regional capacity issues (requests getting queued)

Network distance between your app and Azure region

What You Can Do:

Reduce max_tokens to the minimum required

Enable streaming responses to improve perceived speed

Check Azure Metrics for latency and throttled requests

Increase TPM quota or scale your deployment

Test with GPT-4o or GPT-35-turbo if performance is critical

If possible, please share your model name, region, and token usage so the exact bottleneck can be identified.

Answer 2

Hi prasanna kumar,

You’re not alone this kind of ~2‑minute latency has been reported by multiple users on Azure OpenAI recently, and it’s usually not caused by a single factor.

Based on community experience, the most common contributors are:

• Model choice: Some models (especially heavier or reasoning-focused ones) have noticeably higher response times compared to faster options like GPT‑4o or GPT‑35‑Turbo.
• Token limits and prompt size: Large prompts or high max_tokens values can significantly increase generation time.
• TPM throttling: Even when deployments show higher limits, effective Tokens‑Per‑Minute can be lower, causing requests to queue.
• Regional capacity constraints: Certain regions intermittently experience higher load, which results in delayed responses rather than outright failures.

Things that typically help narrow this down:

• Check Azure Metrics for throttling and latency at the deployment level.
• Temporarily test with a smaller prompt and lower max_tokens to see if latency improves.
• Compare behavior by switching to a different model or region, if possible.
• Enable streaming responses to improve perceived latency, especially for longer outputs.

I Hope this helps. Do let me know if you have any further queries.
Thankyou!