api call hitting api request time out error in azure app service

Abeera Rauf 10

On my Azure account, I'm using the OpenAI GPT-4 API within an application built with Azure App Service. I’m encountering an issue where the API returns a timeout error after every 2 to 4 messages. However, when I use the same API calls in an Azure ML Studio notebook without App Service, the calls work fine without timeouts. Could you help me identify the root cause of this? Is it related to API latency with the latest GPT-4 model, or is there something specific to App Service impacting the API response times?

Abeera Rauf 10 Reputation points

2024-11-03T09:33:15.94+00:00

in addition, is azure web app a reliable framework? #microsoft
Avinash Devarakonda 345 Reputation points Microsoft Vendor

2024-11-04T06:42:32.75+00:00
Hi Abeera Rauf,

Welcome to Microsoft Q&A Forum, thank you for posting your query here!

Azure App Service has a default timeout of 230 seconds. If your API calls take longer, then the request will timeout.

The GPT-4 API can have variable response times. Since it works fine in Azure ML Studio, the issue is likely with App Service.

Below are some recommendations:

If possible, optimize your API calls to reduce the response time. This might involve simplifying the prompts or reducing the number of tokens requested.

If the issue persists, you might consider hosting your API calls in a different environment, such as Azure Functions or Azure Kubernetes Service (AKS), which might offer more flexibility in handling long-running requests.

Yes, Azure App Service is reliable for hosting web apps, APIs, and mobile backends. It offers autoscaling, load balancing, and automated management, ensuring reliability.

If you need more help or have other questions, feel free to ask!

Thank You!!
Avinash Devarakonda 345 Reputation points Microsoft Vendor

2024-11-05T00:56:05.4666667+00:00

Hi Abeera Rauf,
Following up to see if the given response was helpful.

Thank You!!
Abeera Rauf 10 Reputation points

2024-11-05T05:29:16.55+00:00

@Avinash Devarakonda thanks for your response. I tried reducing max token to 50-200 length but api call sent me empty response or most of the time it is still giving api request timeout error on gpt 4 api call. I am using flask api on web app service , inside which i am calling openai endpoint or azure openai endpoint. What can be the issue? Can u help?
Avinash Devarakonda 345 Reputation points Microsoft Vendor

2024-11-05T05:56:32.1333333+00:00

Hi Abeera Rauf,

Try increasing the timeout settings for both your Flask app and OpenAI API requests, use asynchronous requests to handle multiple calls efficiently, check and possibly scale up your Azure App Service resources, review logs for errors or resource constraints, optimize your network configuration.

Thank You!!
Avinash Devarakonda 345 Reputation points Microsoft Vendor

2024-11-06T01:23:54.3166667+00:00

Hi Abeera Rauf,

We haven’t heard from you on the last response and was just checking back to see if the give response was helpful.

Thank You.
Abeera Rauf 10 Reputation points

2024-11-07T06:15:41.0833333+00:00

Hi @Avinash Devarakonda ,

I tried everything you stated but nothing works. What else we can do?

My application is real time application so i cannot afford increasing timeout setting.

I use max_tokens=200 to 500 in api call, it started giving response but randomly oftenly api sending me blank responses.

What can i do?
Avinash Devarakonda 345 Reputation points Microsoft Vendor

2024-11-07T09:01:31.44+00:00
Hi Abeera Rauf,

If you’ve tried all the suggested steps and are still encountering issues with blank responses when using max_tokens=200 to 500, check for rate limits, adjust request parameters like "temperature" or "top_p" can help improve response consistency.

For example:

{ "model": "gpt-4-turbo", "prompt": "Your prompt here", "max_tokens": 500, "temperature": 0.7, "top_p": 0.9 }

Ensure that your requests are not being throttled due to rate limits. Refer to the Azure OpenAI Service quotas and limits

If you still face issue, monitor API logs for errors and contact Azure support with detailed logs and your request configuration.

The Azure support team will review your request and provide assistance as soon as possible Azure support.

Thank You.
Abeera Rauf 10 Reputation points

2024-11-08T07:02:22.8033333+00:00

Hi @Avinash Devarakonda

We tried your way issue still exist...... we are on global deployment with ample tokens limit and max 3 people are using it... Hardly 15000 tokens per 10 messages used and still we have this issue

what can be done?

Share via

api call hitting api request time out error in azure app service

Your answer