Request Timeout for Azure OpenAI when Streaming!

Question

Request Timeout for Azure OpenAI when Streaming!

Anonymous

I get lot of Request timeout when streaming. It works well for sometime however suddenly for sometime it will my cloud function will give me timeout since request isn't received during time period. This was working well when I was not streaming.

Example I am talking about the function has limit of 1.5 minutes it stops after that and I get timeout error.
Screenshot 2023-12-20 at 2.41.47 PM

Below is screenshot of Non Streaming almost no errors.

Screenshot 2023-12-20 at 2.41.36 PM

Am I calling client with timeout right in python?

AzureOpenAI(
        api_key=os.getenv("AZURE_OPENAI_KEY_CANADA"),  
        api_version="2023-10-01-preview",
        azure_endpoint="myendpointurl"
        timeout=15.0, max_retries=4
    ),

I also have higher rate limit quota.

Azar 31,725 Reputation points MVP Volunteer Moderator

2023-12-20T19:45:15.7433333+00:00
Hi

The timeout parameter you've set to 15.0 seconds determines the maximum amount of time the client will wait for a response from the API. If the API call takes longer than this timeout, it will raise a timeout exception. increase this timeout duration.

AzureOpenAI( api_key=os.getenv("AZURE_OPENAI_KEY_CANADA"), api_version="2023-10-01-preview", azure_endpoint="myendpointurl", timeout=120.0, # Set a higher timeout value in seconds max_retries=4 )
Anonymous

2023-12-20T19:49:53.7066667+00:00

Hi Azar, I am aware of that I also have specified max_retries. And added more detail same timeout setup is working in non streaming. 120 second for a response doesn't make any sense for waiting that long. When average response is 3-5 seconds.
YutongTie-9091 54,031 Reputation points Moderator

2023-12-20T20:21:18.9533333+00:00

@aryac Thanks for reaching out to us, could you please provide us with the model and region so that we can take a look at it?
Anonymous

2023-12-20T20:35:03+00:00

@Anonymous Canada East and West US, I have been using both randomly but yesterday from night it was Canada East
Firstly if this might help Here is my server function screenshot (For streaming only) Vs Non Streaming

Below for non streaming (Canada East)

Here is some screenshots from Canada East Resource in Azure. I am noticing my cloud function timeout may always not reflect on azure.
Anonymous

2023-12-20T23:25:29.14+00:00

Update if you read it I have noticed in streaming timeout errors are generally start of every few hours.
YutongTie-9091 54,031 Reputation points Moderator

2023-12-20T23:49:33.6433333+00:00

@aryac Thanks for your information, do you have a support plan? I would suggest you to raise a support ticket at the meantime, I have escalated this case to product team for investigation.

If you have no support plan, I am happy to enable you a free ticket.

Regards,

Yutong

Your answer

Azar 31,725 Reputation points MVP Volunteer Moderator

2023-12-20T19:45:15.7433333+00:00

Hi

The timeout parameter you've set to 15.0 seconds determines the maximum amount of time the client will wait for a response from the API. If the API call takes longer than this timeout, it will raise a timeout exception. increase this timeout duration.

AzureOpenAI( api_key=os.getenv("AZURE_OPENAI_KEY_CANADA"), api_version="2023-10-01-preview", azure_endpoint="myendpointurl", timeout=120.0, # Set a higher timeout value in seconds max_retries=4 )
Anonymous

2023-12-20T19:49:53.7066667+00:00

Hi Azar, I am aware of that I also have specified max_retries. And added more detail same timeout setup is working in non streaming. 120 second for a response doesn't make any sense for waiting that long. When average response is 3-5 seconds.
YutongTie-9091 54,031 Reputation points Moderator

2023-12-20T20:21:18.9533333+00:00

@aryac Thanks for reaching out to us, could you please provide us with the model and region so that we can take a look at it?
Anonymous

2023-12-20T20:35:03+00:00

@Anonymous Canada East and West US, I have been using both randomly but yesterday from night it was Canada East
Firstly if this might help Here is my server function screenshot (For streaming only) Vs Non Streaming

Below for non streaming (Canada East)

Here is some screenshots from Canada East Resource in Azure. I am noticing my cloud function timeout may always not reflect on azure.
Anonymous

2023-12-20T23:25:29.14+00:00

Update if you read it I have noticed in streaming timeout errors are generally start of every few hours.
YutongTie-9091 54,031 Reputation points Moderator

2023-12-20T23:49:33.6433333+00:00

@aryac Thanks for your information, do you have a support plan? I would suggest you to raise a support ticket at the meantime, I have escalated this case to product team for investigation.

If you have no support plan, I am happy to enable you a free ticket.

Regards,

Yutong