How to fix regular Timeout errors when using chatGPT3.5 and Azure Functions?

Question

Hi,

I have a Python V2 Function App that interacts with Azure OpenAI's chatGPT3.5 Turbo. To ensure that the max tokens aren't exceeded, I use tiktoken to count the tokens and segment the prompts as required. In the last week I have been getting many Timeout errors and would appreciate help on how to avoid them or at least where they are coming from.

What I have tried so far:

Retry package from PyPi

@retry(Exception, tries=2, delay=1, backoff=2, logger=logging)
def function_that_calls_chatgpt():
    ....

This gives the following error message before the retry (request id removed)

The operation was timeout. { "error": { "code": "Timeout", "message": "The operation was timeout." } } 408 {'error': {'code': 'Timeout', 'message': 'The operation was timeout.'}} {'Content-Length': '75', 'Content-Type': 'application/json', 'apim-request-id': 'xxxxxx', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'x-content-type-options': 'nosniff', 'x-ms-region': 'East US', 'Date': 'Mon, 27 Mar 2023 16:13:00 GMT'}, retrying in 1 seconds...

Same message after the retry:

error_code=Timeout error_message='The operation was timeout.' error_param=None error_type=None message='OpenAI API error received' stream_error=False

Follow up questions:

1. How can I get further information about the source of this error?
2. Could the timeout be due to rate limiting and I am not receiving the correct error message?

Thanks in advance!

Answer 1

Liza King The current quotas and limits could be causing your operations to timeout but the message seen might not be indicating the same. Enabling diagnostic logging and checking the metrics of your resource is an option that could help check more details of your requests. Please see this page from documentation on available metrics and options to enable logging.

If you are seeing these errors consistently then you probably could also apply for a quota increase request through this form. Since this takes some time to review and approve the request you might as well apply and try checking the metrics and diagnostic logs for more clues.

The final option would obviously be to raise a support case to lookup your request ids if none of the above helps.

P.S: There have been some issues with operation timeout errors yesterday which have been mitigated now. If you have seen any errors yesterday then the service issue could possibly be one of the causes.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.