Azure assistant API no response

Angeline 0

Hi community - we are building a AI agent on FB messenger platform, using the Azure Assistant API. We are running into a problem that the API has no response when user sends a new message after being away for a while.

The behavior is that if a session is successfully activated, users can chat with the agent with no issue. However, if the user stops chatting, and return to the conversation, new request will often result in no response from the assistant API.

Didn't find anyone talking about the same problem online...can you please help?

navba-MSFT 23,000 Reputation points Microsoft Employee

2024-07-29T04:34:15.7633333+00:00
@Angeline Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

.

Plan 1:
Please check if you encounter the same issue with Azure OpenAI Studio Chat playground while the activity is kept idle.

.

.

Plan 2:
Please update the code to implement a polling mechanism to repeatedly check the status of the assistant’s response. Here’s a breakdown of what it does:

Polling Interval: The setTimeout function is used to wait for 500 milliseconds before making another request to check the status.

Status Check: The getRun method is called to fetch the current status of the assistant’s response.

Loop Condition: The loop continues as long as the status is either “queued” or “in_progress”.

You need to add the below logic:

Timeout Handling: You can add a timeout mechanism to break the loop if it takes too long for the assistant to respond. This can prevent the loop from running indefinitely.

Exponential Backoff: Instead of a fixed interval, you could implement an exponential backoff strategy to gradually increase the wait time between checks, reducing the load on the server.

Here’s an example with a timeout and exponential backoff:

// Run the thread let runResponse = await assistantsClient.createRun( assistantThread.id, { assistantId: assistantResponse.id } ); console.log(`Run created: ${JSON.stringify(runResponse)}`); // Wait for the assistant to respond const maxWaitTime = 60000; // Maximum wait time in milliseconds const startTime = Date.now(); let waitTime = 500; do { await new Promise((r) => setTimeout(r, waitTime)); runResponse = await assistantsClient.getRun(assistantThread.id, runResponse.id); // Increase wait time for exponential backoff waitTime = Math.min(waitTime * 2, 5000); // Cap the wait time at 5 seconds // Check if the maximum wait time has been exceeded if (Date.now() - startTime > maxWaitTime) { throw new Error("Timeout: Assistant response took too long."); } } while (runResponse.status === "queued" || runResponse.status === "in_progress");

.

.

Hope this helps. If you have any follow-up questions, please let me know. I would be happy to help.
Angeline 0 Reputation points

2024-07-31T15:25:57.5566667+00:00

Hi @navba-MSFT thank you so much for your response! We tried both ways, with plan 1 I never encountered this issue (does the Azure Play Ground uses the same assistant API?); plan 2 was implemented, but we still encounter the same of no/extended response time from the API.
navba-MSFT 23,000 Reputation points Microsoft Employee

2024-08-01T03:14:57.1766667+00:00
@Angeline Thanks for getting back.

The Absolute maximum duration that 1 session can remain active after "Create Thread" is called is

1 hour. If there is no activity on the thread, it gets timed out after 30 minutes of inactivity. Regardless of what you do, a session cannot go beyond 60 minutes.

.

.

After the "Create Thread" is called every session is killed when either:

There is no activity on it (no run) for 30 minutes (idle timeout) OR

60 minutes have passed since the time "Create Thread" was called.

.

Hope this answers.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Share via

Azure assistant API no response

Your answer