Can one make multiple concurrent calls to the same Azure OpenAI Endpoint?

Question

Can one make multiple concurrent calls to the same Azure OpenAI Endpoint?

Stawsh Murawski 5

Can one make multiple concurrent calls to the same Azure OpenAI Endpoint? We have a two web apps that use Azure OpenAI, make API calls from Python. This concurrent call would happen when a second user of our application takes an action for which our app calls Azure OpenAI, or an OpenAI API, while the call from the first instance has not yet returned. This second user might be running in the same AppServer instance, or another parallel instance, or an instance of our second application.

Stawsh Murawski 5 Reputation points

2024-10-15T21:55:48.3833333+00:00

From the PoV of the Python program, only the initial client=() call is a call to the end point. After that all the calls are to that client, e.g. to client.beta.threads.create() or client.beta.threads.runs.create_and_poll(). Can I make multiple concurrent client.xxxx() calls?
VasaviLankipalle-MSFT 18,676 Reputation points Moderator

2024-10-16T16:50:59.3733333+00:00

Hello @Stawsh Murawski , your use-case looks similar to Multi-Agent Systems functionality. With the Azure Assistant API, you can create multi-agent systems that are persistent and infinitely long, and append messages as users interact, all while the Assistant accesses files and tools in parallel, including a Code Interpreter and function calling capabilities.

Here is the documentation you can refer to: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios/Assistants/multi-agent
VasaviLankipalle-MSFT 18,676 Reputation points Moderator

2024-10-17T14:57:28.7166667+00:00

Hello @Stawsh Murawski , did you get a chance to check my response?
Daniel Fang 1,060 Reputation points MVP

2024-10-21T10:45:45.9633333+00:00

HI. Each restful API call to Azure OpenAI endpoint are stateless, you can have multiple clients calling the endpoint at the same time as long as the total calls does not exceed rate limit.

"Can one make multiple concurrent calls to the same Azure OpenAI Endpoint?" -> yes

"We have a two web apps that use Azure OpenAI, make API calls from Python. This concurrent call would happen when a second user of our application takes an action for which our app calls Azure OpenAI, or an OpenAI API, while the call from the first instance has not yet returned. This second user might be running in the same AppServer instance, or another parallel instance, or an instance of our second application." -> while the first user's call(s) are in progress, the second user's call(s) are separate from Azure OpenAI point of view. No matter the second call is from another app service, another instance or another thread as long as your python program is not single threaded (i.e. blocking the second user to initiate the call).

1 answer

Your answer

Stawsh Murawski 5 Reputation points

2024-10-15T21:55:48.3833333+00:00

From the PoV of the Python program, only the initial client=() call is a call to the end point. After that all the calls are to that client, e.g. to client.beta.threads.create() or client.beta.threads.runs.create_and_poll(). Can I make multiple concurrent client.xxxx() calls?
VasaviLankipalle-MSFT 18,676 Reputation points Moderator

2024-10-16T16:50:59.3733333+00:00

Hello @Stawsh Murawski , your use-case looks similar to Multi-Agent Systems functionality. With the Azure Assistant API, you can create multi-agent systems that are persistent and infinitely long, and append messages as users interact, all while the Assistant accesses files and tools in parallel, including a Code Interpreter and function calling capabilities.

Here is the documentation you can refer to: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios/Assistants/multi-agent
VasaviLankipalle-MSFT 18,676 Reputation points Moderator

2024-10-17T14:57:28.7166667+00:00

Hello @Stawsh Murawski , did you get a chance to check my response?
Daniel Fang 1,060 Reputation points MVP

2024-10-21T10:45:45.9633333+00:00

HI. Each restful API call to Azure OpenAI endpoint are stateless, you can have multiple clients calling the endpoint at the same time as long as the total calls does not exceed rate limit.

"Can one make multiple concurrent calls to the same Azure OpenAI Endpoint?" -> yes

"We have a two web apps that use Azure OpenAI, make API calls from Python. This concurrent call would happen when a second user of our application takes an action for which our app calls Azure OpenAI, or an OpenAI API, while the call from the first instance has not yet returned. This second user might be running in the same AppServer instance, or another parallel instance, or an instance of our second application." -> while the first user's call(s) are in progress, the second user's call(s) are separate from Azure OpenAI point of view. No matter the second call is from another app service, another instance or another thread as long as your python program is not single threaded (i.e. blocking the second user to initiate the call).

Answer 1

Hello @Stawsh Murawski , Thanks for using Microsoft Q&A Platform.

Yes, you can make the concurrent calls to the Azure OpenAI Endpoint. However, the maximum number of concurrent executions is going to be limited by capacity (Tokens-per-Minute (TPM).

Please note that there are rate limits imposed by Azure OpenAI. These limits define the maximum number of requests you can make within a certain time frame. Exceeding these limits might result in throttling or errors.

You can refer to the quota limits here: https://learn.microsoft.com/en-us/azure/ai-services/openai/quotas-limits

Your use-case looks similar to Multi-Agent Systems functionality. With the Azure Assistant API, you can create multi-agent systems that are persistent and infinitely long, and append messages as users interact, all while the Assistant accesses files and tools in parallel, including a Code Interpreter and function calling capabilities.

Here is the documentation you can refer to: https://github.com/Azure-Samples/azureai-samples/tree/main/scenarios/Assistants/multi-agent

I hope this helps.

Regards,

Vasavi

-Please kindly accept the answer and vote 'yes' if you feel helpful to support the community, thanks.

Share via

Can one make multiple concurrent calls to the same Azure OpenAI Endpoint?

1 answer

Your answer