Azure virtual machine: multiple async calls to open AI break

ZZ 40 Reputation points
2025-01-23T09:54:49.33+00:00

I have narrowed down the problem to most likely the use of VM so I ask here.

My code uses Langchain to interface ChatGPT's function calling feature. It makes async calls - 10 in a batch - at a time. The code works perfectly when I run it locally on my PC, but gets stuck when I deploy it to run on the Azure VM.

The basic code structure looks like below.

  • When I run this kind of code locally on my PC, it works fine.
  • When I run this on my azure VM, the last line times out, suggesting that the async calls could not finish. There are no other errors/exceptions other than the cancellation error due to the timeout.

This seems to be due to the VM configuration or how async HTTP calls are handled by Azure. Can someone tell me what to look for?

class PaperInformation (BaseModel):
    author: str = Field(description="...")
    title: str = Field(description="...")
    ...

async def callopenai(chat, openai_functions, text)	
	SYSTEM_BASE = "..."
	prompt = ChatPromptTemplate.from_messages([
    	("system", SYSTEM_BASE),
	    ("user", "{text}")
	])

	try:
    	model_with_functions = chat.bind(
        	functions=openai_functions,
	        function_call={"name": PaperInformation.__name__}
	    )
    	extraction_chain = prompt | model_with_functions | JsonOutputFunctionsParser()
	    with get_openai_callback() as cb:
    	    res = await extraction_chain.ainvoke({"text": email_text.strip()})
        	costs = langchain_util.log_costs(cb)
	    return res, costs
	except:
    	print(traceback.format_exc())    
	
async def make_async_calls(papers:list)
	my_functions=convert_to_openai_function(PaperInformation)
	chat=ChatOpenAI(temperature=0, ...)
	tasks=[]
	for paper in papers:
	   tasks.append(callopenai(chat, my_functions, paper))
	res = await asyncio.gather(*tasks)
    return res


papers= .... #code to load a list of texts, assume its size=10
result=await asyncio.wait_for(make_async_calls(papers),timeout=1800)
Azure Virtual Machines
Azure Virtual Machines
An Azure service that is used to provision Windows and Linux virtual machines.
8,330 questions
{count} votes

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.