Azure virtual machine: multiple async calls to open AI break
ZZ
40
Reputation points
I have narrowed down the problem to most likely the use of VM so I ask here.
My code uses Langchain to interface ChatGPT's function calling feature. It makes async calls - 10 in a batch - at a time. The code works perfectly when I run it locally on my PC, but gets stuck when I deploy it to run on the Azure VM.
The basic code structure looks like below.
- When I run this kind of code locally on my PC, it works fine.
- When I run this on my azure VM, the last line times out, suggesting that the async calls could not finish. There are no other errors/exceptions other than the cancellation error due to the timeout.
This seems to be due to the VM configuration or how async HTTP calls are handled by Azure. Can someone tell me what to look for?
class PaperInformation (BaseModel):
author: str = Field(description="...")
title: str = Field(description="...")
...
async def callopenai(chat, openai_functions, text)
SYSTEM_BASE = "..."
prompt = ChatPromptTemplate.from_messages([
("system", SYSTEM_BASE),
("user", "{text}")
])
try:
model_with_functions = chat.bind(
functions=openai_functions,
function_call={"name": PaperInformation.__name__}
)
extraction_chain = prompt | model_with_functions | JsonOutputFunctionsParser()
with get_openai_callback() as cb:
res = await extraction_chain.ainvoke({"text": email_text.strip()})
costs = langchain_util.log_costs(cb)
return res, costs
except:
print(traceback.format_exc())
async def make_async_calls(papers:list)
my_functions=convert_to_openai_function(PaperInformation)
chat=ChatOpenAI(temperature=0, ...)
tasks=[]
for paper in papers:
tasks.append(callopenai(chat, my_functions, paper))
res = await asyncio.gather(*tasks)
return res
papers= .... #code to load a list of texts, assume its size=10
result=await asyncio.wait_for(make_async_calls(papers),timeout=1800)
Sign in to answer