Phi4 multimodal tools and function calling

Question

Phi4 multimodal tools and function calling

Razak Zaha 20

Does Phi4 multimodal support tools and function calling?

Accepted answer

2 additional answers

Your answer

Answer 1

Marcin Policht 50,495 MVP Volunteer Moderator

As per

https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/models-featured#microsoft

Phi-4-multimodal-instruct chat-completion (with image and audio content) - Input: text, images, and audio (131,072 tokens)
- Output: (4,096 tokens)
- Tool calling: No
- Response formats: Text

If the above response helps answer your question, remember to "Accept Answer" so that others in the community facing similar issues can easily find the solution. Your contribution is highly appreciated.

hth

Marcin

Razak Zaha 20 Reputation points

2025-03-22T23:27:53.38+00:00

yeah, that's why... okay sure thank you.

Might need the learn portal update its learning capabilities then..
Razak Zaha 20 Reputation points

2025-03-23T05:36:09.1066667+00:00
I recently switched from using GPT-4o to Phi-4-multimodel-instruct in my Next.js application using Azure AI services, but I'm encountering the following error:

BadRequestError: 400 {"object":"error","message":""auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set","type":"BadRequestError","param":null,"code":400}

The error occurs when calling the runTools() method, which was working perfectly with GPT-4o. Here's my implementation:

OpenAI Instance Configuration:

export const OpenAIInstance = () => {

try {

if ( !process.env.AZURE_SERVICE_PHI_4_MULTIMODEL_API_KEY || !process.env.AZURE_SERVICE_PHI_4_MULTIMODEL_API_VERSION || !process.env.AZURE_SERVICE_PHI_4_MULTIMODEL_INSTANCE_NAME ) { throw new Error( "Missing required environment variables for OpenAI instance." ); } const azureOpenAI = new AzureOpenAI({ apiKey: process.env.AZURE_SERVICE_PHI_4_MULTIMODEL_API_KEY, apiVersion: process.env.AZURE_SERVICE_PHI_4_MULTIMODEL_API_VERSION, baseURL: `https://${process.env.AZURE_SERVICE_PHI_4_MULTIMODEL_INSTANCE_NAME}.openai.azure.com/models/chat/completions?api-version=${process.env.AZURE_SERVICE_PHI_4_MULTIMODEL_API_VERSION}` }); return azureOpenAI;

} catch (error) {

console.error( "Error initializing OpenAI instance:", (error as Error).message ); throw error;

}

};

Chat API Extension Implementation:

export const ChatApiExtensions = async (props: {

chatThread: ChatThreadModel;

userMessage: string;

history: ChatCompletionMessageParam[];

extensions: RunnableToolFunction<any>[];

signal: AbortSignal;

}): Promise<ChatCompletionStreamingRunner> => {

const { userMessage, history, signal, chatThread, extensions } = props;

const openAI = OpenAIInstance();

const model = process.env.AZURE_SERVICE_PHI_4_MULTIMODEL_MODEL_NAME;

if (!model) {

throw new Error("Model deployment name is not configured");

}

const systemMessage = await extensionsSystemMessage(chatThread);

try {

return await openAI.beta.chat.completions.runTools( { model: model, stream: true, messages: [ { role: "system", content: chatThread.personaMessage + "\n" + systemMessage, }, ...history, { role: "user", content: userMessage, }, ], tools: extensions, temperature: 0.7, max_tokens: 4000, }, { signal: signal, } );

} catch (error) {

console.error("Error in ChatApiExtensions:", error); throw error;

}

};

Based on the error message, it seems Phi-4-multimodel-instruct requires additional parameters for tool usage that weren't needed with GPT-4o. I've researched the Azure documentation but haven't found specifics about these flags (--enable-auto-tool-choice and --tool-call-parser).

Has anyone successfully used tools with Phi-4-multimodel-instruct on Azure? How can I modify my code to make this work?

Environment:

Next.js (server components)

Azure OpenAI service

OpenAI Node.js SDK
SriLakshmi C 6,250 Reputation points Microsoft External Staff Moderator

2025-03-24T08:58:39.49+00:00

Hi Razak Zaha,

Disclaimer: This response contains a reference to a third-party World Wide Web site. Microsoft is providing this information as a convenience to you. Microsoft does not control these sites and has not tested any software or information found on these sites; therefore, Microsoft cannot make any representations regarding the quality, safety, or suitability of any software or information found there. There are inherent dangers in the use of any software found on the Internet, and Microsoft cautions you to make sure that you completely understand the risk before retrieving any software from the Internet.

Please refer this Troubleshooting Azure AI Phi-4-Multimodel-Instruct: Handling Tool Calling.
Razak Zaha 20 Reputation points

2025-03-24T09:17:23.91+00:00

I got your point but why in the azure foundry mention it is supported?
SriLakshmi C 6,250 Reputation points Microsoft External Staff Moderator

2025-03-24T09:41:38.1233333+00:00

@Razak Zaha

The Azure AI Foundry mentions support for Phi-4 multimodal tools and function calling as part of its capabilities. The Phi-4 model is designed for chat-completion tasks and can handle inputs that include text, images, and audio, allowing for a more versatile interaction. Function calling is a feature that enables structured responses from large language models (LLMs) by allowing users to define functions in the API call, which the model can then use to generate JSON objects that can be parsed and executed in code.

This integration enhances the functionality of applications built on Azure, making it easier to manage and utilize complex data types and interactions.
SriLakshmi C 6,250 Reputation points Microsoft External Staff Moderator

2025-03-25T09:30:39.6133333+00:00

@Razak Zaha,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Razak Zaha 20 Reputation points

2025-03-25T09:54:31.62+00:00

the Microsoft support is basically aware about the issue and this is what they explained:

Greetings of the day…! Thank you for your time over the call. Sorry for the inconvenience. According to the documentation, currently the tool calling is not supported for the Phi-4-multimodel.

https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/models-featured#microsoft

Even though the UI says that tool calling is supported, we are advised to follow the documentation. We have also conveyed this information to the internal team to make necessary changes on the UI. Please let us if you have further queries regarding this issue. If you don't have any other queries, could you please confirm the closure of this support request. Looking forward to your response.

Have a great day ahead…! Regards,

| Support Engineer | Azure Cognitive Services

Customer Services and Support – Professional

Answer 2

Razak Zaha 20

Hi, but in this URL: https://techcommunity.microsoft.com/blog/educatordeveloperblog/welcome-to-the-new-phi-4-models---microsoft-phi-4-mini--phi-4-multimodal/4386037

and here: https://github.com/microsoft/PhiCookBook/tree/main/md/02.Application/07.FunctionCalling/Phi4/FunctionCallingBasic

looks like contradict. confusing

Answer 3

Indeed - the information at https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/models-featured#microsoft appears to be incorrect.

Here is additional info that confirms function calling support

https://ai.azure.com/explore/models/Phi-4-multimodal-instruct/version/1/registry/azureml?tid=e7b5993f-5428-477c-8197-7c95209a48aa

If the above response helps answer your question, remember to "Accept Answer" so that others in the community facing similar issues can easily find the solution. Your contribution is highly appreciated.

hth

Marcin

Share via

Phi4 multimodal tools and function calling

2 additional answers

Your answer