Understanding Function Calling in Azure OpenAI - How is the Response Handled?

Question

Hello everyone,

We are working on an application where we use function calling in Azure OpenAI to improve the user’s prompt before sending it to the assistant. On the backend, we have more context about what the user actually means, so we call a function that enhances the prompt by adding structure, a plan, and other relevant details.

However, we have noticed that the assistant does not seem to directly treat the function call’s response as the new prompt. This raises the question:

What happens between the assistant and the function call? Is there an intermediate prompt or processing step before the function’s response is used?

Additionally, if anyone has recommendations on how to ensure the assistant fully utilizes the function call response as the new prompt, we’d love to hear your insights!

Answer 1

When using function calling in Azure OpenAI, the process follows these steps:

1. User sends a prompt to the Azure OpenAI service.
2. The assistant determines if a function call is required based on the model’s reasoning.
3. The function call is executed (via your backend or an API integration).
4. Azure OpenAI receives the function’s response.
5. The assistant then decides how to use the function’s response in its next message.

What Happens Between the Assistant and the Function Call?

The assistant does not automatically replace the user’s prompt with the function response. Instead, the response from the function is treated as additional context, and the assistant integrates it into the conversation.

• Intermediate Processing: The model interprets the function output in the context of the conversation but does not inherently treat it as a new user prompt.
• Model Continuation: The model may ask clarifying questions or modify its response based on the function output rather than directly assuming it as a reformulated prompt.

How to Ensure the Assistant Uses the Function Call’s Response as the New Prompt?

Instead of relying on the model to process the function output implicitly, structure the response like this:

{
  "role": "assistant",
  "content": "Here is the improved prompt with structure and additional details: [Function Response]"
}

This helps the assistant treat it as a new instruction.

After the function returns a response, manually construct a new query for the assistant using both the original prompt and the function response then modify your system prompt to instruct the model to always use the function response as the basis for its next message.

Answer 2

Hi Maris,

Understanding Function Calling in Azure OpenAI:

When you use function calling in Azure OpenAI, the interaction typically follows these steps:

1. User Prompt Submission: The user enters a prompt, and it's sent to the Azure OpenAI model.
2. Model Determines Function Call:
   • If the model detects that a function is needed based on its training and system instructions, it generates a function call request instead of a regular response.
   • This function call is returned in the response JSON.
3. Function Execution (On Backend):
   • Your backend extracts the function call, executes it, and retrieves the response.
   • This function’s output is not automatically used as a new prompt. Instead, you must manually send it back to the model.
4. Sending Function Response Back to Assistant:
   • The function response is sent back to the model as a new message.
   • The assistant processes the function response and generates a final response.

Why is the Function Response Not Being Used as the New Prompt?

Reason:

Azure OpenAI does not automatically treat the function response as a new prompt. Instead, you need to explicitly pass the function response back as part of a new message to continue the conversation.

To make sure the assistant fully utilizes the function response, follow these steps:

1. Capture the Function Call and Execute It

Ensure your backend correctly extracts the function name and arguments, executes the function, and captures the response.

Example function call response:

{
    "role": "assistant",
    "content": null,
    "function_call": {
        "name": "enhance_prompt",
        "arguments": "{\"user_input\": \"Find me a good book on AI\"}"
    }
}

Your backend executes enhance_prompt() and gets:

{
    "enhanced_prompt": "Find me a good book on AI. Preferably a recent publication that covers deep learning and its applications in 2024."
}

2. Send the Function Response Back to Azure OpenAI

The function response must be sent as a new message in the chat history.

Example:

{
    "role": "function",
    "name": "enhance_prompt",
    "content": "{ \"enhanced_prompt\": \"Find me a good book on AI. Preferably a recent publication that covers deep learning and its applications in 2024.\" }"
}

3. Call Azure OpenAI Again with Updated Messages

Now, send the updated conversation history back to Azure OpenAI:

[
    {
        "role": "user",
        "content": "Find me a good book on AI"
    },
    {
        "role": "assistant",
        "content": null,
        "function_call": {
            "name": "enhance_prompt",
            "arguments": "{\"user_input\": \"Find me a good book on AI\"}"
        }
    },
    {
        "role": "function",
        "name": "enhance_prompt",
        "content": "{ \"enhanced_prompt\": \"Find me a good book on AI. Preferably a recent publication that covers deep learning and its applications in 2024.\" }"
    }
]

At this point, the assistant will process the function's response and generate a meaningful reply.

Additionally:

Modify your system prompt to explicitly instruct the assistant to use the function response as a new prompt. Example:

You are an AI assistant. Always use the function's response as the new prompt to generate a complete answer.

Hope this helps. Do let us know if you any further queries.   

------------- 

If this answers your query, do click Accept Answer and Yes for was this answer helpful.

Thank you!