Using Responses API file_search with other custom tools, file_search not prioritized?

Question

Hello everyone,

I’m running into a problem with the o4-mini model and the Responses API when I enable both the built-in file_search tool and any other custom tool:

Standalone:

• With only file_search enabled (I pass in my vector store ID), it works perfectly.
• With only my custom tools enabled, those work too.

Combined:

• As soon as I enable both file_search and another tool, file_search is never invoked, even if I explicitly tell the model, “Use the file_search tool to look up …”.
• Instead, the model cycles through all the other tools repeatedly (all function calls succeed, and are correctly passed into further message input, but loop continues).

Workaround attempt:

• I removed the native file_search and registered my own tool that calls vector_store.search.
• Now the model does invoke the custom search tool, but Azure’s vector-store search endpoint returns 404 Resource Not Found, even though the store status is “completed”. The Q&A thread on this has not been updated for over a month: https://learn.microsoft.com/en-us/answers/questions/2243926/vector-store-search-in-azure-openai-service-return

At this point it seems impossible to use the built-in file_search alongside any other tool. Has anyone else encountered this? Any ideas for a workaround or configuration change? I don't want to spin my own RAG pipeline, given the availability of vector store API, but at this point I'm worried that is the only way forward. Thanks in advance!

Answer 1

Hello @CuriousDeveloper ,

Yes, like you said i can be able to see function call is made even though the user asked for document related query.

Checking on this with our internal team for more details on this behavior.

As of now you can follow below workaround and check if that sets in your use case.

Whenever the function tool is present you make second and final chat completion api call with previous output as input with vector store tool.

Below is the sample code.

# First api call
response = client.responses.create(  
    model="o4-mini",
    tools=[ 
        {
        "type": "file_search",
        "vector_store_ids": ["vs_uUYi8hxFmT7fYncND6nKhqUS"],
        },
        {
        "type": "function",
        "name": "simple_math",
        "description": "Perform basic math operations on two numbers. Use this for any calculation requests.",
        "parameters": {
            "type": "object",
            "properties": {
                "a": {"type": "number", "description": "First number"},
                "b": {"type": "number", "description": "Second number"}, 
                "operation": {
                    "type": "string", 
                    "enum": ["add", "subtract", "multiply", "divide"], 
                    "description": "Math operation to perform"
                }
            },
            "required": ["a", "b", "operation"]
        }
    }
    ],
    input=[{"role": "user", "content": "Summarize the uploaded file"}]  
)

#your tool definition
def simple_math(a: float, b: float, operation: str) -> dict:
    """Simple stateless math function for testing"""
    ops = {"add": a + b, "multiply": a * b, "subtract": a - b, "divide": a / b if b != 0 else "Error: Division by zero"}
    return {"operation": operation, "a": a, "b": b, "result": ops.get(operation, "Invalid operation")}

#Creating inputs to second api call
input = [] 

for output in response.output:  
    if output.type == "function_call":  
        match output.name:  
            case "simple_math":  
                input.append(  
                    {  
                        "type": "function_call_output",  
                        "call_id": output.call_id,  
                        "output": f"{simple_math(**json.loads(output.arguments))}",  
                    }  
                ) 
            case _:  
                raise ValueError(f"Unknown function call: {output.name}")

#Second API call
second_response = client.responses.create(  
    model="o4-mini",  
    previous_response_id=response.id,  
    input=input,
    tools=[{
        "type": "file_search",
        "vector_store_ids": ["vs_uUYi8hxFmT7fYncND6nKhqUS"],
    }]
)  

for i in second_response.output:
    if i.type == "message":
        print(i.content[0].text)

Output:

If you have any query regarding workaround let us know in comments or in private message and will update the issue you are facing with current output once we get details from our internal team.

Thank you