Do knowledge files in Azure AI Foundry require File Search tool (especially with GPT-5 vs GPT-4)?

Question

I’m trying to understand the correct setup for using knowledge files in Azure AI Foundry / Azure AI Agents.

We are uploading knowledge files (documents) to be used by the agent, but I’m unclear whether we must explicitly add the File Search (vector store) tool for them to be usable.

What we’ve observed:

• With GPT-4, the agent sometimes appears to work even without explicitly enabling File Search.
• With GPT-5, the same setup does not seem to reliably use the knowledge files unless the File Search tool is added.

1. Is File Search mandatory for knowledge files to be used at runtime in Foundry agents?
2. Why does GPT-4 appear to work without explicitly enabling it, while GPT-5 seems to require it?
3. Is GPT-5 more strictly dependent on tool-based retrieval rather than implicit context usage?
4. Is there any way to trace or inspect retrieval/tool calls (e.g., see whether File Search was invoked, what chunks were retrieved, or why it was not triggered)?

Answer 1

Hello @Meenakshi Kathiresan ,

Welcome to Microsoft Q&A .Thank you for reaching out to us.

Knowledge files in Azure AI Foundry are not directly accessible to an agent unless a retrieval mechanism is configured. The File Search tool (vector store) serves as this mechanism by ingesting documents, splitting them into chunks, generating embeddings, and retrieving relevant content dynamically at runtime. Without this setup, the model operates only on the prompt and its pre-trained knowledge, which means uploaded documents are not reliably used. For this reason, File Search should be considered essential for consistent and scalable knowledge grounding.

The observed difference between GPT-4 and GPT-5 behavior is primarily due to how each model handles context and retrieval. In some GPT-4-based setups, document content may appear to be used without explicitly enabling File Search. This can happen when small files are implicitly included in the prompt or when preconfigured templates already include a retrieval layer. However, this behavior is not consistent or transparent. GPT-5, by design, follows a stricter and more structured approach, where access to external data depends entirely on explicitly defined tools such as File Search.

GPT-5 separates prompt-based context from tool-based retrieval. Prompt injection involves passing document content directly into the model input, which is limited in size and not suitable for large-scale use. In contrast, File Search enables dynamic retrieval of only the most relevant document chunks at runtime. This improves reliability, ensures better traceability, and avoids hidden or implicit data usage.

To validate and troubleshoot document retrieval behavior, built-in observability features can be used.

1. Connecting Application Insights
   • Open the Foundry project
   • Navigate to Agents > Traces
   • Select Connect and link an existing or new Application Insights resource
   • If not visible: Project Details > Connected Resources > Add Connection > Application Insights
   • Ensure appropriate access e.g - Log Analytics Reader role
2. Generating an agent run
   • Execute a request using
   • Agent Playground
   • SDK or API
   • Deployed endpoint
   • Traces are generated automatically during execution
3. Inspecting execution traces
   • Go to Agents > Traces
   • Open a specific thread/run
   • Then review the
   • Input prompts
   • Model responses
   • Execution flow
   • Tool invocation events
   • Token usage
4. Validating File Search usage
   • Check trace details for
   • Tool invocation entries (File Search)
   • Related inputs and outputs
   • This confirms:
   • Whether retrieval was triggered
   • Whether the agent used external knowledge
   • If an alternate path (no retrieval) was taken

The following references might be helpful , please check them out

• How to use Azure OpenAI Assistants file search (classic) - Microsoft Foundry (classic) portal | Microsoft Learn
• How to use Azure AI Agents file search (classic) - Microsoft Foundry (classic) portal | Microsoft Learn
• Set Up Tracing for AI Agents in Microsoft Foundry - Microsoft Foundry | Microsoft Learn
• Application Insights OpenTelemetry observability overview - Azure Monitor | Microsoft Learn
• Vector stores for file search in Microsoft Foundry Agent Service - Microsoft Foundry | Microsoft Learn

Thank you

Answer 2

1. Knowledge files in Azure AI Foundry agents are backed by the File search tool and its vector store. For agents to search and use uploaded documents at runtime, the File search capability must be in place, because it is what parses, chunks, embeds, stores, and retrieves content from those files. File search is the mechanism that enables agents to “answer questions from proprietary or confidential content” and “search through documents you upload directly (PDFs, Word docs, code files)” without external infrastructure. Without this retrieval mechanism, models only see what is in the prompt and cannot dynamically query the uploaded files.
2. File search performs hybrid retrieval (keyword + semantic) over a vector store with default chunking (800-token chunks, 400-token overlap, up to 20 chunks in context) and query rewriting/reranking before injecting results into the model context. This is a tool-based retrieval pattern, not an implicit feature of GPT-4 or GPT-5 themselves. Any apparent behavior where GPT-4 “works” without explicitly enabling File search is due to how the agent configuration or portal wiring is set up (for example, a default or implicit file-search-backed knowledge base), not because GPT-4 can natively access knowledge files without a retrieval tool. The model alone does not index or search uploaded files.
3. GPT-5 is not described as having a different retrieval mechanism in the provided material; the retrieval behavior is defined by the File search tool and vector store, not by the model version. The tool is responsible for:
   • Query rewriting
   • Breaking complex queries into parallel searches
   • Hybrid search over vector stores
   • Reranking and selecting chunks to send to the model
   This means both GPT-4 and GPT-5 rely on the same tool-based retrieval pattern when using Foundry agents with knowledge files. Any stricter dependence observed with GPT-5 is an effect of configuration (for example, needing the File search tool attached) rather than a model-specific implicit retrieval capability.
4. File search uses a vector store with explicit limits and behavior:
   • Each vector store can hold up to 10,000 files.
   • Only one vector store can be attached to an agent and one to a conversation.
   • Supported file types and encodings are constrained (for example, .pdf, .docx, .json, .md, .txt, .py, etc., with UTF-8/UTF-16/ASCII for text MIME types).
   If files are outside these constraints, or if the vector store is not correctly attached, retrieval will not occur even if the files are uploaded.
5. The context provided does not describe a built-in tracing or inspection feature that exposes detailed File search tool calls (such as which chunks were retrieved, or why a tool was not triggered) for Foundry agents. It only describes the internal behavior (query rewriting, hybrid search, reranking) and the limits. Any deeper tracing (for example, logging tool invocations and retrieved chunks) would need to rely on capabilities outside the provided documentation.

In practice, to ensure knowledge files are used at runtime with both GPT-4 and GPT-5 in Azure AI Foundry agents:

• Ensure the File search tool (vector store) is configured and attached to the agent.
• Ensure files are in supported formats/encodings and within vector store limits.
• Use File search when the scenario matches: searching uploaded documents, answering from proprietary content, or processing large files without external search infrastructure.

---

References:

• File search tool for agents (python)
• File search tool for agents (csharp)
• Azure OpenAI Assistants file search tool (Preview) (classic)
• File search tool for agents (rest)