Azure OpenAI Assistants file search tool (Preview)
File Search augments the Assistant with knowledge from outside its model, such as proprietary product information or documents provided by your users. OpenAI automatically parses and chunks your documents, creates and stores the embeddings, and use both vector and keyword search to retrieve relevant content to answer user queries.
Note
File search is currently not billed.
Note
- File search can ingest up to 10,000 files per assistant - 500 times more than before. It is fast, supports parallel queries through multi-threaded searches, and features enhanced reranking and query rewriting.
- Vector store is a new object in the API. Once a file is added to a vector store, it's automatically parsed, chunked, and embedded, made ready to be searched. Vector stores can be used across assistants and threads, simplifying file management and billing.
- We've added support for the
tool_choice
parameter which can be used to force the use of a specific tool (like file search, code interpreter, or a function) in a particular run.
File search support
Supported regions
File search is available in regions that support Assistants.
API Version
- 2024-05-01-preview
Supported file types
Note
For text/ MIME types, the encoding must be either utf-8, utf-16, or ASCII.
File format | MIME Type |
---|---|
.c | text/x-c |
.cs | text/x-csharp |
.cpp | text/x-c++ |
.doc | application/msword |
.docx | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
.html | text/html |
.java | text/x-java |
.json | application/json |
.md | text/markdown |
application/pdf | |
.php | text/x-php |
.pptx | application/vnd.openxmlformats-officedocument.presentationml.presentation |
.py | text/x-python |
.py | text/x-script.python |
.rb | text/x-ruby |
.tex | text/x-tex |
.txt | text/plain |
.css | text/css |
.js | text/javascript |
.sh | application/x-sh |
.ts | application/typescript |
Enable file search
from openai import AzureOpenAI
client = AzureOpenAI(
api_key=os.getenv("AZURE_OPENAI_API_KEY"),
api_version="2024-05-01-preview",
azure_endpoint = os.getenv("AZURE_OPENAI_ENDPOINT")
)
assistant = client.beta.assistants.create(
name="Financial Analyst Assistant",
instructions="You are an expert financial analyst. Use your knowledge base to answer questions about audited financial statements.",
model="gpt-4-turbo",
tools=[{"type": "file_search"}],
)
Upload files for file search
To access your files, the file search tool uses the vector store object. Upload your files and create a vector store to contain them. Once the vector store is created, you should poll its status until all files are out of the in_progress
state to ensure that all content has finished processing. The SDK provides helpers for uploading and polling.
from openai import AzureOpenAI
client = AzureOpenAI(
api_key=os.getenv("AZURE_OPENAI_API_KEY"),
api_version="2024-05-01-preview",
azure_endpoint = os.getenv("AZURE_OPENAI_ENDPOINT")
)
# Create a vector store called "Financial Statements"
vector_store = client.beta.vector_stores.create(name="Financial Statements")
# Ready the files for upload to OpenAI
file_paths = ["mydirectory/myfile1.pdf", "mydirectory/myfile2.txt"]
file_streams = [open(path, "rb") for path in file_paths]
# Use the upload and poll SDK helper to upload the files, add them to the vector store,
# and poll the status of the file batch for completion.
file_batch = client.beta.vector_stores.file_batches.upload_and_poll(
vector_store_id=vector_store.id, files=file_streams
)
# You can print the status and the file counts of the batch to see the result of this operation.
print(file_batch.status)
print(file_batch.file_counts)
Update the assistant to use the new vector store
To make the files accessible to your assistant, update the assistant’s tool_resources
with the new vector_store
ID.
assistant = client.beta.assistants.update(
assistant_id=assistant.id,
tool_resources={"file_search": {"vector_store_ids": [vector_store.id]}},
)
Create a thread
You can also attach files as Message attachments on your thread. Doing so will create another vector_store
associated with the thread, or, if there is already a vector store attached to this thread, attach the new files to the existing thread vector store. When you create a Run on this thread, the file search tool will query both the vector_store
from your assistant and the vector_store
on the thread.
# Upload the user provided file to OpenAI
message_file = client.files.create(
file=open("mydirectory/myfile.pdf", "rb"), purpose="assistants"
)
# Create a thread and attach the file to the message
thread = client.beta.threads.create(
messages=[
{
"role": "user",
"content": "How many company shares were outstanding last quarter?",
# Attach the new file to the message.
"attachments": [
{ "file_id": message_file.id, "tools": [{"type": "file_search"}] }
],
}
]
)
# The thread now has a vector store with that file in its tool resources.
print(thread.tool_resources.file_search)
Vector stores are created using message attachments that have a default expiration policy of seven days after they were last active (defined as the last time the vector store was part of a run). This default exists to help you manage your vector storage costs. You can override these expiration policies at any time.
Create a run and check the output
Create a Run and observe that the model uses the file search tool to provide a response to the user’s question.
from typing_extensions import override
from openai import AssistantEventHandler, OpenAI
client = OpenAI()
class EventHandler(AssistantEventHandler):
@override
def on_text_created(self, text) -> None:
print(f"\nassistant > ", end="", flush=True)
@override
def on_tool_call_created(self, tool_call):
print(f"\nassistant > {tool_call.type}\n", flush=True)
@override
def on_message_done(self, message) -> None:
# print a citation to the file searched
message_content = message.content[0].text
annotations = message_content.annotations
citations = []
for index, annotation in enumerate(annotations):
message_content.value = message_content.value.replace(
annotation.text, f"[{index}]"
)
if file_citation := getattr(annotation, "file_citation", None):
cited_file = client.files.retrieve(file_citation.file_id)
citations.append(f"[{index}] {cited_file.filename}")
print(message_content.value)
print("\n".join(citations))
# Then, we use the stream SDK helper
# with the EventHandler class to create the Run
# and stream the response.
with client.beta.threads.runs.stream(
thread_id=thread.id,
assistant_id=assistant.id,
instructions="Please address the user as Jane Doe. The user has a premium account.",
event_handler=EventHandler(),
) as stream:
stream.until_done()
How it works
The file search tool implements several retrieval best practices out of the box to help you extract the right data from your files and augment the model’s responses. The file_search tool:
- Rewrites user queries to optimize them for search.
- Breaks down complex user queries into multiple searches it can run in parallel.
- Runs both keyword and semantic searches across both assistant and thread vector stores.
- Reranks search results to pick the most relevant ones before generating the final response.
- By default, the file search tool uses the following settings:
- Chunk size: 800 tokens
- Chunk overlap: 400 tokens
- Embedding model: text-embedding-3-large at 256 dimensions
- Maximum number of chunks added to context: 20
Vector stores
Vector store objects give the file search tool the ability to search your files. Adding a file to a vector store automatically parses, chunks, embeds and stores the file in a vector database that's capable of both keyword and semantic search. Each vector store can hold up to 10,000 files. Vector stores can be attached to both Assistants and Threads. Currently you can attach at most one vector store to an assistant and at most one vector store to a thread.
Creating vector stores and adding files
You can create a vector store and add files to it in a single API call:
vector_store = client.beta.vector_stores.create(
name="Product Documentation",
file_ids=['file_1', 'file_2', 'file_3', 'file_4', 'file_5']
)
Adding files to vector stores is an async operation. To ensure the operation is complete, we recommend that you use the 'create and poll' helpers in our official SDKs. If you're not using the SDKs, you can retrieve the vector_store
object and monitor its file_counts
property to see the result of the file ingestion operation.
Files can also be added to a vector store after it's created by creating vector store files.
file = client.beta.vector_stores.files.create_and_poll(
vector_store_id="vs_abc123",
file_id="file-abc123"
)
Alternatively, you can add several files to a vector store by creating batches of up to 500 files.
batch = client.beta.vector_stores.file_batches.create_and_poll(
vector_store_id="vs_abc123",
file_ids=['file_1', 'file_2', 'file_3', 'file_4', 'file_5']
)
Similarly, these files can be removed from a vector store by either:
- Deleting the vector store file object or,
- By deleting the underlying file object (which removes the file it from all vector_store and code_interpreter configurations across all assistants and threads in your organization)
The maximum file size is 512 MB. Each file should contain no more than 5,000,000 tokens per file (computed automatically when you attach a file).
Attaching vector stores
You can attach vector stores to your Assistant or Thread using the tool_resources parameter.
assistant = client.beta.assistants.create(
instructions="You are a helpful product support assistant and you answer questions based on the files provided to you.",
model="gpt-4-turbo",
tools=[{"type": "file_search"}],
tool_resources={
"file_search": {
"vector_store_ids": ["vs_1"]
}
}
)
thread = client.beta.threads.create(
messages=[ { "role": "user", "content": "How do I cancel my subscription?"} ],
tool_resources={
"file_search": {
"vector_store_ids": ["vs_2"]
}
}
)
You can also attach a vector store to Threads or Assistants after they're created by updating them with the right tool_resources
.
Ensuring vector store readiness before creating runs
We highly recommend that you ensure all files in a vector_store are fully processed before you create a run. This will ensure that all the data in your vector store is searchable. You can check for vector store readiness by using the polling helpers in the SDKs, or by manually polling the vector_store
object to ensure the status is completed.
As a fallback, there is a 60-second maximum wait in the Run object when the thread's vector store contains files that are still being processed. This is to ensure that any files your users upload in a thread a fully searchable before the run proceeds. This fallback wait does not apply to the assistant's vector store.
Managing costs with expiration policies
The file_search
tool uses the vector_stores
object as its resource and you will be billed based on the size of the vector_store objects created. The size of the vector store object is the sum of all the parsed chunks from your files and their corresponding embeddings.
In order to help you manage the costs associated with these vector_store objects, we have added support for expiration policies in the vector_store
object. You can set these policies when creating or updating the vector_store
object.
vector_store = client.beta.vector_stores.create_and_poll(
name="Product Documentation",
file_ids=['file_1', 'file_2', 'file_3', 'file_4', 'file_5'],
expires_after={
"anchor": "last_active_at",
"days": 7
}
)
Thread vector stores have default expiration policies
Vector stores created using thread helpers (like tool_resources.file_search.vector_stores
in Threads or message.attachments
in Messages) have a default expiration policy of seven days after they were last active (defined as the last time the vector store was part of a run).
When a vector store expires, runs on that thread will fail. To fix this, you can recreate a new vector_store with the same files and reattach it to the thread.
all_files = list(client.beta.vector_stores.files.list("vs_expired"))
vector_store = client.beta.vector_stores.create(name="rag-store")
client.beta.threads.update(
"thread_abc123",
tool_resources={"file_search": {"vector_store_ids": [vector_store.id]}},
)
for file_batch in chunked(all_files, 100):
client.beta.vector_stores.file_batches.create_and_poll(
vector_store_id=vector_store.id, file_ids=[file.id for file in file_batch]
)
Feedback
https://aka.ms/ContentUserFeedback.
În curând: Pe parcursul anului 2024, vom elimina treptat Probleme legate de GitHub ca mecanism de feedback pentru conținut și îl vom înlocui cu un nou sistem de feedback. Pentru mai multe informații, consultați:Trimiteți și vizualizați feedback pentru