Is it possible to query multiple Azure Cognitive Search indexes in a single request when using Azure OpenAI chat completion with a data source configuration?

Question

I'm currently working with Azure OpenAI and using Azure Cognitive Search as the data source for RAG (Retrieval-Augmented Generation). From my experience, it seems only one index can be specified at a time for similarity-based retrieval. Is there a supported way to query multiple indexes within a single chat completion request? Or any recommended workaround to achieve this?

Answer 1

Hi Noushida,

Thank you for your question!

Currently, when using Azure OpenAI's "on your data" feature (which connects Azure Cognitive Search as a data source for Retrieval-Augmented Generation or RAG), only one Azure Cognitive Search index can be referenced per data source configuration. This is a platform limitation and is documented in the official Microsoft Azure documentation.

Limitation__:__

Azure OpenAI’s integration expects a single data source index to perform semantic or keyword-based searches. There’s currently no native support to query multiple search indexes in the same request through this built-in mechanism.

Recommended Workarounds:

Although querying multiple indexes directly in one request is not supported, here are three recommended approaches you can explore depending on your architecture and needs:

Option1: Consolidate Multiple Indexes into a Single Index

If the data in your different indexes has similar structure, you can merge them into a single Cognitive Search index and add a field like sourceType to distinguish content origins.

·   Supported natively by Azure OpenAI’s data source config

·   Azure Cognitive Search - Create and manage indexes

Option2: Use a Custom RAG Orchestration (Bring Your Own Data)

For more flexibility, you can build a custom RAG pipeline using tools like Azure Functions, LangChain, or Semantic Kernel. This allows you to:

1.  Query multiple indexes programmatically using the Cognitive Search REST API.

2.  Aggregate and rerank search results.

3.  Pass the top-ranked documents into a Chat Completion request as part of the prompt.

·   LangChain + Azure OpenAI Tutorial: Build a RAG App with LangChain and Azure OpenAI

·   Cognitive Search REST API Reference: Search Documents - REST API

Option3: Multiple Search Queries in the App Layer

You can also perform separate search queries for each index from your application backend, then merge the results manually and feed them into a single Chat Completion call.

·   This approach is simple and works well for quick POCs or demos, although it requires you to handle the logic externally.

Hope this helps, do let me know if you have any further queries.

Thank you!

Answer 2

Hi @Prashanth Veeragoni Thank you for the detailed explanation. It’s very helpful in understanding the current limitations of Azure OpenAI’s native data source integration.

In our case, document access is governed by fine-grained permissions, and each index corresponds to a different set of documents based on those access controls. Users may have access to one or multiple document sets, meaning one or more indexes may need to be queried simultaneously depending on their permissions. We'll likely look into orchestrating this using Azure Functions or LangChain, so we can retain flexibility while respecting security boundaries.