Selecting the number of chunks to use as context when doing RAG?

Question

When you are using the Python SDK for Azure OpenAI to generate text based on user input and RAG, how do you specify the number of chunks of text to use as context?

I read that it involves some parameter called "topNDocuments," but where does that go? It is not recognized as an argument to client.chat.completions.create().

Answer 1

Hello Bao, Jeremy (Cognizant),

Regarding the topNDocuments you mentioned, please refer to this document and try the following example code.

from openai import AzureOpenAI

client = AzureOpenAI(
    azure_endpoint=...,
    api_key=...,
    api_version="2024-02-01",
)

completion = client.chat.completions.create(
    model=...,
    messages=[...],
    extra_body={
        "data_sources": [
            {
                "type": "azure_search",
                "parameters": {
                    "endpoint": "<search_endpoint>",
                    "index_name": "<search_index>",
                    "authentication": {
                        "type": "system_assigned_managed_identity"
                    },
                    "top_n_documents": 5
                }
            }
        ]
    }
)

Best regards,
Charlie

---

If you find my response helpful, please consider accepting this answer and voting 'yes' to support the community. Thank you!