Implemented RAG with AzureOpenAI, but It Doesn't Respond to Irrelevant Questions

Question

Hello,

I have implemented RAG (Retrieval-Augmented Generation) with AzureOpenAI. The RAG functionality can be utilized through the API. However, I have encountered an issue.

When a question is asked that does not hit on any relevant information within RAG, it does not provide any response. I was hoping it would automatically default to a typical ChatGPT-like response when no relevant information is found in RAG.

Is it normal for RAG to only perform searches within its scope once implemented? If no response is generated, do I need to make another API call to obtain an answer?

If anyone has a good solution for this, please let me know.

Thank you.

Best regards,

Answer 1

@KaitoNishimura Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

.

Please follow the below action plan in order:

.

Plan 1:

After you add your own data and the ingestion completes, Expand the Advanced Settings and then please uncheck the Limit responses to your data content option shown below:

Then test the functionality in the chat playground before you deploy your code.

Plan 2:

From your application's API call, if you want to ensure that the responses from the API are not limited to the content of the grounding data only, then you need to rely on the ShouldRestrictResultScope parameter. You need to set this to ___false ___within the AzureSearchChatExtensionConfiguration class.

More info here and here.

.

Plan 3:

Please update / append your System message with the below:

Please answer the question using a combination of retrieved documents and your internal knowledge. Feel free to supplement the information extracted from the documents with additional facts. Remember to cite the documents from which you extract information. Consider step by step what information needs to be added or is missing from the retrieved documents. If the necessary information is not found in the retrieved documents, feel free to use your internal knowledge to provide a comprehensive answer.

.

Then click on Apply Changes as shown below:

Plan 4:

If the above plan doesn't help, I believe the behavior is related to your chat playground and not the API itself. Please try to invoke the Azure Open AI chat completion REST API as shown here from postman and check the behavior:

https://learn.microsoft.com/en-us/azure/ai-services/openai/reference#azure-ai-search

If you want more details about the headers used in the API refer this thread.

Please remember to set the "inScope": false in your request.

.

Hope this helps. If you have any follow-up questions, please let me know. I would be happy to help.