How to make sure that all documents retrieved from RAG (Azure Search) are evaluated by the LLM?

Question

I am trying to evaluate a list of candidates through Azure Open AI. We have built a RAG solution with Azure AI Search and Azure OpenAI (GPT 4o). The candidates are retrieved through Azure AI search by a query filter and passed to LLM. The problem is that the LLM seems to skip some documents while evaluating. For e.g. if 100 candidates are retrieved through Azure AI Search and passed to LLM, only 20-30 will get evaluated and returned as citations, the rest will be skipped and they will not be referenced in citations also.

Could you please suggest some techniques which could solve this problem? We have tried rigorous prompt engineering, but it doesn't work. Also is there a way to see the retrieved documents before they are passed to LLM for evaluation.

Answer 1

Hello Piyush Athawale,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you would like to ensure that all documents retrieved from RAG (Azure Search) are evaluated by the LLM.

It happened sometimes that with large language models (LLMs) where not all retrieved documents are being evaluated or cited. To resolve this issue of skipping documents during evaluation, improve the evaluation process and ensure more documents are considered, you can use several techniques such as batch processing to handle smaller groups of documents, optimizing the context window to include the most relevant parts, preprocessing documents to highlight key information, iterative refinement of the evaluation process, and developing custom evaluation metrics. Additionally, implementing a preview step can help you review the retrieved documents before they are passed to the LLM. Use these links for more detailed info: https://composio.dev/blog/llm-evaluation-guide and https://labelstud.io/blog/llm-evaluations-techniques-challenges-and-best-practices

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

---

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.