How to make sure that all documents retrieved from RAG (Azure Search) are evaluated by the LLM?

Piyush Athawale 5 Reputation points
2024-10-21T09:13:56.8566667+00:00

I am trying to evaluate a list of candidates through Azure Open AI. We have built a RAG solution with Azure AI Search and Azure OpenAI (GPT 4o). The candidates are retrieved through Azure AI search by a query filter and passed to LLM. The problem is that the LLM seems to skip some documents while evaluating. For e.g. if 100 candidates are retrieved through Azure AI Search and passed to LLM, only 20-30 will get evaluated and returned as citations, the rest will be skipped and they will not be referenced in citations also.

Could you please suggest some techniques which could solve this problem? We have tried rigorous prompt engineering, but it doesn't work. Also is there a way to see the retrieved documents before they are passed to LLM for evaluation.

Azure AI Search
Azure AI Search
An Azure search service with built-in artificial intelligence capabilities that enrich information to help identify and explore relevant content at scale.
1,354 questions
Azure OpenAI Service
Azure OpenAI Service
An Azure service that provides access to OpenAI’s GPT-3 models with enterprise capabilities.
4,101 questions
0 comments No comments
{count} vote

1 answer

Sort by: Most helpful
  1. Sina Salam 22,031 Reputation points Volunteer Moderator
    2024-10-21T17:52:37.4666667+00:00

    Hello Piyush Athawale,

    Welcome to the Microsoft Q&A and thank you for posting your questions here.

    I understand that you would like to ensure that all documents retrieved from RAG (Azure Search) are evaluated by the LLM.

    It happened sometimes that with large language models (LLMs) where not all retrieved documents are being evaluated or cited. To resolve this issue of skipping documents during evaluation, improve the evaluation process and ensure more documents are considered, you can use several techniques such as batch processing to handle smaller groups of documents, optimizing the context window to include the most relevant parts, preprocessing documents to highlight key information, iterative refinement of the evaluation process, and developing custom evaluation metrics. Additionally, implementing a preview step can help you review the retrieved documents before they are passed to the LLM. Use these links for more detailed info: https://composio.dev/blog/llm-evaluation-guide and https://labelstud.io/blog/llm-evaluations-techniques-challenges-and-best-practices

    I hope this is helpful! Do not hesitate to let me know if you have any other questions.


    Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.