How to Evaluate a Custom RAG Pipeline in Azure AI Foundry

Question

Hi, I’m working with a custom Retrieval-Augmented Generation (RAG) pipeline built outside of Azure AI Foundry. I would like to evaluate its performance using the evaluation features available in AI Foundry.

I’m unsure which evaluation type is appropriate in this case:

Agent

Model

Dataset

Since my pipeline includes both retrieval and generation handled externally, could you please clarify:

Which evaluation option should be used for a custom RAG pipeline?

How should I structure the evaluation input (for example: query, retrieved context, and generated response)?

Any guidance or recommended approach would be helpful.

Thank you.

Answer 1

Hello Parul Paul,

Welcome to Microsoft Q&A and Thank you for reaching out.

Thank you for the articulate and detailed description of the case. It is understood that you are looking to evaluate your external custom built RAG model’s performance with evaluation metrics available on Azure AI Foundry.

Please go ahead to use Dataset evaluation, since

1. The custom-built RAG is external
2. Azure AI Foundry’s RAG evaluators operate on data and not on how the pipeline was built.

The references below can help you to have a quick headstart with evaluation types and metrics.

Retrieval-Augmented Generation (RAG) Evaluators for Generative AI - Microsoft Foundry | Microsoft Learn

Evaluate Generative AI Models and Apps with Microsoft Foundry - Microsoft Foundry | Microsoft Learn

It can be structured in the below format while using dataset evaluation mode , please upload JSON with:

1. query
2. context
3. response
4. ground_truth (optional but recommended)

It would look something like the below snippet.

{
  "id": "Unique identifier for the evaluation record",
  "query": "User question sent to the RAG system",
  "context": "All retrieved chunks concatenated exactly as provided to the LLM",
  "response": "Final generated answer from the RAG system",
  "ground_truth": "Reference answer created by human annotators for accuracy evaluation",
  "metadata": {
    "domain": "Business or knowledge domain classification",
    "difficulty": "Difficulty level such as easy, medium, hard",
    "source": "Primary document source or knowledge base identifier",
    "language": "Language of the query and response",
    "version": "Dataset or document version identifier"
  },
  "retrieval": {
    "top_k": "Number of retrieved documents returned by the retriever",
    "retrieved_documents": [
      {
        "doc_id": "Unique identifier of the retrieved document",
        "rank": "Position in retrieval ranking",
        "score": "Similarity score returned by the retriever",
        "content_length": "Length of the retrieved document chunk in tokens or characters"
      }
    ],
    "relevant_document_ids": [
      "List of document identifiers that are truly relevant for recall/precision evaluation"
    ]
  },
  "evaluation_tags": {
    "expected_behavior": "Intended behavior classification such as factual, reasoning, multi-hop, policy-based",
    "risk_category": "Risk type such as hallucination-sensitive, compliance-sensitive, safety-critical"
  }
}

Please use the below evaluation metrics to understand the model’s performance

Groundedness + Hallucination + Relevance + Correctness together to ensure together the RAG system

1.      answers the right question (Relevance)

2.      bases its answer on retrieved evidence (Groundedness)

3.      avoids inventing unsupported facts (Hallucination)

4.      is factually accurate against a verified reference (Correctness)

Please refer the following as additional reference material

Evaluating and Optimizing RAG Agents with Azure AI Foundry: Best Practices for Optimization 

Please let me know if you have any questions. 

Thank you!