How to Evaluate Agent in AI Foundry

query vs test_case_description

In Azure AI Foundry evaluations, the important field for driving the agent or model is query. This is the input that is actually sent to the model/agent during evaluation.

When using synthetic data generation (either from the studio UI or via the azure_ai_synthetic_data_gen_preview scenario in cloud evaluation), the service generates synthetic test queries and stores them in a dataset. In that dataset:

• query
  • Contains the actual test input that will be sent to the target (model or agent).
  • In cloud evaluation, this is the field referenced as {{item.query}} in data mappings and message templates.
  • For synthetic data evaluation, the documentation explicitly states that the generator produces queries in {{item.query}}.
• test_case_description
  • Describes the intent or scenario of the test case in natural language (for example, what the query is supposed to test or what behavior is expected).
  • It is metadata for humans or for custom evaluators, not the field that is sent to the agent by default.
  • Built‑in evaluators and targets use query as the primary input; test_case_description is only used if explicitly mapped in a custom schema or custom evaluator.

So, for built‑in automatic evaluation flows, treat query as the operative input and test_case_description as descriptive context about that test.

How automatic evaluation works with an agent

Automatic evaluation with an agent in Azure AI Foundry follows the same general pattern whether using a pre‑existing dataset or synthetic data:

1. Define the evaluation and data schema
   • An evaluation defines the test data schema and testing criteria (evaluators).
   • For agent evaluation, the schema typically includes at least a query field:

          data_source_config = {
              "type": "custom",
              "item_schema": {
                  "type": "object",
                  "properties": {
                      "query": {"type": "string"},
                  },
                  "required": ["query"],
              },
              "include_sample_schema": True,
          }
     

   • include_sample_schema: True ensures that evaluator mappings can reference the agent’s runtime outputs via {{sample.*}}.
2. Choose the data source scenario
   • For agent target evaluation with existing data, use azure_ai_target_completions.
   • For synthetic data evaluation (preview), use azure_ai_synthetic_data_gen_preview with parameters like samples_count, model_deployment_name, and prompt. The service:
     1. Generates synthetic queries into {{item.query}}.
     2. Sends each query to the target (model or agent).
     3. Evaluates the responses.
     4. Stores the generated queries as a dataset for reuse.
3. Send queries to the agent at runtime
   • For agent target evaluation, the input_messages template uses {{item.query}} to construct the message(s) sent to the agent.
   • The target is configured as an azure_ai_agent:
     • The service sends each query from the dataset to the agent.
     • The agent executes its full logic (including tools, RAG, etc.) and returns a response.
4. Capture outputs for evaluators
   • During the run, the service captures:
     • {{sample.output_text}}: the agent’s plain text reply.
     • {{sample.output_items}}: the full structured response, including tool calls and tool results.
   • Evaluators use these fields according to their data mappings:
     • {{item.field}} for input data (for example, {{item.query}}).
     • {{sample.output_text}} for text‑based metrics.
     • {{sample.output_items}} for tool‑aware or context‑aware metrics.
5. Apply evaluators and score results
   • Agent evaluators (for example, Task Completion, Task Adherence, Intent Resolution, Tool Call Accuracy, Tool Selection, etc.) are configured with:
     • Required inputs (for example, query, response, tool_definitions).
     • A deployment_name for the judge model (AI‑assisted evaluators).
   • Each evaluator returns a score (often on a 1–5 Likert scale) plus label, threshold, and pass/fail status.

Why RAG metrics (Similarity, Response Completeness, Retrieval) are not shown in agent evaluation

RAG‑specific evaluators are available, but they are configured based on the data schema and mappings, not automatically surfaced just because the agent uses RAG.

RAG evaluators include:

• Groundedness / Groundedness Pro
• Relevance
• Response Completeness (preview)
• Retrieval
• Document Retrieval

Each requires specific inputs and mappings, for example:

• Groundedness: (response, context) or (query, response).
• Response Completeness: ground_truth, response.
• Retrieval: query, context.
• Document Retrieval: retrieval_ground_truth, retrieved_documents.

In an agent target evaluation:

• The agent’s RAG behavior (retrieval, similarity search, etc.) is internal to the agent.
• The evaluation service only sees what is exposed in the agent’s response (output_text / output_items) and any fields in the dataset.
• To use RAG evaluators with an agent, the evaluation must:
  • Provide or expose the necessary fields (for example, retrieved context, ground truth, or retrieval ground truth) in the dataset or via the agent’s structured output.
  • Map those fields explicitly in the evaluator configuration.

If those fields are not present or mapped, RAG metrics such as Similarity, Response Completeness, and Retrieval will not appear as available options in the agent evaluation configuration, even though the agent itself uses RAG internally.

In other words:

• RAG functionality in the agent does not automatically provide the evaluation inputs required by RAG evaluators.
• To use RAG metrics with an agent, the evaluation must be set up so that the dataset and/or agent outputs expose context, ground_truth, retrieved_documents, etc., and those are wired into the RAG evaluators via data_mapping.

References:

• Evaluate generative AI models and applications by using Microsoft Foundry
• Run evaluations in the cloud by using the Microsoft Foundry SDK
• Evaluate your AI agents
• Retrieval-Augmented Generation (RAG) evaluators
• Agent evaluators