Create evaluation datasets from annotations (preview)

Important

Items marked preview in this article are currently in preview. This preview is provided without a service-level agreement, and Microsoft doesn't recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.

Add annotated traces to an evaluation dataset. Foundry copies each reviewer's expected output into the dataset and can generate a task-level rubric, which is scoring criteria for that one test case. For an overview, see Human in the loop.

Prerequisites

Add annotated traces to a dataset

  1. In the Foundry portal, open Traces, and then open Trace view or Conversation view.
  2. Select the annotated traces or conversations to include.
  3. Select Add to dataset.
  4. Choose an existing dataset or create one.
  5. To generate task-level rubrics, turn on rubric generation and select a model deployment.
  6. Select Create.
  7. When the notification appears, open the dataset to review the generated rows.

Trace-based data generation redacts private content by default.

What gets generated

Each row is one test case. Along with the interaction (messages and tool_definitions), a row contains these fields:

Field What it contains
test_case_id A stable ID for the test case. Re-adding the same conversation replaces the row with this ID.
annotation The reviewer annotations: rating, comment, tags, and expected output.
expected Expected outputs copied directly from annotations, without model changes. Trace rows use output; conversation rows use outcome. Each entry keeps its annotationId.
rubric_dimensions Up to 20 criteria for this test case, each with a service-generated UUID id and a description. No weights.
rubric_metadata Generation status, the annotations used, and the generator model.
test_case_description A scenario brief for simulation. Conversations only.
metadata Source trace, conversation, and agent IDs, plus generation details.

Example conversation row

The following row comes from a shopping-list agent conversation that a reviewer rated as good. It's abbreviated: the agent's instructions, tool_definitions, metadata, most criteria, and part of the description are shortened or omitted.

{
  "test_case_id": "conversation-64f11dd5dd2b762e98c615d4c275caae",
  "messages": [
    {
      "role": "developer",
      "content": [{ "type": "text", "content": "You are a simple shopping-list assistant. ..." }]
    },
    {
      "role": "user",
      "content": [{ "type": "text", "content": "hi, give me coffee and apples" }]
    },
    {
      "role": "assistant",
      "content": [{ "type": "text", "content": "Added." }]
    },
    {
      "role": "user",
      "content": [{ "type": "text", "content": "also eggs.. what do I have now" }]
    },
    {
      "role": "assistant",
      "content": [{ "type": "text", "content": "1. coffee\n2. apples\n3. eggs" }]
    }
  ],
  "annotation": [
    {
      "rating": "good",
      "comment": "did what I asked",
      "tags": ["Task outcome:Success"],
      "role": "builder",
      "human_evaluation_id": "da090929-b325-44be-9414-2eef6d661f53",
      "expected_output": "coffee, apples and eggs"
    }
  ],
  "expected": [
    {
      "annotationId": "da090929-b325-44be-9414-2eef6d661f53",
      "outcome": "coffee, apples and eggs"
    }
  ],
  "rubric_dimensions": [
    {
      "id": "b702089d-6465-4602-88de-1e7381247974",
      "description": "When the user requests adding items, appends every requested item to the list (if multiple items are requested in one utterance, add all of them)."
    },
    {
      "id": "6bdbdad8-665d-44b3-8dc4-de7e70e4657e",
      "description": "When performing an add operation, replies exactly with \"Added.\" and nothing else."
    },
    {
      "id": "6b56c76d-b3fe-4c85-ab7d-208189751fd9",
      "description": "When the user asks to show the list, returns a plain numbered list of items in the exact order they were added, including all entries."
    }
  ],
  "rubric_metadata": {
    "generation_status": "success",
    "failure_description": "",
    "source_annotation_count": 1,
    "annotations_ids": ["da090929-b325-44be-9414-2eef6d661f53"],
    "generator_model": "gpt-5-mini"
  },
  "test_case_description": "Situation: You asked the assistant to give you specific items ... and then asked \"what do I have now\".\n\nExit criteria: You consider the interaction complete when the assistant tells you what you have now ...\n\nVoice: You write in very short, direct messages ...\n\nUser excerpts: \"hi, give me coffee and apples\" \"also eggs.. what do I have now\""
}

The criteria cover behavior from the agent's instructions as well as the reviewer's feedback.

Generated rows store message text in the content property of each text item, as shown. This format differs from the text property in Evaluation dataset schema.

The test_case_description describes the user's situation, when the user is done, and how the user writes, so you can use it as scenario input to simulate conversations.

Review the generated rows

Before you evaluate, check that each row's criteria:

  • Match the reviewer's feedback and the conversation context.
  • Describe specific, observable behavior.
  • Use your actual policies and expected outcomes.
  • Don't repeat the expected output, which is checked separately.
  • Don't add requirements that the trace or annotations don't support.

Also check sensitive content, and confirm each row has enough context to evaluate.

Empty criteria

An empty rubric_dimensions array can mean two things. Check rubric_metadata.generation_status:

Status Meaning
"success" The model found no criteria to add. This result is valid.
"failed" Generation failed for this row, even if the overall job succeeded. Fix or remove the row before you evaluate.

An empty rubric doesn't mean the agent behaved correctly.

Updates and regeneration

Editing annotations in an existing dataset doesn't regenerate its criteria. To refresh criteria, add the traces to the dataset again:

  • Traces: Traces with changed annotations get new criteria. Unchanged rows that failed aren't retried.
  • Conversations: Each re-added conversation replaces its earlier version, and only new or replaced conversations get new criteria.

Evaluate with the dataset

Use the task-level rubric evaluator (builtin.task_level_rubric) to score each row against its own criteria and expected output. A shared rubric evaluator uses only its own criteria. If you run both evaluators, they score independently. For details, see Rubric evaluators.

The following examples score the responses already stored in the dataset. To test an updated agent version against the same rows, run an agent-targeted evaluation that generates new responses, and map the generated response instead of item.response. See Evaluate your AI agents.

Set DATASET_ID to the ID of the dataset to evaluate.

The following example evaluates a dataset created from traces.

import os
from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential

endpoint = os.environ["FOUNDRY_PROJECT_ENDPOINT"]
deployment_name = os.environ["MODEL_DEPLOYMENT_NAME"]
dataset_id = os.environ["DATASET_ID"]

with DefaultAzureCredential() as credential, AIProjectClient(
    endpoint=endpoint, credential=credential
) as project:
    openai_client = project.get_openai_client()
    eval_obj = openai_client.evals.create(
        name="refund-agent-task-level",
        data_source_config={
            "type": "custom",
            "item_schema": {
                "type": "object",
                "properties": {
                    "messages": {"type": "array"},
                    "expected": {"type": "array"},
                    "rubric_dimensions": {"type": "array"},
                },
                "required": ["messages", "rubric_dimensions"],
            },
            "include_sample_schema": True,
        },
        testing_criteria=[
            {
                "type": "azure_ai_evaluator",
                "name": "task_level_rubric",
                "evaluator_name": "builtin.task_level_rubric",
                "initialization_parameters": {
                    "deployment_name": deployment_name
                },
                "data_mapping": {
                    "messages": "{{item.messages}}",
                    "expected": "{{item.expected}}",
                    "rubric_dimensions": "{{item.rubric_dimensions}}",
                },
            }
        ],
    )

    # Turn level is the default, so evaluation_level isn't set.
    run = openai_client.evals.runs.create(
        eval_id=eval_obj.id,
        name="refund-agent-task-level-run",
        data_source={
            "type": "jsonl",
            "source": {"type": "file_id", "id": dataset_id},
        },
    )
    print(f"Started run {run.id}")

The evaluator returns a pass/fail result named task_level_rubric for each row, with an explanation for each criterion. To view results, see Get evaluation results.

Next step