Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Important
Items marked preview in this article are currently in preview. This preview is provided without a service-level agreement, and Microsoft doesn't recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.
A task-level rubric defines what success means for one task in an evaluation dataset, where each row is one task. The task-level rubric evaluator (builtin.task_level_rubric) scores each response or conversation against that row's own criteria and its human-provided expected output.
Shared quality and safety rubrics apply the same criteria to every row. That works well for general standards, such as "confirm reservation details." It's less precise when each case has a different correct outcome. A task-level rubric captures the detail that matters for the individual case, often taken directly from a reviewer's feedback on a production trace.
| Aspect | Shared rubric | Task-level rubric |
|---|---|---|
| Criteria defined | Once, on the evaluator | Per row, in the dataset |
| Typical source | Agent instructions, prompts, reference data, or traces | Human annotations on traces and conversations |
| Checks expected output | No | Yes |
| Weights | Optional in Likert quality rubrics | Not included in generated criteria |
Generate a task-level rubric
Create an evaluation dataset from annotated traces, and turn on rubric generation. Rubric generation requires a model deployment. To annotate traces and add expected outputs first, see Human in the loop.
- In the Foundry portal, open your project and agent, and then open Traces.
- In the trace or conversation view, select the annotated traces or conversations to include.
- Select Add to dataset, and then choose an existing dataset or create one.
- Turn on rubric generation, and select a model deployment.
- Select Create. When the notification appears, open the dataset and review the generated criteria and expected outputs.
For the complete workflow, see Convert agent traces into evaluation datasets.
Dataset fields
Each row in the evaluation dataset is one task. Along with the interaction to evaluate, a row has two task-level fields:
| Field | Description |
|---|---|
expected |
The expected output for the task. Each entry needs an annotationId string. For query and response rows, the entry also needs output. For messages rows, it needs outcome. |
rubric_dimensions |
Criteria for this task only. Each criterion has an id and a description, with no weights. |
You can write these fields yourself, or generate them from annotated traces. Generated rows set annotationId to the source annotation.
Each row needs rubric_dimensions and the interaction to evaluate, as either messages or query and response. Conversation-level evaluation also needs test_case_description. The expected field is optional. If any entry in expected is malformed, the evaluator ignores the whole field and grades only the criteria.
Example
In this illustrative scenario, a reviewer gives a thumbs-down to a refund response and writes: "The response omitted the 30-day deadline and didn't say how long the refund takes." The reviewer also supplies an expected output.
The resulting row contains the interaction, the reviewer's expected output, and criteria for this task:
{
"query": "I'd like a refund for the headphones I bought last week.",
"response": "Sure! Go to Billing, select the headphones, and choose Request refund.",
"expected": [
{
"annotationId": "refund-review-1",
"output": "Refund requests must be submitted within 30 days of purchase. Go to Billing, select the headphones, and choose Request refund."
}
],
"rubric_dimensions": [
{
"id": "no_out_of_policy_refund_promise",
"description": "The response must not promise a refund for a purchase that's older than 30 days."
},
{
"id": "states_processing_time",
"description": "The response should say that refunds take 5-7 business days to process after approval."
}
]
}
The policy, steps, and IDs are illustrative. The criteria cover behavior beyond the expected answer, so they don't repeat it. When you evaluate a new agent version against this row, the evaluator checks:
- The expected output: Does the response give the 30-day deadline and the Billing steps?
- Each criterion: Does the response avoid promising an out-of-policy refund? Does it say how long the refund takes?
The response in this row fails both checks: it omits the deadline and doesn't say how long the refund takes. Another row in the same dataset can have completely different criteria, such as checking that the agent asks for a missing order number.
Review generated criteria and expected outputs before you use the dataset. If you also run a shared rubric evaluator, it uses only its own criteria and scores independently.
Run an evaluation
Add builtin.task_level_rubric to an evaluation run, and map each row's fields to the evaluator. The evaluator requires a model deployment for the LLM judge.
| Evaluator input | Required | Maps from |
|---|---|---|
query and response, or messages |
Yes | The interaction to evaluate. messages takes precedence when present. |
rubric_dimensions |
Yes | The row's task-level criteria |
expected |
No | The row's expected output |
test_case_description |
For conversation-level evaluation | The scenario for the whole conversation |
tool_definitions |
No | The tools available to the agent |
For conversation-level evaluation, set evaluation_level to conversation on the run, and map test_case_description.
The following Python example evaluates the refund row from the previous section:
import os
from azure.ai.projects import AIProjectClient
from azure.identity import DefaultAzureCredential
endpoint = os.environ["FOUNDRY_PROJECT_ENDPOINT"]
deployment_name = os.environ["MODEL_DEPLOYMENT_NAME"]
row = {
"query": "I'd like a refund for the headphones I bought last week.",
"response": (
"Sure! Go to Billing, select the headphones, "
"and choose Request refund."
),
"expected": [
{
"annotationId": "refund-review-1",
"output": (
"Refund requests must be submitted within 30 days of "
"purchase. Go to Billing, select the headphones, and "
"choose Request refund."
)
}
],
"rubric_dimensions": [
{
"id": "no_out_of_policy_refund_promise",
"description": (
"The response must not promise a refund for a purchase "
"that's older than 30 days."
),
},
{
"id": "states_processing_time",
"description": (
"The response should say that refunds take 5-7 "
"business days to process after approval."
),
},
],
}
with DefaultAzureCredential() as credential, AIProjectClient(
endpoint=endpoint, credential=credential
) as project:
openai_client = project.get_openai_client()
eval_obj = openai_client.evals.create(
name="refund-task-level-rubric",
data_source_config={
"type": "custom",
"item_schema": {
"type": "object",
"properties": {
"query": {"type": "string"},
"response": {"type": "string"},
"expected": {"type": "array"},
"rubric_dimensions": {"type": "array"},
},
"required": ["query", "response", "rubric_dimensions"],
},
"include_sample_schema": True,
},
testing_criteria=[
{
"type": "azure_ai_evaluator",
"name": "task_level_rubric",
"evaluator_name": "builtin.task_level_rubric",
"initialization_parameters": {
"deployment_name": deployment_name
},
"data_mapping": {
"query": "{{item.query}}",
"response": "{{item.response}}",
"expected": "{{item.expected}}",
"rubric_dimensions": "{{item.rubric_dimensions}}",
},
}
],
)
run = openai_client.evals.runs.create(
eval_id=eval_obj.id,
name="refund-task-level-rubric-run",
data_source={
"type": "jsonl",
"source": {
"type": "file_content",
"content": [{"item": row}],
},
},
)
print(f"Started run {run.id}")
To evaluate a dataset that you generated from traces, use the dataset as the run's data source instead of inline content. For data sources and run configuration, see Run evaluations from the SDK.
Example output
The evaluator returns a pass/fail result named task_level_rubric. Results for each criterion are listed under properties.dimensions, each with a result and a reason. When the row has an expected field, the expected check appears as properties.expected_output_result for query and response rows, or properties.expected_outcome_result for messages rows.
The following output is for the refund trace in the example:
{
"type": "azure_ai_evaluator",
"name": "TaskLevelRubric",
"metric": "task_level_rubric",
"score": false,
"label": "fail",
"reason": "The expected output check failed.",
"passed": false,
"status": "completed",
"properties": {
"dimensions": [
{
"id": "no_out_of_policy_refund_promise",
"result": "pass",
"reason": "The response gives refund steps but doesn't promise a refund for an older purchase."
},
{
"id": "states_processing_time",
"result": "fail",
"reason": "The response doesn't say how long the refund takes to process."
}
],
"expected_output_result": {
"result": "fail",
"reasoning": "The response omits the 30-day deadline from the expected output."
}
}
}
The task result follows these rules:
- If the expected check fails, the task fails.
- Otherwise, the task passes only when every applicable criterion passes. A criterion that doesn't apply to the interaction shows
skippedand doesn't count. - If no criteria apply and the expected check passes, the task passes.
- If the row has no criteria and no
expectedfield, the task is skipped:statusisskippedandpassedisnull.
Use the reason for each criterion and the expected check's reasoning to see why a task failed.