Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
A quality rubric evaluator scores an agent or model response against criteria that define good behavior for your application. An LLM judge scores each criterion (dimension) and combines the results into an overall score.
Use a quality rubric to measure domain-specific behavior, such as whether an agent follows business rules, uses tools correctly, or confirms important details. For rubric types and lifecycle management, see Rubric evaluators.
Choose a scoring mode
Each rubric uses one scoring mode for all of its dimensions.
| Scoring mode | How it scores | Best for |
|---|---|---|
1–5 rating ("likert", default) |
Rates each dimension from 1 to 5 and combines the ratings into a score from 0.0 to 1.0. The item passes when the score meets pass_threshold (default 0.5). |
Graded quality, where some criteria can matter more than others |
Pass/fail ("boolean") |
Checks each dimension. The item passes only when every applicable check passes. | Mandatory checklists |
In 1–5 rating mode, you can weight dimensions from 1 to 10. Either weight every dimension or none of them. Pass/fail mode doesn't support weights.
Generate a quality rubric
Generate a rubric with a supported judge model from one or more of these sources:
| Source | type |
What it uses |
|---|---|---|
| Foundry agent | agent |
A prompt agent's instructions or a hosted agent's description |
| Prompt | prompt |
Instructions or quality goals that you write |
| Dataset | dataset |
Registered datasets that contain domain guidelines or specifications |
| Traces | traces |
Production traces from Application Insights. Combine traces with at least one other source. |
If the inputs are too thin, generation still succeeds but returns input-quality warnings. Add more detail to your sources and regenerate.
- In your Foundry project, go to Evaluation > Evaluator catalog, and open the Create evaluator pane.
- Enter an Evaluator name. Optionally, add a Display name and Description.
- For Evaluator type, select Rubric. For Category, select Quality.
- For Scoring method, select Numeric for a 1–5 rating or Boolean for pass/fail.
- Turn on Auto-generate rubric.
- For Model, select a supported judge model. Optionally, select a Target agent.
- In Prompt, describe what the evaluator should assess, including criteria and expected behavior.
- Optionally, turn on Add context. Select Use production traces from target agent, or upload
.json,.jsonl,.csv, or.txtfiles under Upload context files. - Create the evaluator. When generation finishes, review the generated dimensions.
The generated rubric is saved as a new evaluator version. Review its dimensions, and then run an evaluation with it.
Choose a judge model
The following models are supported for quality rubric generation and scoring:
| Model | Recommendation |
|---|---|
gpt-5.5, gpt-5.4, gpt-5.4-nano, gpt-5.2, gpt-5.1, gpt-5, gpt-5-mini, gpt-5-nano |
Recommended |
gpt-5.4-mini |
Recommended (best balance of performance and cost) |
gpt-4.1, gpt-4o |
Acceptable |
Create a quality rubric manually
You can also write a rubric yourself. In the portal, turn off Auto-generate rubric in the Create evaluator pane. In code, use project.evaluators.create_version. For an example, see Run an evaluation.
Tip
Start with a generated rubric and refine it. Generation gives you a grounded baseline that you can tailor to your standards.
Dimension fields
A quality rubric can have up to 20 dimensions.
| Field | Required | Description |
|---|---|---|
id |
Yes | A stable snake_case identifier. Keep it unchanged when you edit a dimension so its history stays connected. |
description |
Yes | A clear, observable criterion. |
weight |
No | Relative importance from 1 to 10. 1–5 rating mode only. |
always_applicable |
No | When true, the judge always scores the dimension instead of first checking whether it applies. Generated 1–5 rubrics set this on a catch-all general_quality dimension. |
For the complete schema, see the Foundry REST API reference.
Example rubrics
The following rubrics evaluate a restaurant reservation agent.
This rubric weights intent recognition most heavily:
{
"type": "rubric",
"scoring": { "mode": "likert" },
"pass_threshold": 0.5,
"dimensions": [
{
"id": "intent_recognition",
"description": "Correctly identifies the user's reservation intent (book, modify, cancel, inquire) and pursues the appropriate workflow without unnecessary clarification.",
"weight": 9
},
{
"id": "tool_usage_accuracy",
"description": "Calls the correct tool with correct parameters. Does not call tools unnecessarily, and does not skip tool calls when they are needed.",
"weight": 6
},
{
"id": "policy_enforcement",
"description": "Enforces business rules: dinner service 17:00-22:00, max party size 8, 30-day booking window. Does not create reservations that violate these constraints.",
"weight": 5
},
{
"id": "information_gathering",
"description": "Collects all required information (date, time, party size, contact) before attempting to create a reservation. Does not ask for information already provided.",
"weight": 4
},
{
"id": "communication_clarity",
"description": "Provides clear, concise responses. Confirms reservation details before finalizing. Uses a professional and helpful tone.",
"weight": 2
},
{
"id": "general_quality",
"description": "Other important quality factors not already covered by the listed criteria.",
"weight": 5,
"always_applicable": true
}
]
}
Run an evaluation
Quality rubric evaluators run on your model deployment and incur inference costs. The following Python example creates a weighted 1–5 rubric and evaluates two reservation responses:
import os
from azure.ai.projects import AIProjectClient
from azure.ai.projects.models import EvaluatorCategory
from azure.identity import DefaultAzureCredential
endpoint = os.environ["FOUNDRY_PROJECT_ENDPOINT"]
deployment_name = os.environ["MODEL_DEPLOYMENT_NAME"]
evaluator_name = "reservation-quality"
with DefaultAzureCredential() as credential, AIProjectClient(
endpoint=endpoint, credential=credential
) as project:
# 1. Create the rubric evaluator.
evaluator = project.evaluators.create_version(
name=evaluator_name,
evaluator_version={
"name": evaluator_name,
"categories": [EvaluatorCategory.QUALITY],
"display_name": "Reservation quality",
"definition": {
"type": "rubric",
"scoring": {"mode": "likert"},
"pass_threshold": 0.6,
"dimensions": [
{
"id": "intent_recognition",
"description": (
"Identifies the reservation intent (book, "
"modify, or cancel) and follows that workflow."
),
"weight": 9,
},
{
"id": "completeness",
"description": (
"Confirms party size, date, time, and contact "
"details before completing the task."
),
"weight": 6,
},
{
"id": "professional_tone",
"description": "Keeps a polite, professional tone.",
"weight": 3,
},
],
},
},
)
# 2. Create an evaluation that uses the rubric.
openai_client = project.get_openai_client()
eval_obj = openai_client.evals.create(
name="reservation-quality-eval",
data_source_config={
"type": "custom",
"item_schema": {
"type": "object",
"properties": {
"query": {"type": "string"},
"response": {"type": "string"},
},
"required": ["query", "response"],
},
"include_sample_schema": True,
},
testing_criteria=[
{
"type": "azure_ai_evaluator",
"name": evaluator_name,
"evaluator_name": evaluator.name,
"evaluator_version": str(evaluator.version),
"initialization_parameters": {
"deployment_name": deployment_name
},
"data_mapping": {
"query": "{{item.query}}",
"response": "{{item.response}}",
},
}
],
)
# 3. Run the evaluation on inline data.
run = openai_client.evals.runs.create(
eval_id=eval_obj.id,
name="reservation-quality-run",
data_source={
"type": "jsonl",
"source": {
"type": "file_content",
"content": [
{
"item": {
"query": "Can I book a table for 4 at 7 PM?",
"response": (
"You're booked for 4 at 7:00 PM. "
"What's a good contact number?"
),
}
},
{
"item": {
"query": "I need to cancel my Friday reservation.",
"response": "ok",
}
},
],
},
},
)
print(f"Started run {run.id}")
The first response should pass. The second should fail because it doesn't confirm which reservation to cancel. To view scores, see Get evaluation results. The output format is shown in Example output.
For more samples that generate, edit, and run quality rubrics, see sample_rubric_evaluator_manual.py, sample_rubric_evaluator_generation_basic.py, and the evaluation samples README.
Example output
The evaluator returns a normalized score, passed, threshold, a reason, and results for each dimension. The score is calculated as follows:
effective_weight_i = weight_i / sum(applicable weights in this row)
contribution_i = effective_weight_i * (raw_score_i - 1) / 4
score = sum(applicable contributions)
rating_1_to_5 = 1 + (4 * score)
Dimensions that don't apply to an item don't affect the score. If no dimension applies, the item is "skipped" and excluded from pass rates.
{
"status": "completed",
"score": 0.9274193548,
"label": "pass",
"reason": "The verdict is driven most by intent_recognition (5), tool_usage_accuracy (5), and policy_enforcement (5). The assistant correctly identified the booking intent, called the reservation tool with valid parameters (Friday 7:30 PM, party of 4), and returned a clear confirmation with the reservation details.",
"threshold": 0.5,
"passed": true,
"properties": {
"mode": "likert",
"dimension_results": [
{
"id": "intent_recognition",
"raw_score": 5,
"applicable": true,
"weight": 9,
"effective_weight": 0.2903225806,
"contribution": 0.2903225806,
"reason": "The user's request to book a table is correctly identified, and the assistant pursues the booking workflow without unnecessary clarification."
},
{
"id": "tool_usage_accuracy",
"raw_score": 5,
"applicable": true,
"weight": 6,
"effective_weight": 0.1935483871,
"contribution": 0.1935483871,
"reason": "The reservation tool is called once with the correct date, time, and party size parameters derived from the user's request."
},
{
"id": "policy_enforcement",
"raw_score": 5,
"applicable": true,
"weight": 5,
"effective_weight": 0.1612903226,
"contribution": 0.1612903226,
"reason": "The reservation falls within dinner service hours, the party size is within the maximum of 8, and the date is within the 30-day booking window."
},
{
"id": "information_gathering",
"raw_score": 4,
"applicable": true,
"weight": 4,
"effective_weight": 0.1290322581,
"contribution": 0.0967741935,
"reason": "All required information (date, time, party size, contact) is captured from the request without asking for details already provided."
},
{
"id": "communication_clarity",
"raw_score": 5,
"applicable": true,
"weight": 2,
"effective_weight": 0.064516129,
"contribution": 0.064516129,
"reason": "The confirmation is concise and includes the reservation date, time, and party size in a single clear message."
},
{
"id": "general_quality",
"raw_score": 4,
"applicable": true,
"weight": 5,
"effective_weight": 0.1612903226,
"contribution": 0.1209677419,
"reason": "Overall execution is strong: the assistant handles the booking end to end with no unnecessary turns or recovery steps."
}
]
}
}
Note
Rubrics created without a top-level scoring object keep their legacy weighted 1–5 scoring and report properties.dimension_scores.