Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Important
Items marked preview in this article are currently in preview. This preview is provided without a service-level agreement, and Microsoft doesn't recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.
Important
Microsoft-Decision-1.1 isn't supported as an evaluation judge. Use Microsoft-Decision-1 instead.
AI-assisted evaluators use a model as a judge to score the responses of your models and agents. The judge is usually a generative model, such as a GPT model, that writes a score and an explanation for each response. You can also use a decision model, Microsoft-Decision-1, as the judge. It answers each evaluation question with a probability instead of writing text. Microsoft tested Microsoft-Decision-1 as a judge. It ran evaluations faster and changed fewer results between runs than the generative judges tested.
In this article, you learn what a decision model is and when to use one for evaluations. Then you deploy Microsoft-Decision-1 in Microsoft Foundry and use it as the judge in the Azure AI Evaluation SDK, in cloud evaluations, and in the Foundry portal.
What is a decision model?
A decision model makes a judgment about an input instead of writing text. You give it the input, such as a user query and an agent response, and a question about that input. The model returns a typed answer:
| Question | Answer |
|---|---|
| Is this true? | The probability, from 0 to 1, that it's true. |
| Which option is it? | One option from a list, with a probability for each option. |
| How much? | A score on an ordered scale, with a probability for each level. The score is the probability-weighted average of the levels, so it can fall between two levels. |
A decision model doesn't write a response or explain its answer. This article uses Microsoft-Decision-1, a decision model in the Foundry model catalog that works with text and JSON. For general information about decision models and their API, see Deploy and use Microsoft-Decision-1.1 in Microsoft Foundry. That guide covers a different model, which isn't supported as an evaluation judge.
Why use a decision model for evaluations?
Most evaluation questions are judgments with a fixed set of answers. Did the agent follow the instructions? Is the response grounded in the context? How coherent is the response, on a scale from 1 to 5? A generative judge answers these questions by writing text that the evaluator turns into a score. A decision model answers them directly.
The following table compares decision model judges with generative judges.
| Area | Decision model judge | Generative model judge |
|---|---|---|
| Speed | Answers in a few output tokens. In Microsoft's tests of Microsoft-Decision-1, evaluation calls were several times faster than with the generative judges tested. | Writes a score and an explanation for every call. |
| Cost | With Microsoft-Decision-1, you pay for input tokens only. | You pay for input and output tokens. |
| Consistency | In Microsoft's tests of Microsoft-Decision-1, fewer results changed when the same evaluation ran twice than with the generative judges tested. Results aren't fully deterministic. | Results can change between runs. |
| Probabilities | Scored results include the probabilities behind them. For pass or fail evaluators, you choose how sure the model must be before a response passes. | Results don't include probabilities. |
| Output | A typed answer, so there's no text to parse. | Text that the evaluator parses. Badly formatted text can cause errors. |
| Explanations | No written explanation. The reason field contains a short, standard summary of the score. | A written explanation for each result. |
For current prices, see Foundry Models pricing.
Use a decision model judge when:
- You evaluate large datasets or production traffic regularly, and speed and cost matter.
- You compare results over time, for example in a CI/CD pipeline, and want fewer results to change between runs.
- You want to set your own pass threshold for pass or fail evaluators.
Use a generative model judge when:
- You need a written explanation of each result, for example to understand why responses fail.
- Your inputs are longer than the decision model's context window of 32,768 tokens, or include images or audio.
- The evaluator you need isn't in Supported evaluators.
You can also use both. For example, run a decision model judge on every response, and then re-evaluate the failed or borderline responses with a generative judge to get a second assessment with a written explanation.
How evaluators score with a decision model
Each supported evaluator asks the decision model two questions about every input:
Can the input be evaluated? The evaluator checks that the input has what it needs, such as a nonempty query and response. If the probability is below 0.5, the evaluator skips the input and returns
not_applicable.The evaluation question. How the answer becomes a score depends on the evaluator:
- Pass or fail evaluators score 1 when the probability is at or above the probability threshold, and 0 otherwise. The default threshold is 0.75. The result passes only when the score is 1.
- Scale evaluators don't use the model's weighted average score. They score the most likely level on a scale from 1 to 5, so scores are whole numbers. If two levels are equally likely, the evaluator picks the lower level. The result passes when the score is at or above the evaluator's threshold, the same as with a generative judge.
Because the model doesn't explain its answer, the reason field describes how the score was chosen. For example: Selected coherence level 4 from the highest-probability level. No model rationale is available.
Supported evaluators
The following built-in evaluators accept a decision model judge.
| Evaluator | Result | Public SDK class | Cloud evaluator name |
|---|---|---|---|
| Coherence | Scale | CoherenceEvaluator |
builtin.coherence |
| Fluency | Scale | FluencyEvaluator |
builtin.fluency |
| Groundedness | Scale | GroundednessEvaluator |
builtin.groundedness |
| Relevance | Scale | RelevanceEvaluator |
builtin.relevance |
| Retrieval | Scale | RetrievalEvaluator |
builtin.retrieval |
| Response completeness | Scale | ResponseCompletenessEvaluator |
builtin.response_completeness |
| Similarity | Scale | SimilarityEvaluator |
builtin.similarity |
| Intent resolution | Scale | IntentResolutionEvaluator |
builtin.intent_resolution |
| Tool call accuracy | Scale | ToolCallAccuracyEvaluator |
builtin.tool_call_accuracy |
| Task adherence | Pass or fail | TaskAdherenceEvaluator |
builtin.task_adherence |
| Task completion | Pass or fail | No public class | builtin.task_completion |
| Tool call success | Pass or fail | No public class | builtin.tool_call_success |
| Tool input accuracy | Pass or fail | No public class | builtin.tool_input_accuracy |
| Tool output utilization | Pass or fail | No public class | builtin.tool_output_utilization |
| Tool selection | Pass or fail | No public class | builtin.tool_selection |
You can use the evaluators that have no public SDK class in cloud evaluations. For other evaluators that need a judge, use a generative model.
Prerequisites
An Azure subscription with a valid payment method. If you don't have an Azure subscription, create a paid Azure account.
Permission to create model deployments. The Cognitive Services Contributor role lets you deploy models.
To run evaluations with Microsoft Entra ID, the Foundry User role on the Foundry resource. A role assignment on the project alone doesn't authorize calls to the model. For more information, see Permissions for evaluation workflows.
Important
The Foundry RBAC roles were recently renamed. Foundry User, Foundry Owner, Foundry Account Owner, and Foundry Project Manager were previously named Azure AI User, Azure AI Owner, Azure AI Account Owner, and Azure AI Project Manager. You might still see the previous names in some places while the rename rolls out. The role IDs and core permissions are unchanged by the rename.
Python 3.10 or later for the SDK examples.
Deploy Microsoft-Decision-1
Deploy the model in the Foundry resource that contains your project.
- Open the Foundry portal, and go to your project.
- Select Model catalog.
- Search for and select Microsoft-Decision-1.
- Select Deploy, configure the deployment, and then select Deploy.
- On the deployment details page, copy the resource endpoint and the deployment name.
Alternatively, deploy the model by using the Azure CLI. Replace <ACCOUNT_NAME>, <RESOURCE_GROUP>, and <DEPLOYMENT_NAME> with your values.
az cognitiveservices account deployment create \
--name <ACCOUNT_NAME> \
--resource-group <RESOURCE_GROUP> \
--deployment-name <DEPLOYMENT_NAME> \
--model-name "Microsoft-Decision-1" \
--model-format Microsoft \
--model-version "1" \
--sku-name GlobalStandard \
--sku-capacity 1
Reference: az cognitiveservices account deployment create
The example uses the GlobalStandard deployment type. To keep processing in the data zone closest to your application, use DataZoneStandard where it's available. For more information, see Deployment types for Microsoft Foundry Models and Region availability for Foundry Models sold by Azure.
Run an evaluation with the SDK
Use the Azure AI Evaluation SDK to run evaluations on your own computer. Decision model judges need version 1.18.11 or later.
Install the packages:
pip install "azure-ai-evaluation>=1.18.11" azure-identitySet environment variables for the endpoint and deployment name that you copied. Use the resource endpoint, such as
https://<resource-name>.services.ai.azure.com, not the project endpoint.export AZURE_ENDPOINT="<your-foundry-resource-endpoint>" export DECISION_DEPLOYMENT_NAME="<your-deployment-name>"Sign in with the Azure CLI so that
DefaultAzureCredentialcan use your identity:az loginReference: az login
Create a model configuration with
"type": "decision", and pass it to an evaluator asmodel_config. This configuration sends requests to the Microsoft-Decision-1 decision API on your Foundry resource.import os from azure.identity import DefaultAzureCredential from azure.ai.evaluation import CoherenceEvaluator credential = DefaultAzureCredential() # Use your Microsoft-Decision-1 deployment as the judge decision_judge = { "type": "decision", "azure_endpoint": os.environ["AZURE_ENDPOINT"], "azure_deployment": os.environ["DECISION_DEPLOYMENT_NAME"], } coherence = CoherenceEvaluator(model_config=decision_judge, credential=credential) # Score one response result = coherence( query="How do I reset my password?", response="Open Settings, select Security, then select Reset password and follow the email link.", ) print(result["coherence"], result["coherence_result"]) print(result["coherence_reason"])The output looks similar to the following example. Your values might differ.
4.0 pass Selected coherence level 4 from the highest-probability level. No model rationale is available.To score a dataset, pass the evaluator to
evaluate(). The dataset is a JSONL file in which each line hasqueryandresponsefields.from azure.ai.evaluation import evaluate results = evaluate( data="data.jsonl", evaluators={"coherence": coherence}, ) print(results["metrics"])
A decision model configuration needs a credential or an API key. The SDK doesn't create a default credential for decision models, so pass credential as in the preceding example. To use an API key instead, add an api_key field to the configuration. When the configuration has an API key, the evaluator uses the key.
Set the probability threshold
Pass or fail evaluators accept a probability_threshold from 0 to 1. A response passes when the model's probability is at or above the threshold. The default is 0.75.
from azure.ai.evaluation import TaskAdherenceEvaluator
task_adherence = TaskAdherenceEvaluator(
model_config=decision_judge,
credential=credential,
probability_threshold=0.6,
)
A higher threshold fails more borderline responses, and a lower threshold passes more of them. Choose a threshold based on which mistake costs you more, and check it on examples that you labeled yourself. Scale evaluators don't use probability_threshold. They use the evaluator's threshold setting, the same as with a generative judge.
Run a cloud evaluation
In a cloud evaluation, set the model initialization parameter of an evaluator to your Microsoft-Decision-1 deployment name. The evaluation service uses the deployment as a decision model judge. Each testing criterion has its own model, so you can use a decision model for some evaluators and a generative model for others in the same evaluation.
Cloud evaluations use the Microsoft Foundry SDK, azure-ai-projects version 2.2.0 or later. To install it and create a project client, see Introduction to cloud evaluation. Then define testing criteria that use your Microsoft-Decision-1 deployment:
import os
from azure.ai.projects.models import TestingCriterionAzureAIEvaluator
decision_deployment_name = os.environ["DECISION_DEPLOYMENT_NAME"]
testing_criteria = [
TestingCriterionAzureAIEvaluator(
type="azure_ai_evaluator",
name="coherence",
evaluator_name="builtin.coherence",
initialization_parameters={"model": decision_deployment_name},
data_mapping={
"query": "{{item.query}}",
"response": "{{item.response}}",
},
),
]
Use these testing criteria in the workflow in Evaluate query-response datasets. The rest of the workflow is the same. For the inputs that each evaluator needs, see Built-in evaluators.
Run an evaluation in the Foundry portal
When you create an evaluation in the Foundry portal, select your Microsoft-Decision-1 deployment as the judge model, and then select evaluators from Supported evaluators. For the full workflow, see Run evaluations from the portal.
Review the results
In the SDK, results from a decision model judge have the same fields as results from other judges, plus details from the decision model. The following example shows part of a coherence result:
{
"coherence": 4.0,
"coherence_result": "pass",
"coherence_reason": "Selected coherence level 4 from the highest-probability level. No model rationale is available.",
"coherence_threshold": 3,
"coherence_properties": {
"decision_model": {
"deployment": "<your-deployment-name>",
"probabilities": {"0": 0.002, "1": 0.023, "2": 0.063, "3": 0.863, "4": 0.049}
}
}
}
| Field | Description |
|---|---|
<evaluator> |
The score. Scale evaluators score from 1 to 5. Pass or fail evaluators score 1 or 0. |
<evaluator>_result |
pass, fail, or not_applicable. |
<evaluator>_reason |
A standard summary of how the score was chosen. The model doesn't write it. |
<evaluator>_properties |
Under decision_model, the deployment and the probabilities behind the score. For scale evaluators, probabilities lists each level, starting with key 0 for level 1. For pass or fail evaluators, it lists true and false, and probability_threshold shows the threshold. |
<evaluator>_prompt_tokens, <evaluator>_total_tokens |
The tokens that the call used. |
Use the probabilities to find borderline results to review. For pass or fail evaluators, look for probabilities close to the probability threshold. For scale evaluators, look for results where the probability is split between levels on both sides of the evaluator's threshold. Send these results for human review, or re-evaluate them with a generative judge.
If an input is missing what the evaluator needs, the result is not_applicable, and the reason says that the evaluation was skipped.
To view the results of a cloud evaluation in the Foundry portal, see View evaluation results in the portal.
Limitations
- No written explanations. The model doesn't explain its answers. When you need to know why a response failed, use a generative judge.
- Input size. Each input, together with the evaluator's question, must fit in the model's context window of 32,768 tokens. A longer input fails with an error that reports
context_window_exceeded. Shorten long conversations or tool outputs, or use a generative judge for those inputs. - Text and JSON only. For images or audio, use an evaluator and a judge model that support that input type.
- Not fully deterministic. A small number of results can change when you run the same evaluation again.
- SDK version. Use azure-ai-evaluation 1.18.11 or later. Earlier versions could send an evaluator's inputs to the model in a different order from one run to the next, which can change results.
Best practices
- Compare before you switch. Run the decision model judge next to your current judge on a sample of your data, and review the results where they disagree.
- Tune thresholds on your data. Choose the
probability_thresholdfor each pass or fail evaluator by using labeled examples from your scenario and by deciding which mistake costs you more. - Keep inputs short. Map only the fields that the evaluator needs, so that inputs stay within the context window.
- Combine judges. Use the decision model judge on every response, and re-evaluate the failures with a generative judge when you need a written explanation.
Related content
- Deploy and use Microsoft-Decision-1.1 in Microsoft Foundry (not supported as an evaluation judge)
- Built-in evaluators
- Introduction to cloud evaluation
- Run evaluations from the portal