Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Important
Items marked preview in this article are currently in preview. This preview is provided without a service-level agreement, and Microsoft doesn't recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.
Use a rubric evaluator when you need weighted, multidimensional criteria or want Foundry to generate a starting rubric for you. A prompt-based evaluator is a viable, simpler choice when you only need a single, focused LLM judge prompt, or when you're building on an existing prompt evaluator. Use an endpoint-based evaluator for new custom code that requires your own hosted infrastructure, external data sources, or an existing evaluation service. A code-based evaluator fits an existing scoring script or a scenario constrained to a self-contained, sandboxed Python function that doesn't need network access.
Custom evaluators define domain-specific quality metrics that go beyond the built-in evaluator catalog. Prompt-based and code-based evaluators run inside the Foundry evaluation service, so you don't need to host your own endpoint. Both run in a sandboxed environment: prompt-based evaluators use a judge prompt that an LLM evaluates for each item, and code-based evaluators execute a Python scoring function with restricted packages and no network access.
Create and manage prompt-based and code-based evaluators from the Foundry portal or the SDK. The portal creates both types without writing SDK code. Use the SDK when you want to manage evaluators programmatically or integrate creation into your existing CI/CD workflows.
Prerequisites for SDK examples
The SDK examples in this article assume you've created a project client.
Install the SDK and set up your client once, then reuse project_client and
client across the examples for each evaluator type:
pip install "azure-ai-projects>=2.8.0" azure-identity
import os
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
# Foundry project endpoint
# Example:
# https://<account_name>.services.ai.azure.com/api/projects/<project_name>
endpoint = os.environ["FOUNDRY_PROJECT_ENDPOINT"]
# Model deployment name (required for prompt-based and code-based evaluators)
# Example: gpt-5-mini
model_deployment_name = os.environ["FOUNDRY_MODEL_NAME"]
credential = DefaultAzureCredential()
# Create the project client
project_client = AIProjectClient(endpoint=endpoint, credential=credential)
# Get the OpenAI client for evaluation API
client = project_client.get_openai_client()
Prompt-based evaluators
Tip
Use a rubric evaluator for weighted, multidimensional criteria or when you want Foundry to generate a starting rubric from your agent's context. Use a prompt-based evaluator for one focused judge prompt or to maintain an existing prompt evaluator.
A prompt-based evaluator uses a judge prompt template that an LLM evaluates for each item. Template variables use double curly braces (for example, {{query}}) and map to your input data fields.
Prompt-based evaluators support three scoring methods:
- Ordinal: Integer scores on a discrete scale you define (for example, 1–5). Higher is better.
- Continuous: Float scores for fine-grained measurement on a range you define (for example, 0.0–1.0). Higher is better.
- Binary (true/false): Boolean result for threshold-based checks.
The evaluator must return a JSON object with result and reason. The type of result matches your scoring method: an integer for ordinal, a float for continuous, or a boolean for binary.
The following example prompt uses ordinal scoring (1–5) to evaluate the friendliness of a response:
Friendliness assesses the warmth and approachability of the response.
Rate the friendliness of the response between one and five.
Use the following scale:
1 - Unfriendly or hostile
2 - Mostly unfriendly
3 - Neutral
4 - Mostly friendly
5 - Very friendly
Assign a rating based on the tone and demeanor of the response.
Response:
{{response}}
Output Format (JSON):
{
"result": <integer from 1 to 5>,
"reason": "<brief explanation for the score>"
}
Runtime parameters
Both deployment_name and threshold are required as initialization parameters when you create a prompt-based evaluator.
Supply the parameters, along with the input field mapping, when you add the evaluator to an evaluation:
testing_criteria = [
{
"type": "azure_ai_evaluator",
"name": "friendliness_evaluator",
"evaluator_name": "friendliness_evaluator",
"data_mapping": {
"response": "{{item.response}}",
},
"initialization_parameters": {
"deployment_name": model_deployment_name,
"threshold": 3,
},
},
]
Create a prompt-based evaluator with the SDK
Pass the judge prompt in the prompt_text field. Define the data_schema to declare the input fields your prompt expects, and the metrics to describe the scoring method and range. The init_parameters declare the model deployment and threshold the evaluator needs at runtime.
from azure.ai.projects.models import EvaluatorCategory, EvaluatorDefinitionType
prompt_evaluator = project_client.evaluators.create_version(
name="friendliness_evaluator",
evaluator_version={
"name": "friendliness_evaluator",
"categories": [EvaluatorCategory.QUALITY],
"display_name": "Friendliness Evaluator",
"description": "Evaluates the warmth and approachability of a response",
"definition": {
"type": EvaluatorDefinitionType.PROMPT,
"prompt_text": (
"Friendliness assesses the warmth and approachability "
"of the response.\n"
"Rate the friendliness of the response between one and five.\n"
"Use the following scale:\n\n"
"1 - Unfriendly or hostile\n"
"2 - Mostly unfriendly\n"
"3 - Neutral\n"
"4 - Mostly friendly\n"
"5 - Very friendly\n\n"
"Assign a rating based on the tone and demeanor "
"of the response.\n\n"
"Response:\n{{response}}\n\n"
"Output Format (JSON):\n"
'{\n "result": <integer from 1 to 5>,\n'
' "reason": "<brief explanation for the score>"\n}\n'
),
"init_parameters": {
"type": "object",
"properties": {
"deployment_name": {"type": "string"},
"threshold": {"type": "number"},
},
"required": ["deployment_name", "threshold"],
},
"data_schema": {
"type": "object",
"properties": {
"response": {"type": "string"},
},
"required": ["response"],
},
"metrics": {
"custom_prompt": {
"type": "ordinal",
"desirable_direction": "increase",
"min_value": 1,
"max_value": 5,
}
},
},
},
# Custom evaluator definitions are in preview and require this opt-in header.
headers={"Foundry-Features": "Evaluations=V1Preview"},
)
For a complete example, see the prompt-based evaluator Python SDK sample.
Code-based evaluators
A code-based evaluator is a Python function named grade that receives two dict parameters (sample and item) and returns a float score between 0.0 and 1.0 (higher is better). In practice, all data is accessed through item:
- Dataset evaluation: Input fields like
responseorground_truthcan be retrieved in the Python code likeitem.get("response")oritem.get("ground_truth"). - Model or agent target evaluation: To fetch generated response text, use
item.get("sample", {}).get("output_text").
Note
Currently, generated response text from a model or agent target is accessed through item.get("sample", {}).get("output_text"). This access pattern is subject to change in a future API update.
The following example scores responses based on length, preferring responses between 50 and 500 characters:
def grade(sample: dict, item: dict) -> float:
"""Score based on response length (prefer 50-500 chars)."""
# For dataset evaluation, access fields directly from item:
response = item.get("response", "")
# For model/agent target evaluation, use item.get("sample") instead:
# response = item.get("sample", {}).get("output_text", "")
if not response:
return 0.0
length = len(response)
if length < 50:
return 0.2
if length > 500:
return 0.5
return 1.0
Note
If the grade() function raises an exception or times out, the service
records that item's result as 0.0 and marks it as an error in the evaluation
report. Catch expected exceptions only when you can return a meaningful score.
Supported packages and limits
Code-based evaluators run in a sandboxed Python environment with the following constraints:
- Code size must be less than 256 KB.
- Execution is limited to 2 minutes per grading call.
- No network access is available at runtime.
- Memory limit is 2 GB, disk limit is 1 GB, and CPU is limited to 2 cores.
The following third-party packages are available:
| Package | Version |
|---|---|
numpy |
2.2.4 |
scipy |
1.15.2 |
pandas |
2.2.3 |
scikit-learn |
1.6.1 |
rapidfuzz |
3.10.1 |
sympy |
1.13.3 |
jsonschema |
4.23.0 |
pydantic |
2.10.6 |
deepdiff |
8.4.2 |
nltk |
3.9.1 |
rouge-score |
0.1.2 |
pyyaml |
6.0.2 |
The NLTK corpora punkt, stopwords, wordnet, omw-1.4, and names are preloaded.
If your scoring logic needs a package that isn't on this list, needs network access, or exceeds these limits, use an endpoint-based evaluator instead.
Runtime parameters
pass_threshold and deployment_name are required as initialization parameters when you create a code-based evaluator. Even though code-based evaluators don't call an LLM, the service API schema requires deployment_name for evaluation-run orchestration. You can pass any valid model deployment name from your project.
Supply the parameters when you add the evaluator to an evaluation:
testing_criteria = [
{
"type": "azure_ai_evaluator",
"name": "response_length_scorer",
"evaluator_name": "response_length_scorer",
"initialization_parameters": {
"deployment_name": model_deployment_name,
"pass_threshold": 0.5,
},
},
]
Create a code-based evaluator with the SDK
Pass the grade() function as a string in the code_text field. Define the data_schema to declare the input fields your function expects, and the metrics to describe the score your function returns. Code-based evaluators use the continuous metric type with a range of 0.0 to 1.0.
from azure.ai.projects.models import EvaluatorCategory, EvaluatorDefinitionType
code_evaluator = project_client.evaluators.create_version(
name="response_length_scorer",
evaluator_version={
"name": "response_length_scorer",
"categories": [EvaluatorCategory.QUALITY],
"display_name": "Response Length Scorer",
"description": (
"Scores responses based on length, preferring 50-500 characters"
),
"definition": {
"type": EvaluatorDefinitionType.CODE,
"code_text": (
'def grade(sample: dict, item: dict) -> float:\n'
' """Score based on response length."""\n'
' response = item.get("response", "")\n'
' if not response:\n'
' return 0.0\n'
' length = len(response)\n'
' if length < 50:\n'
' return 0.2\n'
' if length > 500:\n'
' return 0.5\n'
' return 1.0\n'
),
"init_parameters": {
"type": "object",
"properties": {
"deployment_name": {"type": "string"},
"pass_threshold": {"type": "number"},
},
"required": ["deployment_name", "pass_threshold"],
},
"metrics": {
"result": {
"type": "continuous",
"desirable_direction": "increase",
"min_value": 0.0,
"max_value": 1.0,
}
},
"data_schema": {
"type": "object",
"required": ["item"],
"properties": {
"item": {
"type": "object",
"properties": {
"response": {"type": "string"},
},
},
},
},
},
},
# Custom evaluator definitions are in preview and require this opt-in header.
headers={"Foundry-Features": "Evaluations=V1Preview"},
)
For a complete example, see the code-based evaluator Python SDK sample.
Create a prompt-based or code-based evaluator in the portal
The Foundry portal's custom evaluator flow creates prompt-based and code-based evaluators without SDK code. For new LLM-as-judge criteria, create a rubric evaluator.
- In your Foundry project, go to Evaluation > Evaluator catalog.
- Select Custom evaluator > Create.
- Fill in the following fields:
| Field | Description |
|---|---|
| Name | A unique identifier for the evaluator (for example, response_length_scorer). |
| Display name | A human-readable name shown in the evaluator catalog. |
| Description | A short summary of what the evaluator measures. |
| Type | Prompt-based or Code-based. Determines whether you provide a judge prompt or a Python grade() function. |
| Scoring method | Prompt-based evaluators can use ordinal, continuous, or binary scoring with a custom range. Code-based evaluators use continuous (0.0–1.0). |
| Prompt or Code | For prompt-based, write a judge prompt in the prompt editor. For code-based, write a grade() function in the code editor. See the prompt-based and code-based evaluator sections earlier in this article for examples and requirements. |
Use a custom evaluator in a portal evaluation
After you create a custom evaluator, use it in an evaluation run from the portal:
- In your Foundry project, go to Evaluation and select Create.
- Follow the evaluation creation wizard. On the Criteria step, select Add evaluator.
- Choose your custom evaluator from the evaluator catalog.
- Supply the required initialization parameters. For prompt-based evaluators, provide the model deployment and threshold. For code-based evaluators, provide the model deployment and pass threshold.
- Complete the wizard and start the evaluation run.
For detailed steps on running evaluations from the portal, see Run evaluations from the portal.
Conversation-level custom logic
Code-based evaluators support only turn-level evaluation. A conversation-level
evaluation run skips them. To score an entire conversation with custom code,
use an endpoint-based evaluator (preview),
which receives the full conversation when the run uses
evaluation_level set to conversation.