Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Important
Items marked preview in this article are currently in preview. This preview is provided without a service-level agreement, and Microsoft doesn't recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.
For custom scoring logic that you want to host and control, use an endpoint-based evaluator. It can retrieve external sources during scoring, reuse an evaluation service or scoring API that your organization already uses, and run simple or complex custom logic. For custom LLM-as-judge scoring, use a rubric evaluator. For existing code-based or prompt-based custom evaluators, see Other custom evaluators.
An endpoint-based evaluator delegates scoring to an external HTTP endpoint that you own and operate. The evaluation service calls your endpoint for each item and passes the available evaluator data as a JSON payload. Your endpoint processes the data and returns a JSON response with evaluation results. Because the logic runs on infrastructure you control, it isn't subject to the code sandbox limits that apply to code-based evaluators.
Register endpoint-based evaluators by using an SDK, the REST API, or the Foundry portal. This article shows the Python SDK and portal experiences.
When to use
Use an endpoint-based evaluator for custom scoring logic that you want to develop, test, deploy, and operate on your own infrastructure. The logic can range from a simple rule-based check to a complex evaluation pipeline. Common scenarios include:
- Simple or complex scoring logic that you want to version and deploy independently.
- Scoring that depends on external data sources, internal services, or databases that the evaluator must access at runtime.
- Integration with an evaluation service or scoring API that your organization already owns or uses.
How it works
- You deploy an HTTP endpoint that accepts POST requests with evaluation data.
- Using an SDK, the REST API, or the Foundry portal, you create a connection in your Foundry project that stores the endpoint URL and authentication credentials.
- Using an SDK, the REST API, or the Foundry portal, you register an
endpoint-based evaluator that references the connection. Set the
evaluator's
schema_versionto0.0.2to use the recommended request and response contract. - When an evaluation runs, the service resolves the connection, calls your endpoint with the input data, and records the response as the evaluation result.
Before you start
Complete the following setup before you build the sample endpoint and register an evaluator.
Azure Functions walkthrough
Install a supported Python version and Azure Functions Core Tools. Use the same Python major and minor version locally and in your deployed function app. To deploy the sample, you also need a Python function app in Azure. If you don't have one, complete the Azure Functions command-line quickstart.
SDK examples
The SDK examples in this article assume you've created a project client.
Install the SDK and set up your client once, then reuse endpoint,
credential, project_client, and client across the examples:
pip install "azure-ai-projects>=2.11.0" azure-identity \
azure-mgmt-cognitiveservices openai
import os
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
# Foundry project endpoint
# Example:
# https://<account_name>.services.ai.azure.com/api/projects/<project_name>
endpoint = os.environ["FOUNDRY_PROJECT_ENDPOINT"]
credential = DefaultAzureCredential()
# Create the project client
project_client = AIProjectClient(endpoint=endpoint, credential=credential)
# Get the OpenAI client for evaluation API
client = project_client.get_openai_client()
Note
Registering an endpoint-based evaluator also requires the
azure-mgmt-cognitiveservices package, included in the command above. See
Register the evaluator.
Build an endpoint with Azure Functions
Your evaluation endpoint can run on any HTTP service that accepts POST requests and returns JSON. Use the hosting platform that fits your existing development and operational practices. This article uses Azure Functions as the worked example.
The endpoint must respond within the evaluation service timeout (120 seconds) and return a valid JSON response for each request.
The following sample creates a Python function in Azure Functions that checks
whether the assistant ends the scored messages with a text answer. For
turn-level evaluation, it scores only the latest turn. For conversation-level
evaluation, it scores the full conversation. It also counts tool calls and
returns a per-dimension breakdown. It validates the API key that Foundry sends
in the api-key header. See
What the endpoint receives and
Understand endpoint output for field
descriptions.
This sample uses an API key to minimize setup and keep the first run focused on the evaluator contract. For production, prefer Microsoft Entra ID authentication. Microsoft Entra ID requires additional endpoint authentication, token audience, and identity authorization configuration.
Create the function project
Create a project folder and Python virtual environment:
mkdir custom-evaluator-function
cd custom-evaluator-function
python -m venv .venv
Activate the virtual environment with the command for your shell:
| Shell | Command |
|---|---|
| Bash | source .venv/bin/activate |
| Command Prompt | .venv\Scripts\activate |
| PowerShell | .venv\Scripts\Activate.ps1 |
Then initialize the Python v2 project, create the HTTP trigger, and install its dependencies:
func init --worker-runtime python --model V2
func new --name evaluate --template "HTTP trigger" --authlevel anonymous
python -m pip install -r requirements.txt
The trigger uses anonymous Azure Functions authorization because the sample
handler validates the api-key header that Foundry sends. Don't deploy an
anonymous trigger without the API key validation or another authentication
mechanism.
Confirm that requirements.txt contains the Azure Functions package:
azure-functions
Implement the evaluator
The sample implements schema version 0.0.2, the recommended endpoint
contract. It expects the evaluator metric to define a threshold. Foundry
includes the registered threshold in each endpoint request. The
registration example sets it to 3.
Replace the generated contents of function_app.py with the following code:
import hmac
import json
import os
import azure.functions as func
app = func.FunctionApp(http_auth_level=func.AuthLevel.ANONYMOUS)
def json_response(body: dict, status_code: int = 200) -> func.HttpResponse:
"""Return a JSON HTTP response."""
return func.HttpResponse(
json.dumps(body),
mimetype="application/json",
status_code=status_code,
)
def error_response(code: str, message: str, status_code: int) -> func.HttpResponse:
"""Return an endpoint error that Foundry records for the item."""
return json_response(
{"error": {"code": code, "message": message}},
status_code,
)
def message_text(message: dict) -> str:
"""Join the text parts of one message."""
parts = message.get("content", [])
return " ".join(
part.get("text", "")
for part in parts
if isinstance(part, dict) and part.get("type") == "text"
)
@app.route(route="evaluate", methods=["POST"])
def evaluate(req: func.HttpRequest) -> func.HttpResponse:
"""Score one Foundry evaluation item."""
expected_key = os.environ.get("EVALUATOR_API_KEY")
supplied_key = req.headers.get("api-key")
if not expected_key or not supplied_key:
return error_response("401", "Unauthorized", 401)
if not hmac.compare_digest(expected_key, supplied_key):
return error_response("401", "Unauthorized", 401)
try:
payload = req.get_json()
except ValueError:
return error_response("400", "Request body must be valid JSON", 400)
if not isinstance(payload, dict) or payload.get("schema_version") != "0.0.2":
return error_response("400", "Expected schema_version 0.0.2", 400)
threshold = payload.get("threshold")
if isinstance(threshold, bool) or not isinstance(threshold, (int, float)):
return error_response("400", "threshold must be a number", 400)
messages = payload.get("messages", [])
target = payload.get("evaluation_target", {})
if target.get("type") == "last_turn":
messages = messages[target["response_start_index"]:]
assistant_messages = [
message for message in messages if message.get("role") == "assistant"
]
if not assistant_messages:
return json_response(
{"status": "skipped", "reason": "No assistant messages to score."}
)
tool_calls = sum(
1
for message in assistant_messages
for part in message.get("content", [])
if isinstance(part, dict) and part.get("type") == "tool_call"
)
# A tool-call preamble isn't a final answer, even if it includes text.
final = assistant_messages[-1]
final_calls_tool = any(
isinstance(part, dict) and part.get("type") == "tool_call"
for part in final.get("content", [])
)
has_answer = bool(message_text(final).strip()) and not final_calls_tool
score = 5 if has_answer else 1
reason = (
"The assistant ends with a text answer."
if has_answer
else "The assistant doesn't end with a text answer."
)
return json_response(
{
"status": "completed",
"score": score,
"passed": score >= threshold,
"reason": reason,
"properties": {
"tool_call_count": tool_calls,
"dimension_scores": [
{"id": "final_answer", "score": score, "reason": reason},
],
},
}
)
Reference: Azure Functions Python developer reference
The function reads only the top-level messages and evaluation_target
fields. Foundry builds messages in the same format for every scenario, so
the same code scores dataset rows, agent targets, traces, and simulations.
Test the function locally
Set EVALUATOR_API_KEY to a randomly generated secret in local.settings.json.
Add it to the generated Values object:
{
"IsEncrypted": false,
"Values": {
"AzureWebJobsStorage": "UseDevelopmentStorage=true",
"FUNCTIONS_WORKER_RUNTIME": "python",
"EVALUATOR_API_KEY": "<API_KEY>"
}
}
Don't commit this file or the secret. The Azure Functions local settings guidance explains how to store local settings.
Start the function locally:
func start
In another terminal, send a sample evaluator request. Replace <API_KEY> with
the value of EVALUATOR_API_KEY:
curl -X POST http://localhost:7071/api/evaluate \
-H "Content-Type: application/json" \
-H "api-key: <API_KEY>" \
-d '{
"schema_version": "0.0.2",
"evaluator_name": "my-endpoint-evaluator",
"evaluator_version": "1",
"evaluation_target": {"type": "last_turn", "response_start_index": 1},
"context": {},
"messages": [
{
"role": "user",
"content": [{"type": "text", "text": "I forgot my password."}]
},
{
"role": "assistant",
"content": [
{
"type": "text",
"text": "Use the password-reset link on the sign-in page."
}
]
}
],
"agent_definition": {"tool_definitions": []},
"metadata": {"run_type": "dataset"},
"threshold": 3
}'
The function returns a passing score:
{
"status": "completed",
"score": 5,
"passed": true,
"reason": "The assistant ends with a text answer.",
"properties": {
"tool_call_count": 0,
"dimension_scores": [
{
"id": "final_answer",
"score": 5,
"reason": "The assistant ends with a text answer."
}
]
}
}
Deploy the function
Publish the project to your existing function app:
func azure functionapp publish <FUNCTION_APP_NAME>
In the deployed function app, set EVALUATOR_API_KEY as an application
setting. For example:
az functionapp config appsettings set \
--name <FUNCTION_APP_NAME> \
--resource-group <RESOURCE_GROUP> \
--settings EVALUATOR_API_KEY=<API_KEY>
Use the same secret for ENDPOINT_API_KEY when you
create the project connection.
Set ENDPOINT_URL to
https://<FUNCTION_APP_NAME>.azurewebsites.net/api/evaluate.
What the endpoint receives
The evaluation service sends one POST request for each evaluation item. With
schema version 0.0.2, every request uses the same shape, whether the item
comes from a dataset, an agent or model target, a trace, or a simulation.
Request envelope
| Field | Type | Description |
|---|---|---|
schema_version |
string |
Always "0.0.2". |
evaluator_name |
string |
The registered name of the evaluator being run. |
evaluator_version |
string |
The version of the evaluator definition. |
evaluation_target |
object |
What to score. See Evaluation target. |
messages |
array |
The conversation to evaluate, normalized to one message format. See Message format. |
agent_definition |
object |
Contains tool_definitions, the tools available to the agent. The array is empty when no tools apply. |
context |
object |
The original input row, unchanged, for dataset, model target, and agent target runs. Use it to read extra columns such as ground_truth. Trace, simulation, and stored response runs don't have an input row, so context doesn't contain one. |
metadata |
object |
Contains run_type and, when available, identifiers such as response_id, agent_name, agent_version, conversation_id, trace_id, and span_id. |
threshold |
number |
Registered pass or fail threshold. Present when the evaluator metric defines one. |
context and messages can contain the same conversation in two forms.
context holds your row as you submitted it. messages holds the version
that Foundry normalized for scoring. Score from messages.
metadata.run_type is one of dataset, model_target, agent_target,
agent_trace, agent_simulation, or stored_response.
Schema version 0.0.2 ignores data_mapping. Foundry builds messages from
your data automatically:
- A
messagescolumn is used as the conversation. queryandresponsecolumns become a user message and an assistant message.- For model and agent targets, the generated output, including tool calls and tool results, is appended after the input.
Evaluation target
evaluation_target.type tells your endpoint which part of messages to
score:
| Type | When | Fields |
|---|---|---|
last_turn |
Turn-level evaluation | response_start_index: the index in messages where the response being scored starts. Earlier messages are conversation history. |
conversation |
Conversation-level evaluation | None. Score all of messages. |
For example, if response_start_index is 1, messages[0] is the user's
question and messages[1:] is the response to score.
Message format
Each message has a role of system, user, assistant, or tool, and
content is always an array of parts:
| Part type | Fields | Appears in |
|---|---|---|
text |
text |
Any role. |
tool_call |
tool_call_id, name, arguments |
assistant messages. |
tool_result |
tool_result |
tool messages. The message also has a top-level tool_call_id. |
Request examples
The following examples are compact versions of real requests.
The dataset row contains query and response columns. Foundry converts them
to messages and passes the original row in context.
{
"schema_version": "0.0.2",
"evaluator_name": "my-endpoint-evaluator",
"evaluator_version": "1",
"evaluation_target": {"type": "last_turn", "response_start_index": 1},
"context": {
"query": "What is 2 plus 2?",
"response": "2 plus 2 equals 4.",
"ground_truth": "4"
},
"messages": [
{
"role": "user",
"content": [{"type": "text", "text": "What is 2 plus 2?"}]
},
{
"role": "assistant",
"content": [{"type": "text", "text": "2 plus 2 equals 4."}]
}
],
"agent_definition": {"tool_definitions": []},
"metadata": {"run_type": "dataset"},
"threshold": 3
}
Understand endpoint output
For an HTTP 200 response, return a JSON object with the following fields:
| Field | Type | Required | Description |
|---|---|---|---|
status |
string |
Yes | "completed", "skipped", or "error". |
score |
number |
When completed |
Must be within the registered metric range. Must be an integer for ordinal metrics. Omit for skipped and error. |
passed |
boolean |
When completed |
Whether the item passes. Omit for skipped and error. |
reason |
string |
No | Explanation for the score, skip, or error. |
label |
string |
No | A short result label, such as "pass" or "needs_review". |
properties |
object |
No | Evaluator-specific data. Can include dimension_scores. |
usage |
object |
No | Token usage, with optional prompt_tokens, completion_tokens, and total_tokens as nonnegative integers. |
model |
string |
No | The model your endpoint used, if any. |
Foundry rejects responses that include any other top-level field. A rejected response is recorded as an error for that item.
Foundry doesn't derive passed from score. Use threshold from the
request to calculate passed. Set the threshold on EvaluatorMetric when you
register the evaluator.
Dimension scores
To report a breakdown, add properties.dimension_scores, an array of
dimensions:
| Field | Type | Description |
|---|---|---|
id |
string |
Unique, nonempty dimension identifier. |
score |
integer or null |
A score from 1 through 5, or null when the dimension isn't scored. |
reason |
string |
Explanation for the dimension score. |
A completed result needs at least one scored dimension. A skipped result
must use null for every dimension score. Omit dimension_scores from an
error result.
Completed response
{
"status": "completed",
"score": 4,
"passed": true,
"reason": "The answer is correct and cites the policy.",
"properties": {
"dimension_scores": [
{
"id": "accuracy",
"score": 5,
"reason": "The answer matches the policy."
},
{
"id": "citation",
"score": 3,
"reason": "The policy is cited without a link."
}
]
}
}
Skipped response
Return skipped when the item doesn't apply to your evaluator:
{
"status": "skipped",
"reason": "No assistant messages to score."
}
Error response
If the endpoint can't produce a result, return a non-200 HTTP status with an
error object. Foundry records the error for that evaluation item:
{
"error": {
"code": "500",
"message": "Model inference failed"
}
}
Foundry also records an error for an item, without calling your endpoint,
when it can't build a valid request for that item. For example, a dataset
row whose messages value isn't a valid message list gets an error. Other
items in the run are still evaluated.
Schema version 0.0.1
If an endpoint-based evaluator doesn't set schema_version, Foundry uses
schema version 0.0.1. It remains supported, but use 0.0.2 for new
evaluators. Version 0.0.2 sends one normalized message format for every
scenario and validates responses more strictly.
The main differences:
| Area | 0.0.1 |
0.0.2 |
|---|---|---|
| Input fields | Scenario-specific fields under data.item, such as query, output_text, output_items, or messages, plus data.sample |
Normalized messages, original row in context |
| Level | evaluation_level: "turn" or "conversation" |
evaluation_target with last_turn and response_start_index, or conversation |
| Tools and metadata | Mixed into data.item |
agent_definition.tool_definitions and metadata |
data_mapping |
Supported | Ignored |
| Response status | "Completed", "Skipped", or "Error" |
"completed", "skipped", or "error" |
| Response fields | score, passed, status, reason, properties, schema_version |
Strict field list, plus label, usage, model, and dimension_scores |
Configure authentication
Endpoint-based evaluators support two authentication methods through project connections:
| Method | How it works | Best for |
|---|---|---|
| API key | The service passes the key in a request header when calling your endpoint. | Endpoints that validate an api-key header, including the Azure Functions sample in this article. |
| Microsoft Entra ID | The service acquires a managed identity token scoped to your app registration and passes it as a bearer token in the request's Authorization header. |
Azure Functions with role-based access control, Azure Functions with Easy Auth. |
Register the evaluator
Creating an endpoint-based evaluator takes two steps: create a project connection that stores your endpoint URL and credentials, then register an evaluator that references that connection. The tabs show the Python SDK and Foundry portal. For the REST API, see Create an evaluator version.
In addition to FOUNDRY_PROJECT_ENDPOINT, set AZURE_SUBSCRIPTION_ID,
AZURE_RESOURCE_GROUP, ENDPOINT_URL, and ENDPOINT_API_KEY as environment
variables. The following example creates an API key connection and registers an
evaluator against it:
from azure.ai.projects.models import (
EndpointBasedEvaluatorDefinition,
EndpointEvaluatorSchemaVersion,
EvaluationLevel,
EvaluatorCategory,
EvaluatorMetric,
EvaluatorType,
EvaluatorVersion,
)
from azure.mgmt.cognitiveservices import CognitiveServicesManagementClient
from azure.mgmt.cognitiveservices.models import (
ApiKeyAuthConnectionProperties,
ConnectionApiKey,
ConnectionPropertiesV2BasicResource,
)
import os
from urllib.parse import urlparse
subscription_id = os.environ["AZURE_SUBSCRIPTION_ID"]
resource_group = os.environ["AZURE_RESOURCE_GROUP"]
endpoint_url = os.environ["ENDPOINT_URL"]
endpoint_api_key = os.environ["ENDPOINT_API_KEY"]
hostname = urlparse(endpoint).hostname
if hostname is None:
raise ValueError(f"Invalid Foundry project endpoint: {endpoint}")
account_name = hostname.split(".")[0]
# Create the connection that stores the endpoint URL and API key.
mgmt_client = CognitiveServicesManagementClient(
credential=credential,
subscription_id=subscription_id,
)
connection = ConnectionPropertiesV2BasicResource(
properties=ApiKeyAuthConnectionProperties(
category="ApiKey",
target=endpoint_url,
credentials=ConnectionApiKey(key=endpoint_api_key),
),
)
mgmt_client.account_connections.create(
resource_group_name=resource_group,
account_name=account_name,
connection_name="my-endpoint-connection",
connection=connection,
)
# Define the evaluator
evaluator_definition = EndpointBasedEvaluatorDefinition(
connection_name="my-endpoint-connection",
# Use the recommended endpoint contract. Without this, Foundry uses 0.0.1.
schema_version=EndpointEvaluatorSchemaVersion.V0_0_2,
metrics={
"score": EvaluatorMetric(
type="ordinal",
min_value=1,
max_value=5,
threshold=3,
),
},
)
# Register an evaluator that references the connection
endpoint_evaluator = project_client.evaluators.create_version(
name="my-endpoint-evaluator",
evaluator_version=EvaluatorVersion(
categories=[EvaluatorCategory.QUALITY],
evaluator_type=EvaluatorType.CUSTOM,
display_name="My Endpoint Evaluator",
description="Scores responses using a custom evaluation endpoint",
supported_evaluation_levels=[
EvaluationLevel.TURN,
EvaluationLevel.CONVERSATION,
],
definition=evaluator_definition,
),
# Custom evaluator definitions are in preview and require this opt-in header.
headers={"Foundry-Features": "Evaluations=V1Preview"},
)
With the REST API, set "schema_version": "0.0.2" in the definition object
and send the Foundry-Features: Evaluations=V1Preview header.
An evaluator that uses 0.0.2 must define exactly one metric of type
ordinal or continuous.
For Microsoft Entra ID authentication, use AADAuthTypeConnectionProperties instead of ApiKeyAuthConnectionProperties, and configure your endpoint to validate the incoming bearer token (for example, with Azure App Service Easy Auth). With Entra ID, Foundry sends a bearer token instead of an api-key header, so this article's function rejects the request with 401. Replace the function's API key check with bearer token validation. Remove the check only after Easy Auth or another layer validates the token and authorizes the caller.
Run an evaluation
Once your endpoint-based evaluator is registered, use it in an evaluation through an SDK, the REST API, or the Foundry portal. The tabs show the Python SDK and portal experiences.
This example evaluates one dataset row with query, response, document,
and ground_truth fields. Foundry converts query and response to
messages and passes the full row in context, as shown in the Dataset turn
example in What the endpoint receives. No
data_mapping is required. The following code defines input_item,
testing_criteria, and data_source_config, then creates one eval and
one run.
from openai.types.eval_create_params import DataSourceConfigCustom
from openai.types.evals.create_eval_jsonl_run_data_source_param import (
CreateEvalJSONLRunDataSourceParam,
SourceFileContent,
SourceFileContentContent,
)
input_item = {
"query": "I can't sign in to my account.",
"response": "Use the password-reset link on the sign-in page.",
"document": "Account sign-in help article.",
"ground_truth": "Use the password-reset link.",
}
testing_criteria = [
{
"type": "azure_ai_evaluator",
"name": "endpoint_eval",
"evaluator_name": "my-endpoint-evaluator",
},
]
data_source_config = DataSourceConfigCustom(
type="custom",
item_schema={
"type": "object",
"properties": {
"query": {"type": "string"},
"response": {"type": "string"},
"document": {"type": "string"},
"ground_truth": {"type": "string"},
},
"required": ["query", "response"],
},
)
eval_object = client.evals.create(
name="endpoint-evaluator-test",
data_source_config=data_source_config,
testing_criteria=testing_criteria,
)
eval_run = client.evals.runs.create(
eval_id=eval_object.id,
name="endpoint-evaluator-run",
data_source=CreateEvalJSONLRunDataSourceParam(
type="jsonl",
source=SourceFileContent(
type="file_content",
content=[
SourceFileContentContent(
item=input_item,
),
],
),
),
extra_body={"evaluation_level": "turn"},
)
print(f"Run started: {eval_run.id}")
For a conversation-level run, each item needs the whole conversation, not
just query and response. Provide a messages array in each item, add
messages to item_schema, and set evaluation_level to "conversation"
in extra_body. Foundry then sends evaluation_target with type
conversation. For the data format, see
Evaluate conversation datasets.
To evaluate an agent or model target, or traces, see
Evaluate models and agents in the cloud
and Run evaluations from the SDK.
The endpoint receives the same request shape in every case.
For polling, results, cancellation, and troubleshooting, see Get cloud evaluation results. For the complete run and cleanup lifecycle, see the endpoint evaluator Python sample.
Clean up
When you're done testing, delete the resources you created for this article so they don't continue to incur cost or stay reachable: the evaluation, the evaluator version, the connection that stores your endpoint credentials, and the hosted endpoint itself if you no longer need it. For the complete cleanup code, see the endpoint evaluator Python sample.