Endpoint-based custom evaluators (preview)

Important

Items marked preview in this article are currently in preview. This preview is provided without a service-level agreement, and Microsoft doesn't recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.

For custom scoring logic that you want to host and control, use an endpoint-based evaluator. It can retrieve external sources during scoring, reuse an evaluation service or scoring API that your organization already uses, and run simple or complex custom logic. For custom LLM-as-judge scoring, use a rubric evaluator. For existing code-based or prompt-based custom evaluators, see Other custom evaluators.

An endpoint-based evaluator delegates scoring to an external HTTP endpoint that you own and operate. The evaluation service calls your endpoint for each item and passes the available evaluator data as a JSON payload. Your endpoint processes the data and returns a JSON response with evaluation results. Because the logic runs on infrastructure you control, it isn't subject to the code sandbox limits that apply to code-based evaluators.

Register endpoint-based evaluators by using an SDK, the REST API, or the Foundry portal. This article shows the Python SDK and portal experiences.

When to use

Use an endpoint-based evaluator for custom scoring logic that you want to develop, test, deploy, and operate on your own infrastructure. The logic can range from a simple rule-based check to a complex evaluation pipeline. Common scenarios include:

  • Simple or complex scoring logic that you want to version and deploy independently.
  • Scoring that depends on external data sources, internal services, or databases that the evaluator must access at runtime.
  • Integration with an evaluation service or scoring API that your organization already owns or uses.

How it works

  1. You deploy an HTTP endpoint that accepts POST requests with evaluation data.
  2. Using an SDK, the REST API, or the Foundry portal, you create a connection in your Foundry project that stores the endpoint URL and authentication credentials.
  3. Using an SDK, the REST API, or the Foundry portal, you register an endpoint-based evaluator that references the connection. Set the evaluator's schema_version to 0.0.2 to use the recommended request and response contract.
  4. When an evaluation runs, the service resolves the connection, calls your endpoint with the input data, and records the response as the evaluation result.

Before you start

Complete the following setup before you build the sample endpoint and register an evaluator.

Azure Functions walkthrough

Install a supported Python version and Azure Functions Core Tools. Use the same Python major and minor version locally and in your deployed function app. To deploy the sample, you also need a Python function app in Azure. If you don't have one, complete the Azure Functions command-line quickstart.

SDK examples

The SDK examples in this article assume you've created a project client. Install the SDK and set up your client once, then reuse endpoint, credential, project_client, and client across the examples:

pip install "azure-ai-projects>=2.11.0" azure-identity \
  azure-mgmt-cognitiveservices openai
import os
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient

# Foundry project endpoint
# Example:
# https://<account_name>.services.ai.azure.com/api/projects/<project_name>
endpoint = os.environ["FOUNDRY_PROJECT_ENDPOINT"]

credential = DefaultAzureCredential()

# Create the project client
project_client = AIProjectClient(endpoint=endpoint, credential=credential)

# Get the OpenAI client for evaluation API
client = project_client.get_openai_client()

Note

Registering an endpoint-based evaluator also requires the azure-mgmt-cognitiveservices package, included in the command above. See Register the evaluator.

Build an endpoint with Azure Functions

Your evaluation endpoint can run on any HTTP service that accepts POST requests and returns JSON. Use the hosting platform that fits your existing development and operational practices. This article uses Azure Functions as the worked example.

The endpoint must respond within the evaluation service timeout (120 seconds) and return a valid JSON response for each request.

The following sample creates a Python function in Azure Functions that checks whether the assistant ends the scored messages with a text answer. For turn-level evaluation, it scores only the latest turn. For conversation-level evaluation, it scores the full conversation. It also counts tool calls and returns a per-dimension breakdown. It validates the API key that Foundry sends in the api-key header. See What the endpoint receives and Understand endpoint output for field descriptions.

This sample uses an API key to minimize setup and keep the first run focused on the evaluator contract. For production, prefer Microsoft Entra ID authentication. Microsoft Entra ID requires additional endpoint authentication, token audience, and identity authorization configuration.

Create the function project

Create a project folder and Python virtual environment:

mkdir custom-evaluator-function
cd custom-evaluator-function
python -m venv .venv

Activate the virtual environment with the command for your shell:

Shell Command
Bash source .venv/bin/activate
Command Prompt .venv\Scripts\activate
PowerShell .venv\Scripts\Activate.ps1

Then initialize the Python v2 project, create the HTTP trigger, and install its dependencies:

func init --worker-runtime python --model V2
func new --name evaluate --template "HTTP trigger" --authlevel anonymous
python -m pip install -r requirements.txt

The trigger uses anonymous Azure Functions authorization because the sample handler validates the api-key header that Foundry sends. Don't deploy an anonymous trigger without the API key validation or another authentication mechanism.

Confirm that requirements.txt contains the Azure Functions package:

azure-functions

Implement the evaluator

The sample implements schema version 0.0.2, the recommended endpoint contract. It expects the evaluator metric to define a threshold. Foundry includes the registered threshold in each endpoint request. The registration example sets it to 3.

Replace the generated contents of function_app.py with the following code:

import hmac
import json
import os

import azure.functions as func

app = func.FunctionApp(http_auth_level=func.AuthLevel.ANONYMOUS)


def json_response(body: dict, status_code: int = 200) -> func.HttpResponse:
    """Return a JSON HTTP response."""
    return func.HttpResponse(
        json.dumps(body),
        mimetype="application/json",
        status_code=status_code,
    )


def error_response(code: str, message: str, status_code: int) -> func.HttpResponse:
    """Return an endpoint error that Foundry records for the item."""
    return json_response(
        {"error": {"code": code, "message": message}},
        status_code,
    )


def message_text(message: dict) -> str:
    """Join the text parts of one message."""
    parts = message.get("content", [])
    return " ".join(
        part.get("text", "")
        for part in parts
        if isinstance(part, dict) and part.get("type") == "text"
    )


@app.route(route="evaluate", methods=["POST"])
def evaluate(req: func.HttpRequest) -> func.HttpResponse:
    """Score one Foundry evaluation item."""
    expected_key = os.environ.get("EVALUATOR_API_KEY")
    supplied_key = req.headers.get("api-key")
    if not expected_key or not supplied_key:
        return error_response("401", "Unauthorized", 401)
    if not hmac.compare_digest(expected_key, supplied_key):
        return error_response("401", "Unauthorized", 401)

    try:
        payload = req.get_json()
    except ValueError:
        return error_response("400", "Request body must be valid JSON", 400)
    if not isinstance(payload, dict) or payload.get("schema_version") != "0.0.2":
        return error_response("400", "Expected schema_version 0.0.2", 400)

    threshold = payload.get("threshold")
    if isinstance(threshold, bool) or not isinstance(threshold, (int, float)):
        return error_response("400", "threshold must be a number", 400)

    messages = payload.get("messages", [])
    target = payload.get("evaluation_target", {})
    if target.get("type") == "last_turn":
        messages = messages[target["response_start_index"]:]

    assistant_messages = [
        message for message in messages if message.get("role") == "assistant"
    ]
    if not assistant_messages:
        return json_response(
            {"status": "skipped", "reason": "No assistant messages to score."}
        )

    tool_calls = sum(
        1
        for message in assistant_messages
        for part in message.get("content", [])
        if isinstance(part, dict) and part.get("type") == "tool_call"
    )
    # A tool-call preamble isn't a final answer, even if it includes text.
    final = assistant_messages[-1]
    final_calls_tool = any(
        isinstance(part, dict) and part.get("type") == "tool_call"
        for part in final.get("content", [])
    )
    has_answer = bool(message_text(final).strip()) and not final_calls_tool
    score = 5 if has_answer else 1
    reason = (
        "The assistant ends with a text answer."
        if has_answer
        else "The assistant doesn't end with a text answer."
    )
    return json_response(
        {
            "status": "completed",
            "score": score,
            "passed": score >= threshold,
            "reason": reason,
            "properties": {
                "tool_call_count": tool_calls,
                "dimension_scores": [
                    {"id": "final_answer", "score": score, "reason": reason},
                ],
            },
        }
    )

Reference: Azure Functions Python developer reference

The function reads only the top-level messages and evaluation_target fields. Foundry builds messages in the same format for every scenario, so the same code scores dataset rows, agent targets, traces, and simulations.

Test the function locally

Set EVALUATOR_API_KEY to a randomly generated secret in local.settings.json. Add it to the generated Values object:

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "UseDevelopmentStorage=true",
    "FUNCTIONS_WORKER_RUNTIME": "python",
    "EVALUATOR_API_KEY": "<API_KEY>"
  }
}

Don't commit this file or the secret. The Azure Functions local settings guidance explains how to store local settings.

Start the function locally:

func start

In another terminal, send a sample evaluator request. Replace <API_KEY> with the value of EVALUATOR_API_KEY:

curl -X POST http://localhost:7071/api/evaluate \
  -H "Content-Type: application/json" \
  -H "api-key: <API_KEY>" \
  -d '{
    "schema_version": "0.0.2",
    "evaluator_name": "my-endpoint-evaluator",
    "evaluator_version": "1",
    "evaluation_target": {"type": "last_turn", "response_start_index": 1},
    "context": {},
    "messages": [
      {
        "role": "user",
        "content": [{"type": "text", "text": "I forgot my password."}]
      },
      {
        "role": "assistant",
        "content": [
          {
            "type": "text",
            "text": "Use the password-reset link on the sign-in page."
          }
        ]
      }
    ],
    "agent_definition": {"tool_definitions": []},
    "metadata": {"run_type": "dataset"},
    "threshold": 3
  }'

The function returns a passing score:

{
  "status": "completed",
  "score": 5,
  "passed": true,
  "reason": "The assistant ends with a text answer.",
  "properties": {
    "tool_call_count": 0,
    "dimension_scores": [
      {
        "id": "final_answer",
        "score": 5,
        "reason": "The assistant ends with a text answer."
      }
    ]
  }
}

Deploy the function

Publish the project to your existing function app:

func azure functionapp publish <FUNCTION_APP_NAME>

In the deployed function app, set EVALUATOR_API_KEY as an application setting. For example:

az functionapp config appsettings set \
  --name <FUNCTION_APP_NAME> \
  --resource-group <RESOURCE_GROUP> \
  --settings EVALUATOR_API_KEY=<API_KEY>

Use the same secret for ENDPOINT_API_KEY when you create the project connection. Set ENDPOINT_URL to https://<FUNCTION_APP_NAME>.azurewebsites.net/api/evaluate.

What the endpoint receives

The evaluation service sends one POST request for each evaluation item. With schema version 0.0.2, every request uses the same shape, whether the item comes from a dataset, an agent or model target, a trace, or a simulation.

Request envelope

Field Type Description
schema_version string Always "0.0.2".
evaluator_name string The registered name of the evaluator being run.
evaluator_version string The version of the evaluator definition.
evaluation_target object What to score. See Evaluation target.
messages array The conversation to evaluate, normalized to one message format. See Message format.
agent_definition object Contains tool_definitions, the tools available to the agent. The array is empty when no tools apply.
context object The original input row, unchanged, for dataset, model target, and agent target runs. Use it to read extra columns such as ground_truth. Trace, simulation, and stored response runs don't have an input row, so context doesn't contain one.
metadata object Contains run_type and, when available, identifiers such as response_id, agent_name, agent_version, conversation_id, trace_id, and span_id.
threshold number Registered pass or fail threshold. Present when the evaluator metric defines one.

context and messages can contain the same conversation in two forms. context holds your row as you submitted it. messages holds the version that Foundry normalized for scoring. Score from messages.

metadata.run_type is one of dataset, model_target, agent_target, agent_trace, agent_simulation, or stored_response.

Schema version 0.0.2 ignores data_mapping. Foundry builds messages from your data automatically:

  • A messages column is used as the conversation.
  • query and response columns become a user message and an assistant message.
  • For model and agent targets, the generated output, including tool calls and tool results, is appended after the input.

Evaluation target

evaluation_target.type tells your endpoint which part of messages to score:

Type When Fields
last_turn Turn-level evaluation response_start_index: the index in messages where the response being scored starts. Earlier messages are conversation history.
conversation Conversation-level evaluation None. Score all of messages.

For example, if response_start_index is 1, messages[0] is the user's question and messages[1:] is the response to score.

Message format

Each message has a role of system, user, assistant, or tool, and content is always an array of parts:

Part type Fields Appears in
text text Any role.
tool_call tool_call_id, name, arguments assistant messages.
tool_result tool_result tool messages. The message also has a top-level tool_call_id.

Request examples

The following examples are compact versions of real requests.

The dataset row contains query and response columns. Foundry converts them to messages and passes the original row in context.

{
  "schema_version": "0.0.2",
  "evaluator_name": "my-endpoint-evaluator",
  "evaluator_version": "1",
  "evaluation_target": {"type": "last_turn", "response_start_index": 1},
  "context": {
    "query": "What is 2 plus 2?",
    "response": "2 plus 2 equals 4.",
    "ground_truth": "4"
  },
  "messages": [
    {
      "role": "user",
      "content": [{"type": "text", "text": "What is 2 plus 2?"}]
    },
    {
      "role": "assistant",
      "content": [{"type": "text", "text": "2 plus 2 equals 4."}]
    }
  ],
  "agent_definition": {"tool_definitions": []},
  "metadata": {"run_type": "dataset"},
  "threshold": 3
}

Understand endpoint output

For an HTTP 200 response, return a JSON object with the following fields:

Field Type Required Description
status string Yes "completed", "skipped", or "error".
score number When completed Must be within the registered metric range. Must be an integer for ordinal metrics. Omit for skipped and error.
passed boolean When completed Whether the item passes. Omit for skipped and error.
reason string No Explanation for the score, skip, or error.
label string No A short result label, such as "pass" or "needs_review".
properties object No Evaluator-specific data. Can include dimension_scores.
usage object No Token usage, with optional prompt_tokens, completion_tokens, and total_tokens as nonnegative integers.
model string No The model your endpoint used, if any.

Foundry rejects responses that include any other top-level field. A rejected response is recorded as an error for that item.

Foundry doesn't derive passed from score. Use threshold from the request to calculate passed. Set the threshold on EvaluatorMetric when you register the evaluator.

Dimension scores

To report a breakdown, add properties.dimension_scores, an array of dimensions:

Field Type Description
id string Unique, nonempty dimension identifier.
score integer or null A score from 1 through 5, or null when the dimension isn't scored.
reason string Explanation for the dimension score.

A completed result needs at least one scored dimension. A skipped result must use null for every dimension score. Omit dimension_scores from an error result.

Completed response

{
  "status": "completed",
  "score": 4,
  "passed": true,
  "reason": "The answer is correct and cites the policy.",
  "properties": {
    "dimension_scores": [
      {
        "id": "accuracy",
        "score": 5,
        "reason": "The answer matches the policy."
      },
      {
        "id": "citation",
        "score": 3,
        "reason": "The policy is cited without a link."
      }
    ]
  }
}

Skipped response

Return skipped when the item doesn't apply to your evaluator:

{
  "status": "skipped",
  "reason": "No assistant messages to score."
}

Error response

If the endpoint can't produce a result, return a non-200 HTTP status with an error object. Foundry records the error for that evaluation item:

{
  "error": {
    "code": "500",
    "message": "Model inference failed"
  }
}

Foundry also records an error for an item, without calling your endpoint, when it can't build a valid request for that item. For example, a dataset row whose messages value isn't a valid message list gets an error. Other items in the run are still evaluated.

Schema version 0.0.1

If an endpoint-based evaluator doesn't set schema_version, Foundry uses schema version 0.0.1. It remains supported, but use 0.0.2 for new evaluators. Version 0.0.2 sends one normalized message format for every scenario and validates responses more strictly.

The main differences:

Area 0.0.1 0.0.2
Input fields Scenario-specific fields under data.item, such as query, output_text, output_items, or messages, plus data.sample Normalized messages, original row in context
Level evaluation_level: "turn" or "conversation" evaluation_target with last_turn and response_start_index, or conversation
Tools and metadata Mixed into data.item agent_definition.tool_definitions and metadata
data_mapping Supported Ignored
Response status "Completed", "Skipped", or "Error" "completed", "skipped", or "error"
Response fields score, passed, status, reason, properties, schema_version Strict field list, plus label, usage, model, and dimension_scores

Configure authentication

Endpoint-based evaluators support two authentication methods through project connections:

Method How it works Best for
API key The service passes the key in a request header when calling your endpoint. Endpoints that validate an api-key header, including the Azure Functions sample in this article.
Microsoft Entra ID The service acquires a managed identity token scoped to your app registration and passes it as a bearer token in the request's Authorization header. Azure Functions with role-based access control, Azure Functions with Easy Auth.

Register the evaluator

Creating an endpoint-based evaluator takes two steps: create a project connection that stores your endpoint URL and credentials, then register an evaluator that references that connection. The tabs show the Python SDK and Foundry portal. For the REST API, see Create an evaluator version.

In addition to FOUNDRY_PROJECT_ENDPOINT, set AZURE_SUBSCRIPTION_ID, AZURE_RESOURCE_GROUP, ENDPOINT_URL, and ENDPOINT_API_KEY as environment variables. The following example creates an API key connection and registers an evaluator against it:

from azure.ai.projects.models import (
    EndpointBasedEvaluatorDefinition,
    EndpointEvaluatorSchemaVersion,
    EvaluationLevel,
    EvaluatorCategory,
    EvaluatorMetric,
    EvaluatorType,
    EvaluatorVersion,
)
from azure.mgmt.cognitiveservices import CognitiveServicesManagementClient
from azure.mgmt.cognitiveservices.models import (
    ApiKeyAuthConnectionProperties,
    ConnectionApiKey,
    ConnectionPropertiesV2BasicResource,
)

import os
from urllib.parse import urlparse

subscription_id = os.environ["AZURE_SUBSCRIPTION_ID"]
resource_group = os.environ["AZURE_RESOURCE_GROUP"]
endpoint_url = os.environ["ENDPOINT_URL"]
endpoint_api_key = os.environ["ENDPOINT_API_KEY"]

hostname = urlparse(endpoint).hostname
if hostname is None:
    raise ValueError(f"Invalid Foundry project endpoint: {endpoint}")
account_name = hostname.split(".")[0]

# Create the connection that stores the endpoint URL and API key.
mgmt_client = CognitiveServicesManagementClient(
    credential=credential,
    subscription_id=subscription_id,
)
connection = ConnectionPropertiesV2BasicResource(
    properties=ApiKeyAuthConnectionProperties(
        category="ApiKey",
        target=endpoint_url,
        credentials=ConnectionApiKey(key=endpoint_api_key),
    ),
)
mgmt_client.account_connections.create(
    resource_group_name=resource_group,
    account_name=account_name,
    connection_name="my-endpoint-connection",
    connection=connection,
)

# Define the evaluator
evaluator_definition = EndpointBasedEvaluatorDefinition(
    connection_name="my-endpoint-connection",
    # Use the recommended endpoint contract. Without this, Foundry uses 0.0.1.
    schema_version=EndpointEvaluatorSchemaVersion.V0_0_2,
    metrics={
        "score": EvaluatorMetric(
            type="ordinal",
            min_value=1,
            max_value=5,
            threshold=3,
        ),
    },
)

# Register an evaluator that references the connection
endpoint_evaluator = project_client.evaluators.create_version(
    name="my-endpoint-evaluator",
    evaluator_version=EvaluatorVersion(
        categories=[EvaluatorCategory.QUALITY],
        evaluator_type=EvaluatorType.CUSTOM,
        display_name="My Endpoint Evaluator",
        description="Scores responses using a custom evaluation endpoint",
        supported_evaluation_levels=[
            EvaluationLevel.TURN,
            EvaluationLevel.CONVERSATION,
        ],
        definition=evaluator_definition,
    ),
    # Custom evaluator definitions are in preview and require this opt-in header.
    headers={"Foundry-Features": "Evaluations=V1Preview"},
)

With the REST API, set "schema_version": "0.0.2" in the definition object and send the Foundry-Features: Evaluations=V1Preview header. An evaluator that uses 0.0.2 must define exactly one metric of type ordinal or continuous.

For Microsoft Entra ID authentication, use AADAuthTypeConnectionProperties instead of ApiKeyAuthConnectionProperties, and configure your endpoint to validate the incoming bearer token (for example, with Azure App Service Easy Auth). With Entra ID, Foundry sends a bearer token instead of an api-key header, so this article's function rejects the request with 401. Replace the function's API key check with bearer token validation. Remove the check only after Easy Auth or another layer validates the token and authorizes the caller.

Run an evaluation

Once your endpoint-based evaluator is registered, use it in an evaluation through an SDK, the REST API, or the Foundry portal. The tabs show the Python SDK and portal experiences.

This example evaluates one dataset row with query, response, document, and ground_truth fields. Foundry converts query and response to messages and passes the full row in context, as shown in the Dataset turn example in What the endpoint receives. No data_mapping is required. The following code defines input_item, testing_criteria, and data_source_config, then creates one eval and one run.

from openai.types.eval_create_params import DataSourceConfigCustom
from openai.types.evals.create_eval_jsonl_run_data_source_param import (
    CreateEvalJSONLRunDataSourceParam,
    SourceFileContent,
    SourceFileContentContent,
)

input_item = {
    "query": "I can't sign in to my account.",
    "response": "Use the password-reset link on the sign-in page.",
    "document": "Account sign-in help article.",
    "ground_truth": "Use the password-reset link.",
}

testing_criteria = [
    {
        "type": "azure_ai_evaluator",
        "name": "endpoint_eval",
        "evaluator_name": "my-endpoint-evaluator",
    },
]

data_source_config = DataSourceConfigCustom(
    type="custom",
    item_schema={
        "type": "object",
        "properties": {
            "query": {"type": "string"},
            "response": {"type": "string"},
            "document": {"type": "string"},
            "ground_truth": {"type": "string"},
        },
        "required": ["query", "response"],
    },
)

eval_object = client.evals.create(
    name="endpoint-evaluator-test",
    data_source_config=data_source_config,
    testing_criteria=testing_criteria,
)

eval_run = client.evals.runs.create(
    eval_id=eval_object.id,
    name="endpoint-evaluator-run",
    data_source=CreateEvalJSONLRunDataSourceParam(
        type="jsonl",
        source=SourceFileContent(
            type="file_content",
            content=[
                SourceFileContentContent(
                    item=input_item,
                ),
            ],
        ),
    ),
    extra_body={"evaluation_level": "turn"},
)

print(f"Run started: {eval_run.id}")

For a conversation-level run, each item needs the whole conversation, not just query and response. Provide a messages array in each item, add messages to item_schema, and set evaluation_level to "conversation" in extra_body. Foundry then sends evaluation_target with type conversation. For the data format, see Evaluate conversation datasets. To evaluate an agent or model target, or traces, see Evaluate models and agents in the cloud and Run evaluations from the SDK. The endpoint receives the same request shape in every case.

For polling, results, cancellation, and troubleshooting, see Get cloud evaluation results. For the complete run and cleanup lifecycle, see the endpoint evaluator Python sample.

Clean up

When you're done testing, delete the resources you created for this article so they don't continue to incur cost or stay reachable: the evaluation, the evaluator version, the connection that stores your endpoint credentials, and the hosted endpoint itself if you no longer need it. For the complete cleanup code, see the endpoint evaluator Python sample.