Set up continuous and scheduled evaluation

Evaluate your agent's production quality on an ongoing basis by using evaluation rules in Microsoft Foundry:

  • Scheduled evaluation runs on a fixed schedule. It can evaluate the agent's recent traces, or send a fixed set of queries to the agent and score its responses.
  • Continuous evaluation evaluates agent responses as they complete.

Results appear on the Evaluations page and in the evaluation charts on the agent's Monitor tab. For the dashboard, see Monitor agents with the Agent Monitoring Dashboard.

Prerequisites

Install the SDK

pip install "azure-ai-projects>=2.11.0" azure-identity python-dotenv

Set these environment variables with your own values:

  • AZURE_AI_PROJECT_ENDPOINT: The Foundry project endpoint, as found on the project overview page in the Foundry portal.
  • AZURE_AI_AGENT_NAME: The name of the agent to use for evaluation.
  • AZURE_AI_MODEL_DEPLOYMENT_NAME: The deployment name of the model.

Assign permissions

To enable evaluation rules, assign the project managed identity the Foundry User role.

Important

The Foundry RBAC roles were recently renamed. Foundry User, Foundry Owner, Foundry Account Owner, and Foundry Project Manager were previously named Azure AI User, Azure AI Owner, Azure AI Account Owner, and Azure AI Project Manager. You might still see the previous names in some places while the rename rolls out. The role IDs and core permissions are unchanged by the rename.

For all evaluation rule role requirements, see Set up permissions for evaluation workflows.

  1. In the Azure portal, open the resource for your Foundry project.
  2. Select Access control (IAM), and then select Add.
  3. Create a role assignment for Foundry User.
  4. For the member, select your Foundry project's managed identity.

Create an agent

import os
from dotenv import load_dotenv
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
from azure.ai.projects.models import (
    PromptAgentDefinition,
)

load_dotenv()

endpoint = os.environ["AZURE_AI_PROJECT_ENDPOINT"]

# Later sections reuse these clients and the agent.
credential = DefaultAzureCredential()
project_client = AIProjectClient(endpoint=endpoint, credential=credential)
openai_client = project_client.get_openai_client()

agent = project_client.agents.create_version(
    agent_name=os.environ["AZURE_AI_AGENT_NAME"],
    definition=PromptAgentDefinition(
        model=os.environ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
        instructions="You are a helpful assistant that answers general questions",
    ),
)
print(f"Agent created (id: {agent.id}, name: {agent.name}, version: {agent.version})")

References: AIProjectClient, DefaultAzureCredential

Create a recurring evaluation

Scheduled and continuous evaluations are both evaluation rules. A scheduled rule uses a timer trigger, and a continuous rule runs when agent responses complete. The following examples target a Foundry agent by name. For an external agent, use a scheduled trace evaluation and set agent_id to the agent's otel_agent_id in the trace source instead of agent_name. Continuous evaluation runs when Foundry agent responses complete, so it doesn't apply to external agents.

The following examples continue from the previous section, which creates project_client, openai_client, and agent.

Scheduled trace evaluation: Evaluate up to 10 of the agent's traces every hour.

import time
from azure.ai.projects.models import (
    EvaluationRule,
    EvaluationRuleEventType,
    HourlyRecurrenceSchedule,
    RecurrenceTrigger,
    ScheduleProvisioningStatus,
    TraceEvaluationRuleAction,
    TracesEvalRunDataSource,
)

model_deployment_name = os.environ["AZURE_AI_MODEL_DEPLOYMENT_NAME"]
PENDING_STATUSES = (ScheduleProvisioningStatus.CREATING, ScheduleProvisioningStatus.UPDATING)

trace_eval = openai_client.evals.create(
    name="Scheduled Trace Evaluation",
    data_source_config={"type": "azure_ai_source", "scenario": "traces"},  # type: ignore
    testing_criteria=[  # type: ignore
        {
            "type": "azure_ai_evaluator",
            "name": "coherence",
            "evaluator_name": "builtin.coherence",
            "initialization_parameters": {"deployment_name": model_deployment_name},
            "data_mapping": {"query": "{{item.query}}", "response": "{{item.response}}"},
        }
    ],
)

trace_eval_rule = project_client.evaluation_rules.create_or_update(
    id="my-trace-eval-rule",
    evaluation_rule=EvaluationRule(
        display_name="My Trace Eval Rule",
        description="An eval rule that evaluates agent traces every hour",
        action=TraceEvaluationRuleAction(
            eval_id=trace_eval.id,
            source=TracesEvalRunDataSource(agent_name=agent.name, max_traces=10),
        ),
        event_type=EvaluationRuleEventType.TIMER_TRIGGER,
        schedule=RecurrenceTrigger(interval=1, schedule=HourlyRecurrenceSchedule()),
        enabled=True,
    ),
)

# The schedule is provisioned in the background.
# Wait for it before you update or delete the rule.
while trace_eval_rule.provisioning_status in PENDING_STATUSES:
    time.sleep(5)
    trace_eval_rule = project_client.evaluation_rules.get(id=trace_eval_rule.id)
print(f"Trace Evaluation Rule provisioning status: {trace_eval_rule.provisioning_status}")

Scheduled dataset evaluation: Every day at 9 AM UTC, send a fixed set of queries to the agent and score its responses. The evaluator reads the agent's response from sample.output_text.

from azure.ai.projects.models import (
    AzureAIAgentTargetParam,
    DailyRecurrenceSchedule,
    DatasetEvaluationRuleAction,
    TargetCompletionEvalRunDataSource,
)
from openai.types.evals.create_eval_completions_run_data_source_param import (
    SourceFileContent,
    SourceFileContentContent,
)

dataset_eval = openai_client.evals.create(
    name="Scheduled Agent Evaluation",
    data_source_config={  # type: ignore
        "type": "custom",
        "item_schema": {
            "type": "object",
            "properties": {"query": {"type": "string"}},
            "required": ["query"],
        },
        "include_sample_schema": True,
    },
    testing_criteria=[  # type: ignore
        {
            "type": "azure_ai_evaluator",
            "name": "coherence",
            "evaluator_name": "builtin.coherence",
            "initialization_parameters": {"deployment_name": model_deployment_name},
            "data_mapping": {"query": "{{item.query}}", "response": "{{sample.output_text}}"},
        }
    ],
)

agent_queries = TargetCompletionEvalRunDataSource(
    type="azure_ai_target_completions",
    source=SourceFileContent(
        type="file_content",
        content=[
            SourceFileContentContent(item={"query": "How many days are in a leap year?"}),
            SourceFileContentContent(item={"query": "How many legs does a spider have?"}),
        ],
    ),
    input_messages={
        "type": "template",  # type: ignore
        "template": [
            {"type": "message", "role": "user", "content": {"type": "input_text", "text": "{{item.query}}"}}
        ],
    },
    target=AzureAIAgentTargetParam(type="azure_ai_agent", name=agent.name, version=agent.version),
)

dataset_eval_rule = project_client.evaluation_rules.create_or_update(
    id="my-dataset-eval-rule",
    evaluation_rule=EvaluationRule(
        display_name="My Agent Dataset Eval Rule",
        description="An eval rule that evaluates an agent on a fixed set of queries every day",
        action=DatasetEvaluationRuleAction(eval_id=dataset_eval.id, source=agent_queries),
        event_type=EvaluationRuleEventType.TIMER_TRIGGER,
        schedule=RecurrenceTrigger(interval=1, schedule=DailyRecurrenceSchedule(hours=[9]), time_zone="UTC"),
        enabled=True,
    ),
)

while dataset_eval_rule.provisioning_status in PENDING_STATUSES:
    time.sleep(5)
    dataset_eval_rule = project_client.evaluation_rules.get(id=dataset_eval_rule.id)
print(f"Dataset Evaluation Rule provisioning status: {dataset_eval_rule.provisioning_status}")

To pause a scheduled rule without deleting it, set enabled to False, call create_or_update again, and wait for provisioning to finish.

Continuous evaluation: Evaluate agent responses as they complete.

from azure.ai.projects.models import (
    ContinuousEvaluationRuleAction,
    EvaluationRuleFilter,
)

continuous_eval = openai_client.evals.create(
    name="Continuous Evaluation",
    data_source_config={"type": "azure_ai_source", "scenario": "responses"},  # type: ignore
    testing_criteria=[  # type: ignore
        {"type": "azure_ai_evaluator", "name": "violence_detection", "evaluator_name": "builtin.violence"}
    ],
)

continuous_eval_rule = project_client.evaluation_rules.create_or_update(
    id="my-continuous-eval-rule",
    evaluation_rule=EvaluationRule(
        display_name="My Continuous Eval Rule",
        description="An eval rule that runs on agent response completions",
        action=ContinuousEvaluationRuleAction(eval_id=continuous_eval.id, max_hourly_runs=100),
        event_type=EvaluationRuleEventType.RESPONSE_COMPLETED,
        filter=EvaluationRuleFilter(agent_name=agent.name),
        enabled=True,
    ),
)
print(f"Continuous Evaluation Rule created (id: {continuous_eval_rule.id})")

References: EvaluationRuleEventType, EvaluationRule

Verify evaluation results

  1. For trace-based or continuous evaluation, generate agent traffic (for example, run your app or test the agent in the portal).
  2. In the Foundry portal, open the agent and select Monitor.
  3. Review evaluation-related charts for the selected time range.

If the setup is successful, the evaluation-related charts display scores for your selected time range, and the evaluation runs list shows entries with status Completed.

You can also list recent evaluation runs and open the report URL. The following examples check the continuous evaluation. To check a scheduled rule, use its evaluation ID instead: trace_eval.id or dataset_eval.id in Python, traceEvaluationId or datasetEvaluationId in C#, or traceEval.id or datasetEval.id in JavaScript.

eval_run_list = openai_client.evals.runs.list(
    eval_id=continuous_eval.id,
    order="desc",
    limit=10,
)

if len(eval_run_list.data) > 0 and eval_run_list.data[0].report_url:
    print(f"Report URL: {eval_run_list.data[0].report_url}")

References: Get cloud evaluation results

Use custom evaluators

In addition to built-in evaluators, you can add your own evaluators to evaluation rules. Rubric evaluators are generally available. Other custom evaluator types, such as code-based, prompt-based, and endpoint-based evaluators, are in preview.

In the portal, select your evaluators in the Custom view of the Evaluator catalog. In code, set evaluator_name in testing_criteria to your evaluator's name.

Full sample code

To view the full sample code, see:

Troubleshooting

Issue Cause Resolution
Continuous evaluation results don't appear Continuous evaluation isn't enabled or rule creation failed Confirm that your rule is enabled and that agent traffic is flowing. Confirm the project managed identity has the Foundry User role. See Set up permissions for evaluation workflows.
Scheduled evaluation never runs The schedule didn't finish provisioning, the rule is disabled, or no traces match the agent name Get the rule, and confirm that provisioning_status succeeded and enabled is true. For trace evaluation, confirm that the agent has recent traces.
Continuous evaluation runs are skipped Hourly run limit reached Increase max_hourly_runs in the continuous evaluation rule, or wait for the next hour. The default limit is 100 runs per hour.