Evaluation establishes a quality baseline for your agent and lets you set acceptance thresholds, such as a task adherence passing rate, before you release changes to users.
Each step offers five paths. Use whichever you prefer:
Evaluation runs against a deployed, invokable agent. Confirm your agent is deployed and available before you set up the evaluation.
From your azd project directory, verify the agent is deployed and invokable:
azd ai agent show
Send a test prompt:
azd ai agent invoke "Write a haiku about deploying cloud applications."
You should see a response within a few seconds.
- Open the Foundry portal and go to your project.
- Select your agent, and then select the Playground tab.
- Send a test prompt, such as
Write a haiku about deploying cloud applications.
You should see a response within a few seconds.
Install the Foundry SDK:
pip install "azure-ai-projects>=2.0.0" azure-identity
Set two environment variables, and then create the project client. Set FOUNDRY_PROJECT_ENDPOINT to your project endpoint and FOUNDRY_MODEL_NAME to a chat-completion deployment to use as the judge model. The following code samples assume you run them in this context:
import os
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
endpoint = os.environ["FOUNDRY_PROJECT_ENDPOINT"]
model_deployment = os.environ["FOUNDRY_MODEL_NAME"]
credential = DefaultAzureCredential()
project_client = AIProjectClient(endpoint=endpoint, credential=credential)
client = project_client.get_openai_client()
Confirm your deployed agent is registered and available. Replace <your-agent-name> with your hosted agent's name:
agent = project_client.agents.get("<your-agent-name>")
print(f"Found agent: {agent.name}")
The call returns the agent if it exists, or raises an error if the name is wrong or the agent isn't deployed.
Install the Foundry SDK and the OpenAI evals client:
dotnet add package Azure.AI.Projects --prerelease
dotnet add package OpenAI
dotnet add package Azure.Identity
Set two environment variables, and then create the clients. Set FOUNDRY_PROJECT_ENDPOINT to your project endpoint and FOUNDRY_MODEL_NAME to a chat-completion deployment to use as the judge model. The following code samples assume you run them in this context:
using System.ClientModel;
using System.Text.Json;
using Azure.AI.Projects;
using Azure.AI.Projects.Agents;
using Azure.Core;
using Azure.Identity;
using OpenAI;
using OpenAI.Evals;
#pragma warning disable AAIP001, OPENAI001
var endpoint = Environment.GetEnvironmentVariable("FOUNDRY_PROJECT_ENDPOINT")!;
var modelDeployment = Environment.GetEnvironmentVariable("FOUNDRY_MODEL_NAME")!;
var credential = new DefaultAzureCredential();
AIProjectClient projectClient = new(new Uri(endpoint), credential);
// OpenAI-compatible evals client bound to the Foundry project endpoint.
// A Microsoft Entra token works as the credential because both use "Authorization: Bearer".
var token = credential.GetToken(new TokenRequestContext(["https://ai.azure.com/.default"])).Token;
EvaluationClient evalClient = new(
new ApiKeyCredential(token),
new OpenAIClientOptions { Endpoint = new Uri($"{endpoint}/openai/v1") });
Confirm your deployed agent is registered and available. Replace <your-agent-name> with your hosted agent's name:
ProjectsAgentRecord agent = projectClient.AgentAdministrationClient.GetAgent("<your-agent-name>");
Console.WriteLine($"Found agent: {agent.Name}");
The call returns the agent if it exists, or raises an error if the name is wrong or the agent isn't deployed.
Install the Foundry SDK:
npm install @azure/ai-projects @azure/identity dotenv
Set two environment variables, and then create the project client. Set FOUNDRY_PROJECT_ENDPOINT to your project endpoint and FOUNDRY_MODEL_NAME to a chat-completion deployment to use as the judge model. The following code samples assume you run them in this context:
import { DefaultAzureCredential } from "@azure/identity";
import { AIProjectClient } from "@azure/ai-projects";
import "dotenv/config";
const endpoint = process.env["FOUNDRY_PROJECT_ENDPOINT"] || "";
const modelDeployment = process.env["FOUNDRY_MODEL_NAME"] || "";
const projectClient = new AIProjectClient(
endpoint,
new DefaultAzureCredential(),
);
const client = projectClient.getOpenAIClient();
Confirm your deployed agent is registered and available. Replace <your-agent-name> with your hosted agent's name:
const agent = await projectClient.agents.get("<your-agent-name>");
console.log(`Found agent: ${agent.name}`);
The call returns the agent if it exists, or raises an error if the name is wrong or the agent isn't deployed.
Reference: AIProjectClient class
Start with built-in evaluators to score your agent against a test dataset.
First, create a JSONL file of test queries for your agent. Each line is a JSON object with a query field. Save it inside your agent's source folder, as src/<your-agent-name>/tests/queries.jsonl:
{"query": "Write a haiku about deploying cloud applications."}
Then create an eval.yaml file in the same agent source folder, as src/<your-agent-name>/eval.yaml. It points to your dataset and lists the built-in evaluators to apply. The dataset.local_uri path is relative to this folder. Replace <your-agent-name> with your hosted agent's name and <your-chat-completion-deployment> with the judge model deployment:
name: agent-eval
agent:
name: <your-agent-name>
kind: hosted
dataset:
local_uri: tests/queries.jsonl
evaluators:
- builtin.intent_resolution
- builtin.task_adherence
options:
eval_model: <your-chat-completion-deployment>
max_samples: 15
The eval_model value is the judge model that scores responses; you can reuse the deployment your agent already uses.
- In the Foundry portal, open your agent and select the Evaluation tab, then select Create.
- For Select evaluation target, select Agent.
- For Select evaluation scope, select Individual turns.
- For Select data source, select Existing dataset and choose a CSV or JSONL file of test queries from your project's data assets.
- If the Configure agents step appears, review the agent and accept the default user prompt,
{{item.query}}. Adjust it only if your agent expects a different input format.
- For Select testing criteria, select one or more agent evaluators, such as Task Adherence and Intent Resolution.
Keep the wizard open. You submit the evaluation in the next step.
First, create a JSONL file of test queries for your agent. Each line is a JSON object with a query field. Save it as queries.jsonl:
{"query": "Write a haiku about deploying cloud applications."}
Upload the file as a dataset in your project:
dataset = project_client.datasets.upload_file(
name="agent-test-queries",
version="1",
file_path="./queries.jsonl",
)
Next, choose built-in evaluators and map their inputs. The data_mapping parameter tells each evaluator where to find the query and the agent response. AI-assisted evaluators need a judge model in initialization_parameters; the value must be a chat-completion deployment in your project.
from azure.ai.projects.models import TestingCriterionAzureAIEvaluator
testing_criteria = [
TestingCriterionAzureAIEvaluator(
type="azure_ai_evaluator",
name="Intent Resolution",
evaluator_name="builtin.intent_resolution",
initialization_parameters={"model": model_deployment},
data_mapping={
"query": "{{item.query}}",
"response": "{{sample.output_items}}",
},
),
TestingCriterionAzureAIEvaluator(
type="azure_ai_evaluator",
name="Task Adherence",
evaluator_name="builtin.task_adherence",
initialization_parameters={"model": model_deployment},
data_mapping={
"query": "{{item.query}}",
"response": "{{sample.output_items}}",
},
),
]
Create the evaluation. It defines the test data schema and testing criteria, and serves as a container for one or more runs:
from openai.types.eval_create_params import DataSourceConfigCustom
data_source_config = DataSourceConfigCustom(
type="custom",
item_schema={
"type": "object",
"properties": {"query": {"type": "string"}},
"required": ["query"],
},
include_sample_schema=True,
)
evaluation = client.evals.create(
name="Agent Quality Evaluation",
data_source_config=data_source_config,
testing_criteria=testing_criteria,
)
print(f"Evaluation created: {evaluation.id}")
First, create a JSONL file of test queries for your agent. Each line is a JSON object with a query field. Save it as queries.jsonl:
{"query": "Write a haiku about deploying cloud applications."}
Upload the file as a dataset in your project:
AIProjectDataset dataset = projectClient.Datasets.UploadFile(
name: "agent-test-queries",
version: "1",
filePath: "./queries.jsonl");
Next, choose built-in evaluators and map their inputs. The data_mapping parameter tells each evaluator where to find the query and the agent response. AI-assisted evaluators need a judge model in initialization_parameters; the value must be a chat-completion deployment in your project.
var testingCriteria = new object[]
{
new
{
type = "azure_ai_evaluator",
name = "Intent Resolution",
evaluator_name = "builtin.intent_resolution",
initialization_parameters = new { model = modelDeployment },
data_mapping = new { query = "{{item.query}}", response = "{{sample.output_items}}" },
},
new
{
type = "azure_ai_evaluator",
name = "Task Adherence",
evaluator_name = "builtin.task_adherence",
initialization_parameters = new { model = modelDeployment },
data_mapping = new { query = "{{item.query}}", response = "{{sample.output_items}}" },
},
};
Create the evaluation. It defines the test data schema and testing criteria, and serves as a container for one or more runs:
var createEvaluation = new
{
name = "Agent Quality Evaluation",
data_source_config = new
{
type = "custom",
item_schema = new
{
type = "object",
properties = new { query = new { type = "string" } },
required = new[] { "query" },
},
include_sample_schema = true,
},
testing_criteria = testingCriteria,
};
ClientResult evaluationResult = evalClient.CreateEvaluation(
BinaryContent.Create(BinaryData.FromObjectAsJson(createEvaluation)));
string evaluationId = JsonDocument.Parse(evaluationResult.GetRawResponse().Content.ToString())
.RootElement.GetProperty("id").GetString()!;
Console.WriteLine($"Evaluation created: {evaluationId}");
First, create a JSONL file of test queries for your agent. Each line is a JSON object with a query field. Save it as queries.jsonl:
{"query": "Write a haiku about deploying cloud applications."}
Upload the file as a dataset in your project:
const dataset = await projectClient.datasets.uploadFile(
"agent-test-queries",
"1",
"./queries.jsonl",
);
Next, choose built-in evaluators and map their inputs. The data_mapping parameter tells each evaluator where to find the query and the agent response. AI-assisted evaluators need a judge model in initialization_parameters; the value must be a chat-completion deployment in your project.
const testingCriteria = [
{
type: "azure_ai_evaluator",
name: "Intent Resolution",
evaluator_name: "builtin.intent_resolution",
initialization_parameters: { model: modelDeployment },
data_mapping: {
query: "{{item.query}}",
response: "{{sample.output_items}}",
},
},
{
type: "azure_ai_evaluator",
name: "Task Adherence",
evaluator_name: "builtin.task_adherence",
initialization_parameters: { model: modelDeployment },
data_mapping: {
query: "{{item.query}}",
response: "{{sample.output_items}}",
},
},
];
Create the evaluation. It defines the test data schema and testing criteria, and serves as a container for one or more runs:
const dataSourceConfig = {
type: "custom",
item_schema: {
type: "object",
properties: { query: { type: "string" } },
required: ["query"],
},
include_sample_schema: true,
};
const evaluation = await client.evals.create({
name: "Agent Quality Evaluation",
data_source_config: dataSourceConfig,
testing_criteria: testingCriteria,
});
console.log(`Evaluation created: ${evaluation.id}`);
Run the suite against your deployed agent. The service sends each test query to the agent, captures the response, and scores it with your selected evaluators.
Run the evaluation from the azd workspace root:
azd ai agent eval run --config eval.yaml
Note
azd ai agent eval run resolves the --config path relative to your agent's source folder under src/ (for example, src/<your-agent-name>/eval.yaml), not the current directory. Keep eval.yaml, and the dataset that its local_uri points to, inside that folder.
The command reads eval.yaml, sends each query to your agent, scores the responses, and prints a summary when it finishes:
Eval run started
Eval: eval_b36748dede424e4ba3f8e6c99ca2cf27
Run: evalrun_5f72ef189ad24790a32128e6f230b131
(✓) Done Eval run
Results: 1 total, 1 passed, 0 failed, 0 errored
Per-criteria results:
intent_resolution: 1 passed, 0 failed, 0 errored
task_adherence: 1 passed, 0 failed, 0 errored
- On the Review and submit step, enter a name for the evaluation.
- Review the target, scope, data source, and selected evaluators.
- Select Submit to start the run.
Create a run that sends each test query to your agent and applies the evaluators. Replace <your-agent-name> with your hosted agent's name:
eval_run = client.evals.runs.create(
eval_id=evaluation.id,
name="Agent Evaluation Run",
data_source={
"type": "azure_ai_target_completions",
"source": {"type": "file_id", "id": dataset.id},
"input_messages": {
"type": "template",
"template": [
{
"type": "message",
"role": "user",
"content": {"type": "input_text", "text": "{{item.query}}"},
}
],
},
"target": {
"type": "azure_ai_agent",
"name": "<your-agent-name>",
# "version": "1", # Optional; omit to use the latest version
},
},
)
print(f"Evaluation run started: {eval_run.id}")
Create a run that sends each test query to your agent and applies the evaluators. Replace <your-agent-name> with your hosted agent's name:
var createRun = new
{
name = "Agent Evaluation Run",
data_source = new
{
type = "azure_ai_target_completions",
source = new { type = "file_id", id = dataset.Id },
input_messages = new
{
type = "template",
template = new object[]
{
new { type = "message", role = "user", content = new { type = "input_text", text = "{{item.query}}" } },
},
},
// Add a "version" property to the target to pin a specific agent version; omit to use the latest.
target = new { type = "azure_ai_agent", name = "<your-agent-name>" },
},
};
ClientResult runResult = evalClient.CreateEvaluationRun(
evaluationId, BinaryContent.Create(BinaryData.FromObjectAsJson(createRun)));
string runId = JsonDocument.Parse(runResult.GetRawResponse().Content.ToString())
.RootElement.GetProperty("id").GetString()!;
Console.WriteLine($"Evaluation run started: {runId}");
Create a run that sends each test query to your agent and applies the evaluators. Replace <your-agent-name> with your hosted agent's name:
const evalRun = await client.evals.runs.create(evaluation.id, {
name: "Agent Evaluation Run",
data_source: {
type: "azure_ai_target_completions",
source: { type: "file_id", id: dataset.id },
input_messages: {
type: "template",
template: [
{
type: "message",
role: "user",
content: { type: "input_text", text: "{{item.query}}" },
},
],
},
target: {
type: "azure_ai_agent",
name: "<your-agent-name>",
// version: "1", // Optional; omit to use the latest version
},
},
});
console.log(`Evaluation run started: ${evalRun.id}`);
Evaluations typically complete in a few minutes, depending on the number of queries.
List recent evaluations:
azd ai agent eval list
Eval ID Name Status of last run Runs
------- ---- ------------------ ----
* eval_b36748dede424e4ba3f8e6c99ca2cf27 agent-eval Completed 1
* = active eval in current environment
Show the most recent evaluation and its runs:
azd ai agent eval show
Eval: eval_b36748dede424e4ba3f8e6c99ca2cf27
Name: agent-eval
Agent: <your-agent-name>
Runs: 1
Recent runs:
Run ID Status Passed Failed Created
------ ------ ------ ------ -------
evalrun_5f72ef189ad24790a32128e6f230b131 Completed 1/1 0 2026-06-17 14:52 UTC
Use the results to confirm which agent version was evaluated and which evaluator scores were produced. To see per-evaluator details and a link to the report in the Foundry portal, run azd ai agent eval show <eval-id> --eval-run-id <run-id>.
- The details page shows the target, dataset, status, token usage, and an aggregate score for each evaluator.
- Select the run name to view row-level results: each query, the agent response, the evaluator score, and the score explanation.
Poll for completion, then print the status and the report URL that opens the results in the Foundry portal:
import time
while True:
run = client.evals.runs.retrieve(run_id=eval_run.id, eval_id=evaluation.id)
if run.status in ["completed", "failed"]:
break
time.sleep(5)
print(f"Status: {run.status}")
print(f"Report URL: {run.report_url}")
At the run level, you can see aggregated pass and fail counts for each evaluator:
print(run.result_counts)
for criteria in run.per_testing_criteria_results:
print(criteria.testing_criteria, "passed:", criteria.passed, "failed:", criteria.failed)
ResultCounts(errored=0, failed=0, passed=1, total=1, skipped=0)
Intent Resolution passed: 1 failed: 0
Task Adherence passed: 1 failed: 0
For row-level detail, list the output items. Each result includes the evaluator name, pass or fail, and a score:
for item in client.evals.runs.output_items.list(run_id=eval_run.id, eval_id=evaluation.id):
for result in item.results:
print(item.id, result.name, "passed:", result.passed, "score:", result.score)
Poll for completion, and then print the status and the report URL that opens the results in the Foundry portal:
JsonElement run = default;
while (true)
{
ClientResult runStatus = evalClient.GetEvaluationRun(evaluationId, runId, options: null);
run = JsonDocument.Parse(runStatus.GetRawResponse().Content.ToString()).RootElement;
string status = run.GetProperty("status").GetString()!;
if (status is "completed" or "failed") break;
Thread.Sleep(TimeSpan.FromSeconds(5));
}
Console.WriteLine($"Status: {run.GetProperty("status").GetString()}");
Console.WriteLine($"Report URL: {run.GetProperty("report_url").GetString()}");
At the run level, you can see aggregated pass and fail counts for each evaluator:
Console.WriteLine(run.GetProperty("result_counts").GetRawText());
foreach (JsonElement criteria in run.GetProperty("per_testing_criteria_results").EnumerateArray())
{
Console.WriteLine(
$"{criteria.GetProperty("testing_criteria").GetString()} " +
$"passed: {criteria.GetProperty("passed").GetInt32()} " +
$"failed: {criteria.GetProperty("failed").GetInt32()}");
}
For row-level detail, list the output items. Each result includes the evaluator name, pass or fail, and a score:
ClientResult outputItems = evalClient.GetEvaluationRunOutputItems(
evaluationId, runId, limit: 100, order: null, after: null, outputItemStatus: null, options: null);
foreach (JsonElement item in JsonDocument.Parse(outputItems.GetRawResponse().Content.ToString())
.RootElement.GetProperty("data").EnumerateArray())
{
foreach (JsonElement result in item.GetProperty("results").EnumerateArray())
{
Console.WriteLine(
$"{item.GetProperty("id").GetString()} {result.GetProperty("name").GetString()} " +
$"passed: {result.GetProperty("passed")} score: {result.GetProperty("score")}");
}
}
Poll for completion, and then print the status and the report URL that opens the results in the Foundry portal:
let run = evalRun;
while (!["completed", "failed"].includes(run.status)) {
run = await client.evals.runs.retrieve(run.id, {
eval_id: evaluation.id,
});
await new Promise((resolve) => setTimeout(resolve, 5000));
}
console.log(`Status: ${run.status}`);
console.log(`Report URL: ${run.report_url}`);
At the run level, you can see aggregated pass and fail counts for each evaluator:
console.log(JSON.stringify(run.result_counts));
for (const criteria of run.per_testing_criteria_results) {
console.log(
criteria.testing_criteria,
"passed:",
criteria.passed,
"failed:",
criteria.failed,
);
}
{"errored":0,"failed":0,"passed":1,"total":1,"skipped":0}
Intent Resolution passed: 1 failed: 0
Task Adherence passed: 1 failed: 0
For row-level detail, list the output items. Each result includes the evaluator name, pass or fail, and a score:
for await (const item of client.evals.runs.outputItems.list(run.id, {
eval_id: evaluation.id,
})) {
for (const result of item.results) {
console.log(item.id, result.name, "passed:", result.passed, "score:", result.score);
}
}
This quickstart registers a dataset, an evaluation, and run history in your Foundry project. These assets incur little or no ongoing cost.
To remove the hosted agent and the Azure resources you created, follow the cleanup steps in Deploy your first hosted agent.
