Evaluation in Microsoft Foundry

Evaluation helps you measure the quality, safety, and reliability of models and agents throughout development and production. In Microsoft Foundry, you can evaluate existing outputs in a test dataset, generate responses from a model or agent by using test inputs, or evaluate production traces.

An evaluation applies one or more evaluators to your data or application. Evaluators produce metrics and, when supported, explanations that help you compare versions, find quality problems, and decide whether an application is ready to deploy.

Build an evaluation workflow

Most evaluation workflows follow these steps:

1. Choose what to evaluate. Select an individual turn or a complete conversation, and use an existing dataset, generated responses, simulated conversations, or production traces.
2. Select evaluators. Choose the quality and safety metrics that matter for your application, such as task completion, relevance, groundedness, similarity, or content safety.
3. Run the evaluation. Run it once during development, add it to CI/CD, or configure recurring evaluation for production.
4. Review and improve. Analyze aggregate metrics and individual results, compare versions, and address recurring failure patterns.

Choose where to start

Note

Before you begin, review the permissions for evaluation workflows, supported regions and rate limits, and virtual network requirements.

Choose an evaluation approach

Choose the evaluation unit before you select evaluators. Turn-level evaluation scores individual responses. Conversation-level evaluation scores a complete multi-turn interaction and can measure outcomes such as task completion across multiple steps. Evaluators support specific evaluation levels.

I want to	Start here
Evaluate a hosted agent for the first time	Quickstart: Evaluate your hosted agent
Evaluate existing query-response test data	Evaluate query-response datasets
Generate and evaluate responses from test inputs	Evaluate model and agent responses
Evaluate individual interactions captured from a deployed model or agent	Evaluate deployed interactions
Evaluate a dataset of complete conversations	Evaluate conversation datasets
Evaluate complete conversations captured in production traces	Evaluate deployed conversations
Simulate multi-turn agent behavior before deployment	Simulate agent conversations
Generate scenarios and conversations without hand-authoring test data	Generate and evaluate synthetic conversations
Run adversarial safety and security testing	Learn about AI red teaming

Choose how to run evaluations

I want to	Start here
Run an evaluation in the Foundry portal	Run evaluations from the Microsoft Foundry portal
Run evaluations programmatically	Introduction to cloud evaluation with the Microsoft Foundry SDK
Run agent evaluations from the command line	Run agent evaluations with the azd CLI
Evaluate production traffic on a recurring basis	Monitor agents and set up continuous evaluation
Add evaluations to a deployment pipeline	Run evaluations in GitHub Actions or Azure DevOps

Find the components you need

• Evaluators: Start with the built-in evaluators, or create a custom evaluator for your own criteria.
• Evaluation data: Learn when to use an evaluation dataset, how to generate synthetic data, or how to create a dataset from production traces.
• Results: Review results in the Foundry portal, retrieve cloud evaluation results with the SDK, or use cluster analysis to find common failure patterns.