Evaluate an agent (preview)

[This article is prerelease documentation and is subject to change.]

The Evaluate tab in the new agent experience provides structured, repeatable testing for your agent. Use evaluations to measure agent quality across test cases, track improvements over time, and validate your agent's behavior before you publish it.

Important

What is agent evaluation?

Agent evaluation lets you systematically test your agent's responses against quality standards. Instead of testing each scenario manually in the Preview tab, create evaluations with test conversations, define how to measure responses, and run the evaluations to get quantitative results.

Evaluation helps you answer questions like these:

  • Does the agent answer correctly across a range of expected scenarios?
  • Did a configuration change improve or degrade response quality?
  • Does the agent call tools when it should?

Key concepts

The following concepts are key when running evaluations in Copilot Studio:

Conversations

A conversation is a test case that represents a scenario you want your agent to handle. Each conversation includes user messages and optionally expected agent responses. Organize conversations into evaluations. Create conversations manually, generate them with AI, or upload them from a CSV file.

Evaluations

An evaluation is a named test set that combines conversations with a test method. You create evaluations from the Evaluate tab, add conversations to them, and then run them to produce scored results.

Test methods

Test methods define how the agent's responses are scored. Currently, the only available test method is General quality, an AI-based assessment of whether responses meet quality standards, such as relevance and completeness.

Note

The General quality test method doesn't compare responses to expected answers.

User profile

Evaluations run under a user profile. Manage which authenticated profile runs the evaluation so the agent's tools and connections are fully testable.

Evaluation workflow

Follow these steps to evaluate an agent:

  1. Create an evaluation: On the Evaluate tab, start a new evaluation. Learn more in Create a test set for an agent.
  2. Add conversations: Add test conversations by writing them manually, generating them with AI, or uploading a CSV file.
  3. Configure: Name the evaluation, select a test method, choose the agent version, and set the user profile.
  4. Run the evaluation: Select Evaluate to run the test and wait for the results. Learn more in Run an evaluation for an agent.
  5. Review results: Analyze scores and identify areas for improvement. Learn more in View evaluation results for an agent.
  6. Iterate: Adjust your agent's instructions, knowledge, or tools, and then run the evaluation again to measure the impact.
  • Last updated on