Plan and run agent evaluations

Completed

Before you share an agent, test how it responds to different requests. Copilot Studio provides two tabs for this work: use the Preview tab for quick checks and the Evaluate tab for tests that you want to run again.

Priya uses the Preview tab while she builds the support policy agent. Before she releases it, she creates a set of tests that she can run after every important change.

Understand the difference between the preview and evaluate tabs

The Preview tab contains an interactive chat where you send messages to the agent and review its responses. Use the Preview tab to:

  • Checking whether a new instruction takes effect immediately.
  • Exploring how the agent handles a specific phrase or edge case.
  • Confirming that a knowledge change is reflected in the response.

The Preview tab supports interactive testing, but it doesn't run a saved, scored test set in the way the Evaluate tab does. A change that appears correct in one preview session might behave differently after a later update.

The Evaluate tab runs a saved set of test conversations and scores the responses. Use the Evaluate tab when you need to:

  • Establish a quality baseline before releasing an agent or an update.
  • Detect regressions after a change to instructions, knowledge, tools, or connected agents.
  • Cover scenarios that span different intent types, edge cases, and boundary conditions in a consistent way.

Get started with the evaluate tab

Important

The Evaluate tab for agents powered by the Microsoft Copilot harness is in production-ready preview. It might not be available in every environment, and the interface might change. Ask your administrator whether the Evaluate tab is available before you use it as a release check.

To open the Evaluate tab, select Evaluate in the top navigation of the Copilot Studio agent editor. The tab includes several components:

  • Test set: A collection of conversations that represents the scenarios you want to evaluate.
  • Conversation: A test case with one or more user messages and optional expected agent responses.
  • General quality: The currently available test method, which uses AI to assess whether responses meet quality standards such as relevance and completeness.
  • User profile: The currently signed-in user's identity, which the evaluation and tool connections use. You can't select a different identity while signed in. To evaluate with typical user permissions, sign out, sign in with an appropriate test account, and then start the evaluation.

Create conversations

A conversation is a test case that defines the exchange of messages you want to evaluate. You add a user question and can optionally add an expected agent response for your own review. The General quality test method doesn't compare the actual response with the expected response.

Note

General quality assessment evaluates responses against quality standards such as relevance and completeness. It doesn't compare responses against optional expected agent responses. Use expected responses as a reference during your manual review.

Create at least one conversation for each type of interaction your agent is expected to handle:

Scenario type What to include
In-scope answer A question the agent should answer directly from its knowledge or instructions
Ambiguity or clarification A vague or multi-part request where the agent should ask a clarifying question
Out-of-scope boundary A request outside the agent's defined scope where the agent should decline or redirect
Escalation A situation where the user needs human assistance or a specific contact the agent should provide
Connected-agent delegation A request the main agent should delegate to a specialized connected agent
Delegation boundary A similar request the main agent should handle without invoking the connected agent

These scenario types give you a useful starting point. If one type often receives a low score, review that part of the agent. For a connected agent, include both positive and negative routing tests. Confirm that the main agent delegates requests in the specialist's domain and retains requests outside that domain.

Run an evaluation

On the Evaluate tab, select New evaluation. Choose Or, write some questions yourself, and then use Add conversations > Write to create the test cases. In the Configure test set panel, enter a name, confirm General quality as the test method, review the signed-in User profile, and select Evaluate.

Copilot Studio saves each run separately. After you update the agent's instructions, knowledge, or tools, run the same test set again and compare the results with the earlier run.

Interpret evaluation results

Select an evaluation run to see results for each conversation. Each test case reports General quality as Pass or Fail, and the evaluation summary shows an aggregate score across conversations. This is an AI-based assessment, not a deterministic check of every required behavior or a comparison against your expected answers. Relevance and completeness are qualities to investigate during review, not separate dimension scores that you should expect to see.

When you review results:

  • Select a conversation to review the agent's response and the available result details.
  • Look for patterns: One failed test case might be an unusual case. Several failures in the same area might show a gap in the instructions, knowledge, or tool setup.
  • Review the response: Check the answer, any citations, and the available result details. The Evaluate tab doesn't show the model's private reasoning.
  • Verify execution when needed: If the evaluation results don't include execution details, test the request in Preview and inspect the activity trace. For a published interaction, review the session details in Monitor. Use this evidence to confirm the selected tool or connected agent, its result, and any error.
  • Check expected behavior manually: Verify boundaries, escalation, and connected-agent routing even when General quality reports a pass. An answer with the expected contact information doesn't, by itself, prove that the agent delegated the request.
Signal What to investigate
Irrelevant response Instructions may be too broad, too narrow, or missing context for this scenario
Incomplete response The response is partial; knowledge may be absent or not retrieved
Unsupported response The response isn't supported by the configured information; check instructions and knowledge
Wrong capability used The tool name or description may not match the scenario well enough to guide selection
Wrong agent selected Connected-agent descriptions might overlap, omit the specialist's domain, or conflict with the main agent's instructions
Delegation failure The specialist might be unpublished, unavailable for connection, or in a different environment, or the maker might lack access to configure the connection
Capability failure A connection, permission, or configuration problem may prevent a tool or workflow from completing

The maker needs ownership of or shared access to a specialist to connect it. When you share the main agent, users can invoke its connected Copilot Studio agents without separate sharing of each specialist. Access to data still depends on the tools' authentication settings and connection permissions. For details, see Connected-agent sharing and connector access.

Tip

After you correct an issue, run the same test set again and compare its responses and scores with the baseline run.

For more information, see Evaluate an agent and View evaluation results.