Configure and run evaluations on the Evaluate tab
A test set produces trustworthy results only when the evaluation runs under the right settings and you know what its scores can and can't show. In this unit, you learn how to configure, run, rerun, and download an evaluation on the Evaluate tab, and how to decide which evidence to add to its General quality results.
Configure the test set
When you create or edit an evaluation, the Configure test set panel on the right side of the Evaluate tab (preview) includes these options:
- Name: A required name for the evaluation. The name is how you find the evaluation later in the Evaluation dropdown.
- Test method: The method that scores the agent's responses. General quality is currently the only option.
- User profile: The authenticated profile that runs the evaluation and invokes the agent's tools and connections. By default, evaluations use the profile of the currently signed-in user.
Of the three options, the user profile needs the most thought, because the profile is what makes the agent's tools and connections testable during a run. Before you run conversations that depend on a tool, make sure the profile can use the connections those tools need.
If User profile shows a Manage option in the panel, set up the profile there:
- Under User profile, select Manage.
- Select the Select or add an account dropdown, and then select Add an account.
- Sign in with the account that should run the evaluation.
- Select Save to save the profile and its connection.
Otherwise, the evaluation runs as the account you're signed in with. To run it with a different user profile, sign out, and then sign in with that account before you start the evaluation.
For the IT help agent, a conversation such as "What's the status of my support ticket?" depends on the ticket lookup tool, so the evaluation needs a user profile that can use the tool's connection. The Evaluation summary for each run also records which user profile ran it, so you can check the profile behind a result when you compare runs.
Evaluations support agents set to use No authentication or Authenticate with Microsoft. Agents set to Authenticate manually aren't supported.
Run an evaluation
When the panel is set, select Evaluate to run the evaluation, or select Save to save it without running. The evaluation processes each conversation in the test set. Depending on the number of conversations, a run might take several minutes.
To run an evaluation that you saved earlier:
- On the Evaluate tab, select the evaluation in the Evaluation dropdown.
- Confirm that the test set contains at least one conversation.
- In the Configure test set panel, verify that the Name is set and the Test method is configured.
- Select Evaluate.
Evaluation runs use Copilot Credits, and the Evaluate tab shows the estimated credits for each run. The estimates in Preview and Evaluate are near-real-time approximations. The Monitor tab shows the final, accurate consumption within a few hours, after the usage is processed.
Rerun after an agent change
Each run is saved separately, so you can compare runs and see whether a change to the agent improved or degraded quality. After you change the agent's instructions, knowledge, or tools, rerun the evaluation to measure the impact:
- On the Evaluate tab, select the evaluation in the Evaluation dropdown.
- Under Recent results, select Evaluate test set again, or select the Evaluate test set icon within the test set.
- Compare the new run's results with earlier runs.
For example, after you revise how the IT help agent's instructions handle device policy questions, rerun the policy evaluation and compare the new run with the run from before the change.
Download a run for offline review
A downloaded session packages one run for use outside Copilot Studio. It contains the run's metadata: the evaluation name, run identifier, agent version, and test method. It also contains each turn's input, agent response, and score.
- On the Evaluate tab, open the evaluation.
- Under Recent results, select the run.
- Select Download.
Use a downloaded session to:
- Review agent responses offline when you don't have access to Copilot Studio.
- Share a specific run with a reviewer for feedback.
- Keep a record of a run alongside your other test artifacts.
The download reflects the run as it was when you downloaded it. To capture later changes to the agent, rerun the evaluation and download the new run.
What the test method measures
Currently, General quality is the only test method on the Evaluate tab, so every evaluation you run uses it. General quality is an AI-based assessment of whether the agent's responses meet quality standards, such as relevance and completeness. A Pass means a response met those standards.
Two limits define how much a pass proves:
- Expected responses: General quality doesn't compare responses to expected responses. A response can pass and still differ from the expected response you wrote.
- Knowledge and tool use: General quality assesses the response the agent returned. A pass doesn't show which knowledge sources the agent consulted or whether a tool ran.
Add evidence beyond the test method
With one test method available, choosing an evaluation method becomes a decision about which evidence to add for each behavior you need to confirm:
| Behavior to confirm | Evidence to add |
|---|---|
| Relevance and completeness of responses | The General quality result for each test case on the Evaluate tab |
| Accuracy against a known answer | Your own review of the agent response against the expected response |
| Which knowledge sources the agent consulted | Knowledge nodes and citations in the Preview activity trace |
| Whether a tool ran, with its inputs and results | Tool nodes in the Preview activity trace |
| Which model processed a conversation | Session details in Preview conversation history |
| Credit consumption | Estimates on the Evaluate tab, and final consumption on the Monitor tab |
The Preview and Evaluate tabs complement each other. Evaluate measures quality across many test cases and compares results across runs. Preview is where you debug a specific response or tool call. Its activity trace shows how the agent processed each message, including which knowledge sources it consulted, which tools it invoked, and what results the tools returned. To check a specific test case, send the same user message in Preview and inspect the trace.
For the IT help agent, a ticket status conversation might pass General quality with a response that reads as relevant and complete. If the trace for the same question in Preview shows no node for the ticket lookup tool, the lookup never ran. The pass alone doesn't reveal that gap.
Learn more in Use the activity trace to debug your agent.