Interpret evaluation results and triage failures
An evaluation run is useful only when its results lead you to the right next change for your agent. In this unit, you learn how to read a run's results, compare runs, and triage failed test cases to a root cause and a next action.
Read the results of a run
To open a run's results, select the Evaluate tab (preview). Then, under Recent results, select an evaluation that has a prior test run.
The results have two parts. The Test run result table lists each conversation with the number of messages between the user and the agent, and its General quality score. The Evaluation summary describes the run as a whole:
- Score: The overall score of the conversations that passed the General quality test method.
- Duration: The time the evaluation took to complete.
- Test cases: The number of test cases in the evaluation.
- Data type: The type of test set. Conversation is the only data type available.
- User profile: The user profile that ran the evaluation.
- Credits used: The estimated Copilot Credits for the run, split between generating agent responses and scoring them.
Review a test case
Select a conversation in the Test run result table to expand its Test case details:
- User messages: The test messages sent to the agent.
- Agent responses: The agent's actual responses.
- General quality: The test method's result for the responses, either Pass or Fail.
- Credits used: The estimated Copilot Credits for the test case, which show you which scenarios consume the most resources.
Triage starts with the agent's actual responses, because a Fail result alone doesn't say what went wrong.
Compare runs to spot patterns
Each run is saved separately, so an evaluation's history shows how quality changes as you change the agent. On the evaluation detail page, review the list of runs with their scores and timestamps:
- Compare scores across runs to see whether a change improved or degraded performance.
- Compare the estimated Copilot Credits for each run to see consumption across runs and to estimate what a larger test suite might need.
- Look for patterns. If a category of conversations consistently scores low, that area might need more instructions or knowledge.
To analyze results outside Copilot Studio, export them. On the run results page, select the three dots (…) > Export test results. The CSV file includes all conversations, responses, and scores.
Classify each failure by root cause
A Fail can come from the test, the agent, or the platform. Evaluation failures map to three root cause types, and each type has a different owner:
| Root cause type | Owner | What it means |
|---|---|---|
| Evaluation setup | Evaluation author | The test case or the evaluation's configuration is wrong. The agent might be performing correctly. |
| Agent configuration | Agent builder | The agent produces an incorrect response that a configuration change can fix. |
| Platform limitation | Platform team | Platform behavior causes the failure, and configuration can't resolve it. |
When a run has many failures, decide where to start. Triage safety and compliance failures first, then failures in core business scenarios, then failures in the lowest-scoring evaluation, and then failures that recur across runs. Within a large group of failures, review a few first. If they share a root cause, one fix can resolve many failures at once.
Verify the evaluation setup first
Check the evaluation before you investigate the agent, so you don't spend effort changing an agent that might be working correctly.
Start with the health of the run. Problems with dependencies can produce failures that look like agent or evaluation problems. Confirm these conditions for the run:
- The agent's knowledge sources are accessible.
- Tools and connectors don't return errors, timeouts, or rate-limit responses.
- The evaluation's user profile has valid authentication for the connections that the agent's tools need.
- The environment you evaluated in matches the agent configuration you intended to test.
If a dependency is unhealthy, fix it and rerun the evaluation before you triage anything.
Next, work through these questions for each failed case, in order. Stop when a question gives you an answer.
- Would a real user accept the response? If yes, classify the failure as an evaluation setup issue, and leave the agent unchanged. General quality is an AI-based assessment, so rerun the evaluation to check whether the verdict is consistent.
- Does the test case reflect realistic user input? If not, edit the conversation so it matches how users actually ask. For example, a four-paragraph request is unrealistic when real users type a few words.
- Is General quality the right check for this behavior? If the case exists to confirm a tool lookup or a specific fact, the method doesn't fit the case. Treat that mismatch as an evaluation setup issue, and check the evidence that fits the behavior.
When the response isn't acceptable and both the test case and the method fit, the evaluation is valid, and the failure is an agent problem to diagnose.
For the IT help agent, suppose several ticket status cases fail in the same run, and you find that the connection the lookup tool needs is expired for the account that ran the evaluation. That failure is an evaluation setup issue. Restore the connection and rerun, and leave the agent's instructions as they are.
Diagnose the agent configuration
With the setup verified, map the failure to the part of the agent that most likely caused it:
| What you see | Where to look |
|---|---|
| Low General quality results | The agent's instructions |
| Missing or incorrect tool use | Tool descriptions, and instructions that say when to use each tool |
| Incorrect information | Whether the relevant knowledge sources are added and configured correctly |
Confirm the mapping with evidence before you edit. Send the failing case's user message in Preview and inspect the activity trace. A Knowledge node shows which sources the agent consulted, and a Tool node shows the tool the agent invoked and the result it returned. When a tool runs as expected but returns an error or incorrect data, the failure is in the tool or its backend system, so fix the tool.
For the IT help agent, a policy answer might fail because it gives an outdated device policy. If the trace shows the agent consulted the right source and that document is out of date, the fix is to update the source document.
Sometimes the evaluation is valid, but reasonable configuration changes don't improve the result. A failure that persists across several instruction and configuration variations suggests a platform limitation. Document what fails, what you tried, and the evidence that configuration isn't the cause. Apply a workaround where you can, and escalate with that evidence to the platform team.
Confirm a fix with a rerun
A diagnosis stays a hypothesis until a rerun confirms it. After each fix, rerun the evaluation and compare the new run with the previous one. If the failure persists, triage it again.
When a failure has more than one cause, fix them in order. Fix the evaluation setup first, so the next run gives a clean signal. Then fix the agent configuration. Document a platform limitation only after the first two are addressed. Rerun after each change before you move on.
Results can vary between runs of the same set, because both the agent's responses and the General quality assessment are AI-based. Run important scenarios more than once before you conclude that a change helped or hurt. A test case that passes incorrectly hides a quality gap, so also read a sample of passing responses, especially for cases that failed before your change and pass now.
For the IT help agent, after you restore the connection, rerun the ticket status evaluation more than once. If the lookup cases pass consistently, the setup fix is confirmed. If some still fail, triage those cases again.
Learn more in Understand the goals of the triage and remediation framework.