Preview and refine an agent
The Preview tab provides an interactive environment for testing an agent before publication. Use it to establish a baseline, observe how the agent applies instructions and knowledge, and determine what to refine.
Build a baseline test set
Start with a small set of prompts that represent different behavior classes:
| Test type | Purpose | Benefits agent example |
|---|---|---|
| In scope | Confirm that the agent answers a supported request from approved knowledge | When can I change my medical plan? |
| Paraphrase | Confirm that different wording produces consistent behavior | Is there a time each year when I can switch health plans? |
| Ambiguous | Confirm that the agent requests information it needs | Tell me about coverage. |
| Missing information | Confirm that the agent doesn't invent unsupported details | What is the exact employee premium next year? |
| Out of scope | Confirm that the agent declines or redirects unrelated requests | Write a sales proposal for me. |
| Escalation | Confirm that the agent refers decisions to the correct destination | Am I eligible to add my partner to my plan? |
Record the expected result for each prompt before testing. An expected result describes the required behavior rather than exact wording. This approach allows natural variation while still identifying failures.
Observe the result
Open the agent and select Preview. Select New chat for each baseline prompt so earlier responses don't affect later tests. This practice isolates test cases. Changes you make on Build normally apply on the next turn of an active conversation, so a new chat isn't required just to pick up configuration changes. Review:
- Whether the response follows the agent's scope and tone.
- Whether factual claims match the approved knowledge.
- Whether the response acknowledges missing information.
- Whether the agent asks useful clarification questions.
- Whether boundaries and escalation instructions are followed.
- Whether source information or citations support the answer.
Use the activity information in Preview to check which knowledge source the agent used. A response that sounds correct can still be a problem if the agent uses the wrong source or doesn't follow its limits.
Refine one behavior at a time
When a test fails, identify the smallest instruction or source change that addresses the observed behavior. Make the rule more specific, save the agent, start a new test conversation, and rerun the same prompt. Then rerun nearby baseline cases to check that the change didn't introduce a regression.
For example, if the agent estimates eligibility, change Don't make eligibility decisions to Don't state or imply that an employee qualifies for a benefit. Explain only the published criteria and direct the employee to Human Resources for a personal decision. The revised instruction defines the prohibited behavior and the required alternative.
Preview testing is iterative. Test a prompt, compare the response with the expected result, make one focused change, and test the prompt again. Repeat this process until the agent responds consistently without affecting behavior that already works as expected.
Create an evaluation
Agent evaluation is a production-ready preview feature. Use the Evaluate tab to save a set of test conversations that you can run again after changing the agent. Select New evaluation, select Or, write some questions yourself, and then select Add conversations > Write to add your test questions. In Configure test set, enter a name, review the General quality test method and the User profile, and then select Evaluate.
Note
Generating test conversations and running evaluations can consume Copilot Credits. Confirm your environment's approved capacity before you run an evaluation. Preview also displays an estimated credit total for the current test session.
Evaluations and tool connections run with the currently signed-in user's profile. You can't select a different identity while signed in. To evaluate as another user, sign out, sign in with that user's account, and then start the evaluation.
Currently, General quality is the only available test method. It uses AI to assess qualities such as relevance and completeness; it doesn't compare responses with expected answers. Review the results for each conversation, and manually check the responses against your expected behaviors, especially eligibility boundaries and escalation requirements. A quality score doesn't replace those checks.
For more information, see Test an agent and Evaluate an agent.
Note
We recognize that different people like to learn in different ways. You can choose to complete this module in video-based format or you can read the content as text and images. The text contains greater detail than the videos, so in some cases you might want to refer to it as supplemental material to the video presentation.