Introduction
Agents rarely stay the same after their first release. Over time, you refine an agent's instructions, update its knowledge sources, adjust its tools, and sometimes move it to a different AI model, and any of these changes can improve some responses while degrading others. Interactive testing, where you chat with the agent and inspect each reply, works well for exploring a conversation or debugging a single response. However, it doesn't scale to rechecking every important scenario after each change, and a handful of spot checks can't tell you whether a change helped or hurt overall. Evaluation answers that question.
Suppose you maintain an employee IT help agent that answers IT policy questions from its knowledge sources and uses a tool to look up the status of an employee's support ticket. Before you release each update, you need confidence that its policy answers and ticket lookups still work. Your next update moves the agent to a different AI model, so you also need evidence that the new model performs at least as well as the current one.
In this module, you explore:
- How to create a test set that represents an agent's real scenarios.
- How to run an evaluation on the Evaluate tab.
- How to verify agent behaviors that the General quality test method doesn't check, such as tool use.
- How to triage failed test cases to a root cause.
- How to validate an AI model migration by comparing evaluation results against acceptance gates.
By the end of this module, you'll be able to use repeatable evaluations to measure whether each change to your agent, including a change of AI model, improves or degrades its quality.
Note
Agent evaluation on the Evaluate tab is a production-ready preview feature, so its labels and options might change.