Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
[This article is prerelease documentation and is subject to change.]
Use the Evaluate tab to measure your agent's performance across a batch of test questions or conversations before you publish it.
Evaluate tab overview
When you open the Evaluate tab for the first time, it shows an empty state: Evaluate your agent's performance, with a Create a test set button. After you create one or more test sets, the tab instead shows an Evaluation sets table with Name, Run status, Test methods, and Last run by columns, and a New evaluation button.
Create a test set
On the Evaluate tab, select Create a test set (or New evaluation if you already have test sets).
Choose a Data type:
- Single response: Check how the agent responds to a specific question. Best for targeted capability testing.
- Conversation (preview): Check the quality of longer agent interactions with users. Best for general assessments.
Choose a Data source:
- Upload a CSV file: Drag a CSV file onto the upload area, or select browse. Use the provided CSV template to make sure the file contents and formatting are correct. Maximum file size is 5 MB.
- Quick question set: Generate 10 questions based on your agent's description, instructions, and capabilities.
- Full question set: Generate up to 100 questions, using knowledge or topics as a reference.
- Use your test chat conversation: Gather the inputs and responses from your current manual testing session. (Shown as disabled/unavailable in the confirmed screenshot; verify when it becomes available.)
- Or, write some questions yourself: Manually add questions and, optionally, expected responses.
Review your test case list. Use Add questions to add more, either by selecting Generate 10 questions or Write to add one manually.
Configure the test set
In the Configure test set panel:
- Data type: Shows the data type you selected (for example, Single response).
- Name: A required name for the test set.
- Test method: The criteria used to check the data. Select up to five methods. Select Add test method to open the Select a test method dialog, which offers:
- General quality: Responses meet quality standards, such as relevance and completeness. Doesn't compare to expected responses.
- Compare meaning: Answers have the same meaning as the expected response, even if worded differently.
- Custom: Checks and labels agent answers according to instructions you define.
- Keyword match: Looks for matching words and phrases.
- Text similarity: Measures how close the text of an agent's answer is to the expected response. This method might miss differences in meaning.
- Exact match: Agent answers must match the expected response completely.
- Agent version: Choose whether to evaluate the current draft or the published agent (for example, Current draft).
- User profile: Select Manage to open the Manage profile and connections dialog, where you select or add an authenticated account. Turn on Use these settings for all my evaluations to reuse the profile for future test sets. We recommend using an authenticated profile with all required connections so the evaluation can test all agent capabilities.
View results
After you run an evaluation, review the results to see how your agent performed. Reviewing evaluation results for an agent powered by the Customer Conversation harness is similar to reviewing results for an agent powered by the standard harness. To learn more, see Run evaluations and view results.