Change the details of a test set

[This article is prerelease documentation and is subject to change.]

Use test sets to evaluate and refine how your agent handles real-world response scenarios. This article explains how to modify a test set's name and settings, manage the user profile and connections used for authentication, edit or delete individual test cases, and remove entire test sets or results. These steps help you ensure evaluations reflect the right user contexts and maintain secure access to knowledge sources and tools.

You have a choice of three test methods, also referred to as graders, to evaluate agent responses: quality, similarity, and text match. For more information about the different test methods, see Test methods.

Edit the details of a test set

You can change the basic details of a test set and which account it uses for authentication.

1. In your agent's Evaluation page, select the test set.

2. In the test set, select the test case you want to edit.

3. Optionally, change the name of the test set.

4. Select Manage profile to add an account, or select the current profile to manage connections or choose a different profile. Automated testing uses the authentication of the selected test account. If your agent has knowledge sources or tools that require specific authentication, you can select the appropriate account for your testing.

Note

When Copilot Studio generates test cases or runs an evaluation, it uses the authentication credentials of a connected account to access your agent's knowledge sources and tools. The generated test cases or results can include sensitive information that the connected account can access, and all makers who can access the test results can see this information.

Manage user profiles and connections

You can run test sets by using the authentication of a user's account. This authentication method allows your agent to access the knowledge sources and tools it needs to complete its goals.

You can also use profiles to simulate specific user experiences. For example, a director's user profile can access different knowledge sources than an intern's user profile, and the agent returns different results.

Test results show which user profile was used.

Note

When Copilot Studio generates test cases or runs an evaluation, it uses the authentication credentials of a connected account to access your agent's knowledge sources and tools. The generated test cases or results can include sensitive information that the connected account can access, and all makers who can access the test results can see this information.

Agent evaluations that use user authentication require access through the Microsoft Copilot Studio connector. If your admin turns off this connection, you can't run tests by using the evaluation tool. For more information, see Copilot Studio connectors and data groups.

To add or change a user profile in a test set:

1. Open a test set, then select Manage profile.

1. Select the User dropdown, then:

• Select the user account you want to use, or
• Select Add an account, then sign in to the account.

1. For connections with a warning, select the connection. Change the existing connection to a working connection, select Reconnect to try to reestablish the existing connection, or select Create new connection to try a new connection.

For new connections, enter the requested connection details, then select Connect.

1. When all connections are working, select Save.

Delete a test set or test result

You can delete a test case or test set that you no longer need.

1. Go to your agent's Evaluation page.
2. In the Test sets section, delete a test set by doing either of the following steps:
   • Select the three dots (…) on the test set card, and then select Delete.
   • Select the test set to open it, then select the three dots (…) in the Configure test set pane, and then select Delete.
3. In the Recent results section, delete a test result by doing either of the following steps:
   • Hover over the test result you want to delete, select the three dots (…) and then select Delete test.
   • Select the test result to open it, then select the three dots (…) in the Evaluation summary pane, and then select Delete.