Create a test set for an agent (preview)

[This article is prerelease documentation and is subject to change.]

A test set is a collection of conversations you use to evaluate your agent's performance. Each conversation represents a scenario you want your agent to handle.

Important

Prerequisites

An agent created and saved with the GitHub Copilot harness. See Create an agent (preview).

Start a new evaluation

To start a new evaluation:

  1. Open your agent in Copilot Studio.
  2. Select the Evaluate tab.
  3. Select New evaluation.

Select from one of these options:

Option 1: Upload conversations from a CSV file

  1. On the Evaluate tab, in the Data source section, drag a CSV file onto the upload area or select it to browse for a file.

    Tip

    Select the CSV link to download a template that shows the correct file format. The maximum file size is 5 MB.

  2. Review the imported conversations and adjust them as needed.

  3. In the Configure test set panel on the right, enter a Name for the evaluation.

  4. Select Evaluate to run the evaluation, or select Save to save it without running.

Option 2: Generate conversations with AI

  1. On the Evaluate tab, in the More ways to start section, select Quick conversation set.
  2. Wait while the system generates 10 conversations based on the agent's description, instructions, and topics.
  3. Review the generated conversations in the Review your test cases list.
  4. In the Configure test set panel, enter a Name, and then select Evaluate or Save.

You can also generate more conversations from the evaluation view. Select Add conversations > Generate 10 conversations. After you run an initial evaluation, select Add conversations > Generate 25 conversations or Generate 50 conversations to generate more.

Option 3: Write conversations manually

  1. On the Evaluate tab, in the More ways to start section, select Or, write some questions yourself.
  2. In the Review your test cases list, select Add conversations > Write to manually create a test case.
  3. Add a user question for each conversation. Optionally, add an expected agent response.
  4. In the Configure test set panel, enter a Name and select Evaluate or Save.

Configure the test set

When you create or edit an evaluation, the Configure test set panel includes these options:

  • Name: A required name for the evaluation.
  • Test method: The method that scores responses. General quality checks whether responses meet quality standards like relevance and completeness.
  • User profile: The authenticated profile that runs the evaluation. Select Manage to set up which profile runs the tests.

Edit conversations in a test set

To edit conversations in a test set:

  1. On the Evaluate tab, open an existing test set from the Evaluation dropdown.
  2. Select a conversation to edit its messages.
  3. Use the Add conversations dropdown to add more conversations.
  4. Select Save when you're done.
  • Last updated on