Create test set conversations that represent real scenarios

  • 10 minutes

An evaluation checks your agent against a fixed collection of sample conversations, called a test set, so its results are only as useful as those conversations are representative of what your users actually ask. In this unit, you learn how an evaluation works, what a test set contains, three ways to add conversations to one, and how to choose conversations that cover your agent's real scenarios.

How an evaluation works

An evaluation follows a repeatable cycle on the Evaluate tab (preview):

  1. Create a test set: Collect conversations that represent the scenarios you want the agent to handle.
  2. Configure and run the evaluation: Name the evaluation, confirm its test method, and run it. Copilot Studio sends each conversation to the agent and scores the responses.
  3. Review the results: Check the overall score, and then open individual conversations to see where the agent falls short.
  4. Change the agent and run again: After you adjust the agent's instructions, knowledge, or tools, run the same test set again and compare the new results with the earlier ones.

Because you reuse the same test set in every run, the test set is the fixed reference that makes runs comparable. The rest of this unit covers how to build it.

What a test set holds

A test set is a collection of conversations that you use to evaluate your agent. Each conversation represents one scenario you want the agent to handle. Copilot Studio also calls a conversation a test case, so you see both terms on the Evaluate tab.

A conversation contains user messages and, optionally, an expected agent response. An evaluation is a named test set combined with a test method.

Expected responses are optional, and their role depends on the test method. Currently, the only test method is General quality, an AI-based assessment of whether responses meet quality standards such as relevance and completeness. General quality doesn't compare the agent's responses to expected responses. Treat an expected response as a reference for your own review. It records what a good answer looks like, which gives you something specific to compare the agent's actual response with.

Choose how to add conversations

Every path starts when you select New evaluation on the Evaluate tab. If the agent doesn't have an evaluation yet, select Create your first evaluation instead. Actions that consume Copilot Credits display a Uses Copilot Credits badge, so you can see which actions use credits before you run them.

Each path fits a different source of test questions:

Path Where you start Fits when
Upload a CSV file Data source section You already have questions, such as real user questions or an existing regression list.
Generate with AI More ways to start > Quick conversation set You want broad starting coverage quickly, based on how the agent is configured.
Write manually More ways to start > Or, write some questions yourself You need specific cases, such as boundary requests, tool lookups, or failures you saw before.

The paths also work together. The Add conversations menu lets you generate more conversations or write your own in a set you already started. For an employee IT help agent, you might upload policy questions from your help desk history, generate a quick set for breadth, and then write targeted cases for ticket status lookups and out-of-scope requests.

Upload conversations from a CSV file

A CSV upload fits when your test questions already exist outside Copilot Studio, such as questions gathered from real users or a regression list you already maintain.

  1. On the Evaluate tab, select New evaluation.
  2. In the Data source section, drag a CSV file onto the upload area, or select the area to browse for a file.
  3. Review the imported conversations and adjust them as needed.
  4. In the Configure test set panel, enter a Name for the evaluation. Then select Evaluate to run it, or select Save to save it without running.

Before you prepare the file, select the CSV link to download a template that shows the correct file format. The maximum file size is 5 MB.

Generate conversations with AI

A Quick conversation set gives you a starting set without writing any questions. Copilot Studio generates 10 conversations based on the agent's description, instructions, and topics.

  1. On the Evaluate tab, select New evaluation.
  2. In the More ways to start section, select Quick conversation set.
  3. Wait while Copilot Studio generates the conversations, and then review them in the Review your test cases list.
  4. In the Configure test set panel, enter a Name, and then select Evaluate or Save.

To grow the set, select Add conversations > Generate 10 conversations. After you run an initial evaluation, the same menu also offers Generate 25 conversations and Generate 50 conversations.

Review each generated conversation before you run the set, and compare it with requests your users actually make. Generation works from the agent's own configuration. Pair generated conversations with real questions and targeted cases from the other two paths, so the set also reflects how users phrase requests and where the agent failed before.

Write targeted conversations manually

Manual writing gives you precise control over individual cases. Use it for boundary requests, requests that need the agent's tool, and past failures you want every later run to check.

  1. On the Evaluate tab, select New evaluation.
  2. In the More ways to start section, select Or, write some questions yourself.
  3. In the Review your test cases list, select Add conversations > Write.
  4. Enter a user question. Optionally, enter an expected agent response.
  5. Repeat the previous two steps for each conversation you need.
  6. In the Configure test set panel, enter a Name, and then select Evaluate or Save.

For the IT help agent, you might write "What's the status of my support ticket?" to cover the ticket lookup, with an expected response that gives the ticket's current status. You might also write "Can you approve my expense report?" as an out-of-scope request, with an expected response that declines and describes what the agent can help with.

Plan coverage for real scenarios

The paths decide how conversations get into a set. Coverage decides which scenarios the set represents, and a test set that omits an area can't detect a regression in it.

Start with business-critical and high-volume requests, the ones that matter most and the ones users send most often. Then add difficult cases: edge cases, requests the agent should refuse or escalate, and requests where a tool can fail. Also include both kinds of work the agent does. A set of only policy questions never exercises the ticket lookup, so add conversations that need the tool alongside conversations the agent answers from its knowledge sources.

As the set grows, these practices keep it representative:

  • Start from real traffic: Recorded conversations and real user questions produce more representative cases than invented ones.
  • Favor coverage over polish: A larger set of imperfect cases finds more regressions than a small set of perfectly worded ones.
  • Split sets by risk area: Keep separate evaluations for distinct areas, such as policy answers and ticket lookups, so each run's results describe one area.
  • Turn incidents into cases: When an incident or user feedback reveals a failure, add a conversation for it so later runs check it.

Edit a saved test set

A test set changes as you learn more about your agent and its users. To edit a saved set:

  1. On the Evaluate tab, open the test set from the Evaluation dropdown.
  2. Select a conversation to edit its messages.
  3. To add more conversations, use the Add conversations dropdown.
  4. Select Save when you're done.

When you compare runs, such as before and after a model change, rerun the same test set unchanged so that both runs test the same scenarios.