Improve an agent from session transcripts
Improving a published agent depends on finding the problems that recur across sessions and confirming that each fix works. In this unit, you learn how to group session evidence into patterns, turn each pattern into evaluation conversations, and confirm a targeted change in production data.
Group session findings into patterns
Diagnosis often turns up the same finding in several sessions. A pattern is a problem that recurs across sessions and shares a common element. A pattern shows where one change can help many users, and it separates a recurring problem from a single unusual request.
Group the sessions and responses you investigate by the element they share:
- Request type: Users ask for the same kind of information or task, such as an order status or a policy detail.
- Tool: The same tool appears in failed sessions or in responses that users rate poorly.
- Knowledge source: Poorly rated responses draw on the same source.
- Channel: The problem appears in sessions from one channel only.
Combine reactions with transcripts when you group findings. A comment gives the user's view of the problem. The transcript of the same session shows which tools and knowledge sources the agent used.
Before you act on a pattern, check that it's ready for a fix:
- It recurs: Several sessions or users show the same problem.
- It shares a cause: The transcripts, and the activity trace when you reproduce a request on the Preview tab, point to the same finding, such as the same wrong source or the same tool error.
- It maps to one component: The cause points to one place to change, such as the agent's instructions, a knowledge source, or a tool.
When a group of sessions points to two different causes, split it into two patterns. Each pattern then leads to its own change and its own test conversations.
Download session data for wider review
Opening sessions one at a time works for a few examples. For a pattern that spans many sessions, download the session data. On the Monitor tab, select See all to open the sessions details page, and then download session data from that page.
Viewing and downloading transcripts depends on access that administrators control. If you can't view or download transcripts, check these requirements with your administrator:
- Security role: The Bot Transcript Viewer security role, which only administrators can grant when they share the agent.
- Environment setting: The environment setting that allows agent owners and editors to see session transcripts. When an administrator turns off this setting, the option to see or download transcripts isn't available.
Transcript data stays in Copilot Studio storage for up to 28 days and is then deleted automatically. To keep studying a pattern, download its session data while the sessions are still available.
Transcripts can contain sensitive data that the agent acts on or retrieves. Protecting that data is one reason administrators restrict transcript access or download for an environment. Follow your organization's data-handling requirements when you store downloaded session data or copy production requests into test conversations.
Learn more in Control transcript access and retention.
Turn each pattern into evaluation conversations
The Evaluate tab (preview) runs a named set of test conversations against your agent and scores the responses. When a pattern's requests become conversations in an evaluation, you can rerun them after every change. Requests from real sessions produce more representative test cases than invented ones. Conversations that you generate with AI are based on the agent's configuration, such as its description and instructions. Production requests show what users actually send.
Choose the conversations for each pattern:
- Requests from the pattern: Add one or more requests in the wording users sent, including different phrasings of the same request.
- A request that already works: Add at least one request that the agent handles well today and that your change shouldn't affect. If its result drops after the change, the change caused a regression.
To write conversations by hand:
- Open your agent in Copilot Studio and select the Evaluate tab.
- Select New evaluation.
- In the More ways to start section, select Or, write some questions yourself.
- In the Review your test cases list, select Add conversations > Write.
- Enter a user question for each conversation. Optionally, add an expected agent response.
- In the Configure test set panel, enter a Name that identifies the pattern, and then select Evaluate or Save.
To add a pattern to an existing evaluation, open the evaluation from the Evaluation dropdown, use the Add conversations dropdown, and then select Save. For many requests at once, upload a CSV file in the Data source section of a new evaluation. Select the CSV link to download a template that shows the correct format. The file can be up to 5 MB.
A few settings affect what the results tell you:
- Test method: General quality is currently the only test method. It scores whether responses meet quality standards such as relevance and completeness. It doesn't compare responses with the expected response you enter.
- User profile: The evaluation runs under the selected authenticated profile. If User profile appears in the panel, select Manage to add or choose an account whose connections let the agent's tools run.
- Credits: Evaluation runs use Copilot Credits, and the results show the estimated credits for each run. Actions that consume credits show a Uses Copilot Credits badge.
A General quality score rates the response, not which tool ran. To confirm which tool the agent invoked for a pattern about a tool, also send the request on the Preview tab and check the activity trace.
Learn more in Create a test set for an agent.
Confirm a change with evaluation and production data
Run the evaluation before you change the agent, so you have a baseline to compare against. Then make one targeted change in the component the pattern points to. When you change several things at once, a new score doesn't show which change caused it.
To rerun the evaluation on the Evaluate tab, select the evaluation from the Evaluation dropdown. Under Recent results, select Evaluate test set again. Each run is saved separately, so you can compare the new run with the baseline.
| Result after the change | What it suggests | Next step |
|---|---|---|
| The pattern's conversations improve, and the other conversations hold steady | The change addresses the pattern | Publish the change and confirm it in production |
| The pattern's conversations stay flat | The change doesn't target the cause | Revisit the diagnosis for the pattern |
| A request that worked before now scores lower | The change introduced a regression | Undo or adjust the change, and then rerun |
Agent responses and AI-based scores vary from run to run. Treat a small score difference with caution, and rerun the evaluation before you decide.
An evaluation measures response quality on your test conversations. It doesn't report reliability measures such as failed calls, and it can't show how real users respond. After you publish the change, confirm it on the Monitor tab:
- Period: Set the date filter to a period that starts after you publish the change, and allow time for new sessions to arrive.
- Signal: Check the same signal that revealed the pattern. For a tool pattern, look for fewer failed sessions that list that tool. For a request type, look for fewer negative reactions on those responses.
Keep the pattern's conversations in the evaluation after the production signal improves. If a later change brings the problem back, the next evaluation run can reveal the regression.
Learn more in View evaluation results for an agent.
Reflect: Think of one change you might make to an agent you manage or plan to publish. Which request that works today would you add to its evaluation to catch a regression from that change?