Validate an AI model migration against acceptance gates
Moving your agent to a different AI model is a decision, and repeatable evaluations supply the evidence for it. In this unit, you learn how to set acceptance gates, baseline the current model, test a candidate model with the same test set, and decide whether to adopt it.
Define acceptance gates first
An acceptance gate is a threshold or condition that the candidate model must meet before you adopt it. Whether the change is a proactive upgrade or a response to a model retirement, define acceptance gates before you evaluate the candidate model. Gates turn the evaluation's results into a decision instead of only a set of scores.
Set a gate for each of these areas:
- Overall pass rate: The minimum share of test cases that pass.
- Business-critical scenarios: The pass rate these scenarios must reach.
- Blocking failures: Failures that stop the migration regardless of the overall score.
- Latency and reliability: The variation in response time and reliability you accept.
- Safety and compliance: The approvals the change needs, including regional data processing.
- Consumption: The change in Copilot Credit consumption you accept.
- Signoff: The agent owner and release approver who approve the result.
Compare the current and candidate models with the same agent configuration, test set, user profile, and environment assumptions, so the model is the only difference between the runs.
For the IT help agent, you might treat any skipped ticket status lookup as a blocking failure, so the migration stops even when the overall pass rate holds.
Cover what a model change can break
A candidate model can match the current model's overall score and still behave differently. A newer model can interpret instructions more literally, choose tools differently, and change the length and format of its responses. A test set that omits a behavior can't detect a regression in it. Before you baseline, choose the test set you rerun for this migration, your regression set, and confirm that it includes conversations for each behavior that matters to your agent:
| Behavior | What a model change can break |
|---|---|
| Answer quality | Relevance, completeness, and accuracy on frequent questions |
| Groundedness | Answers from the model's training data instead of your knowledge sources, or dropped citations |
| Out-of-scope requests | Answering a request the agent should decline or escalate |
| Instruction following | Skipped boundaries, escalation rules, or required disclaimers |
| Fixed values and format | Invented or paraphrased codes and IDs, or a changed output structure |
| Tool selection and inputs | A different tool, no tool, or different parameters |
| Confirmation and failure handling | No confirmation before a consequential action, or tool errors reported differently |
| Multiturn context | Lost or reinterpreted context from earlier turns |
| Messy input and languages | Weaker handling of typos, fragments, unclear intent, or other languages |
General quality checks whether responses meet quality standards such as relevance and completeness. For the other behaviors, plan the evidence you add, such as your review of responses against expected responses and the knowledge and tool nodes in the Preview activity trace.
Baseline the current model
Run the regression set against the current model while it's still in place. After the model changes, you can't reconstruct the baseline.
- On the Evaluate tab (preview), select the evaluation that holds the regression set, and then select Evaluate.
- On the run results page, select the three dots (…) > Export test results.
The CSV file keeps a record of what the current model scored, so you can compare the candidate with it later.
Record latency and consumption separately from the score. The Duration in the evaluation summary is how long the whole run took, not the agent's response time. For response time, measure representative conversations in Preview, where the activity trace shows processing time for each step and conversation history shows each session's duration. For consumption, note the estimated Copilot Credits that Evaluate shows for the run. These values are estimates, and the Monitor tab shows the final consumption.
Test the candidate model in a nonproduction copy
Before you select a candidate, check its release status. Experimental and preview models are for exploring and testing, not for production. Their performance, response quality, latency, and consumption can vary, and they can time out or be unavailable. Administrators also control whether makers can add preview, experimental, and external models to agents, so a candidate in one of those groups might need an administrator to allow it first.
Test the candidate in a nonproduction copy of the agent that represents production. Give the copy the same instructions, knowledge, tools, and connections as production. Use representative test identities, and apply the same data policies and administrator controls. Create the copy by following your organization's application lifecycle management process.
To change the model in the copy:
- Open the agent copy in Copilot Studio.
- Select the Build tab.
- In the components panel, select the Model list.
- Select the candidate model.
- Select Save to apply the change.
Then run the same regression set against the candidate, unchanged, with the same user profile. To confirm which model answered, test a conversation in Preview and open it in conversation history, where the session details show the model that processed the request.
Compare the runs against the gates
Compare the candidate's results with the exported baseline, gate by gate. A similar overall score isn't enough to approve the candidate. Investigate individual regressions. Find the test cases that passed on the current model and fail on the candidate, and read both responses. Model outputs are probabilistic, so run important scenarios more than once before you conclude that a result changed.
Then check the behaviors that the score doesn't cover. Send key conversations in Preview, and confirm in the activity trace which tools the candidate selected, in what order, and with which inputs. Measure latency and consumption the same way as for the baseline.
For the IT help agent, suppose the candidate's overall score is close to the baseline. In Preview, the activity trace for two ticket status questions shows no tool node, so the candidate answers without running the lookup. That result is a blocking failure that the overall score alone doesn't reveal.
Remediate regressions
Update the agent artifacts that each regression points to, and then rerun the same regression set. A fix that you don't retest is only a guess.
Common artifacts to update include:
- Instructions: Make implicit expectations explicit, and restate boundaries and escalation rules.
- Tool descriptions: Clarify when to use each tool. The model reads the description to decide whether and when to call it.
- Input descriptions: Tighten the format, units, examples, and required or optional status of each input.
- Knowledge configuration: Recheck which sources the agent uses, and verify citation behavior.
- Response formatting instructions: Restate the required structure, length, and exact values, because newer models can change default verbosity.
- The regression set: Add the failure patterns you find during the migration.
For the IT help agent, you might rewrite the ticket lookup tool's description to state that the agent uses it for every ticket status request. Then you rerun the regression set and check the trace for those questions again.
Iterate until the candidate meets every gate, or conclude that it isn't suitable and document why. Deciding not to upgrade is a legitimate outcome, and your baseline and gate results are the evidence behind it.
Sign off on the migration
When the candidate meets the gates, get signoff from the agent owner and the release approver, including safety and compliance approval where the change requires it. Evaluation measures correctness and performance. An agent can pass every test case and still produce an inappropriate answer, so pair evaluation with a responsible AI review.
Deploy the change through your organization's release process. Where the channel allows it, stage the rollout, for example to a pilot audience first. After deployment, monitor the agent against the same gates, and add each new failure pattern to the regression set so your next model evaluation starts from better coverage.
Learn more: