Run and validate an AI model migration for Copilot Studio agents

The following sections apply to every model change, whether it's a proactive upgrade or a response to a retirement. Define the gates first, capture a baseline from the current model, and then work through the phases.

Define migration acceptance gates

Define acceptance criteria before evaluating the candidate model, so that the agent evaluation produces a decision instead of only a set of scores. Include:

  • Minimum overall pass rate
  • Required pass rate for business-critical scenarios
  • Critical failures that block migration regardless of aggregate score
  • Allowed latency and reliability variance
  • Safety, compliance, and regional-processing approval
  • Acceptable consumption or cost impact
  • Required owner and release approver signoff

Compare the current and candidate model by using the same agent configuration, test data, test set, user profiles, and environment assumptions. Investigate individual regressions instead of relying only on an average score.

Model outputs are probabilistic. Run important scenarios more than once when variability can affect the decision.

Establish a reusable evaluation baseline

Before you change the model, create a test set that represents the agent's business-critical and high-frequency scenarios. Run the test set against the current production model to establish a baseline.

Use both test chat and agent evaluation:

The test methods available to you depend on the agent's harness, so confirm them before designing the test set. Learn more in Confirm which test methods your agent supports.

Evaluate the full agent behavior

Don't approve a replacement model just because its aggregate pass rate is similar to the current model.

Cover the following areas. Each one is a behavior that commonly changes when the model changes, so a test set that omits an area can't detect a regression in it.

Evaluation area What a model change can break How to check it
Answer quality Relevance, completeness, accuracy, clarity, and consistency on the questions the agent receives most often. General quality
Groundedness and knowledge The model answers from training data instead of the configured knowledge source, drops citations, or handles incomplete or conflicting sources differently. General quality
Abstention The model answers an out-of-scope question instead of declining or escalating. General quality, or Custom with Answered and Declined labels
Instruction following Instructions that the model reliably followed are skipped, such as escalation rules, boundaries, prohibited behaviors, or a mandatory disclaimer. Keyword match on required phrases, or Custom with instruction-specific labels
Fixed values and output format A precise value such as a phone number, code, or ID is paraphrased or invented, or the output shape changes and breaks a downstream parser or channel integration. Exact match, or Keyword match on the required format
Tool selection and restraint The model selects a different tool, calls none, or calls a tool when no tool is needed. Tool use with the expected tools or topics defined, plus a review of the activity map
Tool inputs and sequencing Parameters are extracted, formatted, or defaulted differently, or steps in a multitool task are reordered, merged, or dropped. Tool use with all expected tools, plus General quality
Confirmation and failure handling The model stops asking for confirmation before a consequential action, or a tool error is surfaced differently instead of reported. Custom with confirmation labels, plus Keyword match on the expected error language
Multiturn behavior Context from earlier turns is lost or reinterpreted, or clarification, topic changes, and recovery are handled differently. General quality on a conversation test set
Messy and adversarial input Weaker handling of typos, fragments, and unclear intent, or a different response to prompt injection and role override attempts. General quality, plus Custom with Refused and Complied labels
Safety and compliance Harmful-content handling, data access, permissions, regional processing, and responsible AI requirements. Custom labels, alongside a responsible AI review
Latency and reliability Response time, timeouts, variability, failed calls, and retries under representative conditions. Not reported by evaluation. Measure in test chat and production monitoring.
Consumption and cost Copilot Credit consumption or other model-related costs for representative scenarios. Not reported by evaluation. Measure in capacity and consumption reporting.
Languages and channels Quality and behavior across supported languages, user profiles, and deployment channels. Run the test set for each language, user profile, and channel that matters

Pay particular attention to instruction following and latency. A candidate model can improve answer quality but introduce slower responses, different tool-selection behavior, or failures in instructions that were reliable with the previous model.

Confirm which test methods your agent supports

The test methods available to your agent depend on its harness.

Learn more in:

Build and maintain the test set

  • Cover business-critical happy paths and high-volume requests first, then difficult cases: edge cases, adversarial inputs, requests that should be refused or escalated, tool failures, long conversations, and multilingual requests.
  • Start from real traffic. Analytics themes and recorded conversations produce more representative test cases than invented ones.
  • Favor coverage over polish. A larger set of imperfect cases finds more regressions than a small set of perfectly worded ones.
  • Score the current model first, then the candidate. The comparison, not the absolute number, tells you whether the migration is safe.
  • Keep the set with the agent in source control, and rerun it unchanged for every model change.
  • Split the set by risk area. A test set powered by the standard harness holds up to 100 test cases, so use separate sets for knowledge accuracy, tool behavior, and safety-sensitive scenarios.
  • Export results. Evaluation results are retained for 89 days, so export them to CSV to keep a record of what each model scored at migration time.
  • Convert production incidents and user feedback into new regression tests.

Learn more in Design and operationalize agent evaluation.

Note

Agent evaluation measures correctness and performance, not AI ethics or safety problems. An agent can pass every test case and still produce an inappropriate answer. Use responsible AI reviews and content safety filters alongside evaluation.

Automate recurring evaluations

Copilot Studio supports running evaluations through the Power Platform API, so you can integrate model validation into release workflows and continuous integration pipelines. For a large agent estate, automation is what makes repeated candidate testing sustainable instead of manual. Learn more in Automate evaluations with the Power Platform API.

Update the agent artifacts a model change affects

A model change rarely affects only the model setting. Translate the applicable provider guidance into the agent artifacts you control, and then validate every change against your evaluation baseline. A remediation that isn't retested is a guess.

Artifact Typical change
Agent instructions Make implicit expectations explicit, remove workarounds written for the previous model, restate boundaries and escalation rules and prohibited behaviors unambiguously, resolve conflicting instructions, and calibrate how much independent action the agent should take.
Topic and node instructions Apply the same treatment at the topic level, and verify that generative answer nodes still behave as expected.
Tool and action descriptions Rewrite for clarity. This description is what the model reads to decide whether and when to call a tool, and vague descriptions cause both over-calling and under-calling.
Input and output parameter descriptions Tighten the format, units, examples, and required or optional semantics so that parameter generation stays correct.
Knowledge source configuration Recheck source selection, scoping, and grounding instructions, and verify citation behavior.
Response formatting instructions Restate the required structure, length, disclaimers, and exact values explicitly, because newer models change default verbosity.
Confirmation and safety gates Reassert explicit confirmation requirements before consequential actions.
Downstream consumers Update Power Automate flows, adaptive cards, channel integrations, and any parser that reads agent output.
The test set itself Add the new failure patterns discovered during the migration.

Migration phases

The following phases put the preceding sections in execution order. Use them as the working plan for a single agent's model change, whether the change is proactive or driven by a retirement.

Phase 0: Prepare

  1. Confirm the candidate model is valid against the prerequisites: generally available or default, available in-region, cross-geo posture acceptable, administrator-enabled, and the right use category for the agent's purpose.
  2. Read the model provider's upgrade guidance and note the instruction and tool changes it implies.
  3. Identify the affected agents from the inventory, recording environment, owners, and criticality.
  4. Confirm which evaluation test methods the agent's harness supports.
  5. Define and approve the migration acceptance gates.

Phase 1: Baseline the current model

  1. Build or refresh the regression test set so it covers business-critical and high-frequency scenarios.
  2. Run the test set against the current production model to establish the baseline. Do this while the current model is still in place, because after the model changes the baseline can't be reconstructed.
  3. Record latency and Copilot Credit consumption separately. Evaluations don't report them.
  4. Export the results to CSV to preserve the record beyond the retention window.

Phase 2: Evaluate the candidate in a nonproduction environment

  1. Prepare a nonproduction copy of the agent, following the Copilot Studio guidance for application lifecycle management and agent testing. Configure the environment so it represents production:

    • Use the same agent instructions, topics, knowledge configuration, tools, flows, connectors, languages, and security assumptions.
    • Use representative test identities and connections.
    • Apply the same data policies and relevant administrator controls.
    • Confirm regional availability and whether cross-geo data movement is required.
    • Record any differences between test and production that could affect the result.

  2. Change the model. Go to the agent's Overview page and select the candidate primary model in the Model section. Separate settings exist for deep reasoning, generative responses, and the prompt builder, so check whether the agent uses those capabilities and whether they need to change too.

  3. Rerun the same test set, unchanged, against the candidate model.

  4. Compare the two runs against the acceptance gates to identify improvements and regressions.

  5. Inspect orchestration qualitatively in test chat, using the activity map to confirm which tools were selected, in what order, and with which parameters.

  6. Measure latency and consumption for the candidate and compare them against the baseline.

Phase 3: Remediate

  1. Update the agent artifacts the model change affects, then rerun the test set against the remediated agent. Learn more in Improve agents using evaluation-driven triage and remediation.
  2. Iterate until the acceptance gates are met, or conclude that the candidate model isn't suitable and document why. Deciding not to upgrade is a legitimate, evidence-based outcome.

Phase 4: Approve and deploy

  1. Get signoff on the acceptance gates from the agent owner and release approver, including safety and compliance approval when cross-geo or external models are involved.
  2. Deploy through the established ALM process, promoting the solution from test to production. Don't manually edit the production agent.
  3. Stage the rollout when the channel allows it. Publish to a pilot audience or a single channel first, observe the result, then broaden.

Phase 5: Monitor and close

  1. Monitor production behavior against the acceptance gates.
  2. Add newly discovered failure patterns to the regression test set.
  3. Close the migration only after the acceptance criteria remain satisfied in production.
  4. Preserve the evaluation evidence and the migration decision record for audit and for the next lifecycle event.

Important

The rollback path for a model change is redeploying the previously validated solution version, which is slower than a configuration toggle. This difference makes the evaluation gate in phase 2 so important. Catching a regression before deployment is less expensive than reversing one afterward.

Monitor after migration

Model lifecycle work continues after deployment. Monitor:

  • Agent evaluation results and critical scenario pass rates.
  • Production analytics, transcripts, activity, errors, and user feedback.
  • Instruction-following and tool-selection failures.
  • Latency, timeouts, and reliability.
  • Consumption changes.
  • Safety, compliance, and regional-processing concerns.

When production monitoring identifies a new failure pattern, add a representative case to the regression test set. This practice improves the next model evaluation and turns production learning into a durable quality baseline.

Environment-level telemetry emits OpenTelemetry GenAI spans to Application Insights, including the model used for each agent invocation. Use it to confirm which model production traffic actually runs on, and to compare tool selection and reliability before and after a migration.

  • Last updated on