Summary

  • 2 minutes

You now have a repeatable way to measure whether each change to an agent powered by the GitHub Copilot harness, including a move to a different AI model, improves or degrades its quality.

What you learned

In this module, you learned how to:

  • Create a test set by uploading a CSV file, generating conversations with AI, or writing them yourself, and cover business-critical requests, difficult cases, and tool lookups.
  • Configure and run an evaluation on the Evaluate tab (preview) with a user profile that can use the agent's connections, and rerun it after you change the agent's instructions, knowledge, or tools.
  • Verify behaviors that the General quality test method doesn't check, such as using the Preview activity trace to confirm tool use.
  • Read a run's evaluation summary and test case details, compare runs, and trace each failure to an evaluation setup, agent configuration, or platform cause before you change the agent.
  • Validate an AI model migration by baselining the current model, running the same test set against a candidate model in a nonproduction copy, and comparing both runs with acceptance gates.

Learn more