Automate AI evaluations with Microsoft Foundry and GitHub Actions

Module
9 Units

Intermediate

Data Scientist

AI Engineer

DevOps Engineer

Developer

Azure

Azure AI services

GitHub

Learn how to implement automated evaluations for AI agent responses using Microsoft Foundry evaluators, create evaluation datasets from production data and synthetic generation, run batch evaluations with Python scripts, and integrate evaluation workflows into GitHub Actions for continuous quality assurance.

Learning objectives

By the end of this module, you're able to:

Explain why automated evaluations complement human evaluations in AI quality assurance.
Select evaluators that align with human evaluation criteria for validation.
Create evaluation datasets with appropriate composition for comprehensive testing.
Implement batch evaluations using Python scripts with Microsoft Foundry.
Integrate automated evaluation workflows into GitHub Actions for continuous testing.

Prerequisites

Before starting this module, you should be familiar with fundamental generative AI concepts and services in Azure. Consider completing the Fundamentals of Generative AI module first. Basic familiarity with Python and GitHub workflows is recommended.

Get started with Azure

Choose the Azure account that's right for you. Pay as you go or try Azure free for up to 30 days. Sign up.

Introduction min
Understand why automated evaluations matter min
Align evaluators with human criteria min
Create evaluation datasets min
Implement batch evaluations with Python min
Integrate evaluations into GitHub Actions min
Exercise - Set up automated evaluations min
Knowledge check min
Summary min

Start