CI/CD for Azure AI Foundry "AI Agent Service" Agents

Question

Hello,

We have a function app containing functions to orchestrate/interact with AI Agents in Azure AI Foundry Agent Service.

Within the repo of the function app, I have a folder containing yaml files with definitions of each agent used by the app (including name, instructions, model, tools, auth, etc - ie in same format as the vscode AI Foundry extension agent .yamls):

Within these yamls, I have placed wildcards for any items which depend on the environment (for example connection ids, urls within openapi specs, ai search service resource name, etc).

Example of "connection id" wildcard in the yaml for an agent which uses an openapi tool:

My question is: when commits in the function app are pushed to the repo, and CI/CD pipeline is initiated to move the function app through environments, is there a recommended way to trigger (re)deployment of AI Agents into the relevant AI Foundry project in each environment (for any agent definitions which have changed)?

My "brute force" way I'm envisaging at the moment:

In the pipeline, trigger a utility program which will:

• search for any commits in the current "push" of the function app code which contain new or amended agent definition yaml files
• for any new/changed agent definition yaml files:
  • load the yaml and substitute any wildcards with environment-specific values
  • create/update the relevant agent in Foundry based on the yaml definition, using sdk routines

In summary, is there an existing and/or better way? (ie am I re-inventing any wheels here?)

Answer 1

Hello Russell,

To implement a robust CI/CD pipeline for deploying AI Agents in Azure AI Foundry's Agent Service, you can enhance your current approach by integrating Azure's native tools and practices. Here's a structured method to achieve this:

Recommended CI/CD Strategy for Azure AI Foundry Agents

1. Environment-Specific Configuration Management
   • Use Parameterization: Instead of hardcoding environment-specific values (like connection IDs or URLs) in your YAML files, parameterize these values.
   • Leverage Variable Groups: In Azure DevOps, utilize variable groups to manage environment-specific configurations. This allows for cleaner YAML files and easier management of different environments.
2. Automated Detection and Deployment of Agent Definitions
   • Implement Change Detection: Set up your pipeline to detect changes in agent definition YAML files. This can be achieved by analyzing the commit history or using tools that monitor file changes.
   • Automate Deployment with Azure AI Foundry SDK: Use the Azure AI Foundry SDK to programmatically deploy or update agents based on the modified YAML definitions. This ensures consistency and reduces manual intervention.
3. Integration with Azure DevOps or GitHub Actions
   • Azure DevOps Pipelines: Create pipelines that trigger on commits to the repository. Incorporate tasks that handle the substitution of environment-specific parameters and invoke deployment scripts.
   • GitHub Actions: Similarly, set up GitHub Actions workflows that respond to repository changes, manage configurations, and deploy agents using the SDK.
4. Incorporate Evaluation Steps
   • Pre-Deployment Evaluation: Before deploying agents to production, run evaluations to assess their performance. Azure AI Foundry provides tools for offline evaluation within CI/CD pipelines, ensuring agents meet quality standards before release.
5. Monitoring and Observability
   • Enable Logging and Monitoring: Utilize Azure's monitoring tools to track the performance and health of deployed agents. This aids in proactive issue detection and resolution.

Sample Workflow Outline

1. Trigger: On commit to the repository.
2. Tasks:
   • Checkout the repository.
   • Identify modified agent YAML files.
   • Replace placeholders with environment-specific values using variable groups.
   • Deploy or update agents using Azure AI Foundry SDK.
   • Run evaluations to validate agent performance.
   • Monitor deployment status and log outcomes.

By adopting this structured approach, you can achieve a scalable and maintainable CI/CD process for your Azure AI Foundry agents, ensuring efficient deployments across different environments.

Best Regards,

Jerald Felix