Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Important
Agent Optimizer is currently in limited preview and only available through a sign-up process. To access the service, complete the intake form. This preview is provided without a service-level agreement, and we don't recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.
In this quickstart, you install the optimization CLI extension, deploy a hosted agent, run the agent optimizer, and deploy the winning candidate.
Prerequisites
| Tool | Required | Purpose |
|---|---|---|
| azd CLI | Yes | Azure Developer CLI. Provisions, deploys, and manages your agent. |
| Azure CLI | Yes | Azure authentication (az login) |
Your Azure subscription must be on the allow list for the agent optimizer. Contact your Microsoft representative to request access.
Install the CLI extension
Install the azure.ai.agents extension for the azd CLI:
azd ext install azure.ai.agents
Verify the installation:
azd ai agent optimize --help
Create the project
Initialize a new project from the agent optimizer sample. Create a folder, then use azd ai agent init with the sample manifest:
mkdir my-agent && cd my-agent
azd ai agent init -m https://github.com/microsoft-foundry/foundry-samples/blob/main/samples/python/hosted-agents/bring-your-own/responses/optimization-customer-support/agent.manifest.yaml .
The interactive flow prompts for your Azure subscription, region, and model deployment settings. It downloads the sample and generates agent.yaml, .agent_configs/baseline/, the evaluation dataset, and infrastructure-as-code files for provisioning.
Tip
If you already have an existing agent project, skip this step and see Make your agent optimizer-ready to add optimization support.
If you already have a Foundry project and model deployments, add -p <project-resource-id> to target existing resources:
azd ai agent init -m https://github.com/microsoft-foundry/foundry-samples/blob/main/samples/python/hosted-agents/bring-your-own/responses/optimization-customer-support/agent.manifest.yaml -p "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.CognitiveServices/accounts/<account>/projects/<project>"
Authenticate
az login
azd auth login
If you have multiple subscriptions, select the one you want:
az account set --subscription "<subscription-name-or-id>"
Configure and provision
Choose one of the following options based on whether you need to create new Azure resources or already have an existing Foundry project.
Option A: Create new resources
Provision the Azure resources. This step takes approximately two minutes:
azd provision
This step creates:
- A Foundry account and project
- An Azure Container Registry
- Model deployments (gpt-4.1-mini for eval, gpt-5.4 for optimization)
Option B: Use an existing Foundry project
If you already have a Foundry project with models deployed, use agent init to configure your environment:
azd ai agent init --project-id "/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.CognitiveServices/accounts/<account>/projects/<project>"
Tip
Find your project resource ID in the Azure portal → your Foundry project → Properties → Resource ID.
Set your model deployment name:
azd env set AZURE_AI_MODEL_DEPLOYMENT_NAME "gpt-4.1-mini"
Deploy the agent
azd deploy
This command builds the container image, pushes it to Azure Container Registry, and registers the hosted agent. The process takes approximately 1.5 minutes. The output includes a portal playground link you can use to chat with the agent.
Test the deployment:
azd ai agent invoke "What is 2+2?"
Generate custom dataset and evaluations
For meaningful optimization with your own scenarios, generate a dataset with eval init:
azd ai agent eval init --gen-instruction "You are a helpful customer support agent."
The command creates an eval.yaml with a dataset and evaluators tuned to your agent's domain.
For details, see Create an evaluation dataset.
Run optimization
Then run optimization with the generated configuration:
azd ai agent optimize
The CLI reads the name field from your agent.yaml file to determine which deployed agent to optimize. It'll also automatically detect and use the generated eval.yaml file to run optimization. If you have multiple agents or configurations or want to target a different one, use the --agent and --config flags:
azd ai agent optimize --agent <your-agent-name> --config <your-config-file>.yaml
For more details on agent targeting, see Which agent gets optimized.
The agent optimizer completes the following steps:
- Evaluates your baseline agent against a built-in dataset that contains 3 tasks and 12 criteria.
- Generates improved candidates (instructions, skills, tools, or model configurations depending on your target).
- Evaluates each candidate.
- Ranks the candidates by score.
This process takes a few minutes. You see real-time progress:
Optimizing agent "customer-support-py"...
Config: C:\Dev\my-agent\eval.yaml
Baseline saved to .agent_configs\baseline\metadata.yaml
Job ID: opt_162bd0f09070432c9ca4a699a908abb0
Status: pending
Portal: <OPTIMIZATION-JOB-URL>
Use the URL provided in the CLI to view and monitor your job in the Foundry portal.
The eval model scores each response (any chat-completion model works). The optimization model generates improved candidates and must be from the supported list (gpt-5 family or DeepSeek).
Warning
If the eval model is not deployed, all scores are zero with no error message. Verify that your eval model exists before running optimization.
Deploy the winner
The ★ indicates the best candidate. The recommended workflow is to apply the optimized config locally, then deploy:
# Apply the winning candidate locally
azd ai agent optimize apply --candidate <candidate-id>
# Deploy with the optimized config
azd deploy
This downloads the optimized configuration into .agent_configs/<candidate_id>/ in your project. On next deploy, your agent uses the improved instructions and tool descriptions.
Alternatively, for quick testing you can deploy directly:
azd ai agent optimize deploy --candidate <candidate-id>
The azure-ai-agentserver-optimization package's load_config() function picks up the new configuration automatically at startup.
Invoke your agent again to verify the improvement:
azd ai agent invoke "What is your return policy?"
You can also run evaluation separately to confirm the score improvement:
azd ai agent eval run
Monitor and manage
Use the job ID, which is formatted as opt_<hex> and is printed in optimization output, to track and manage runs:
# Watch a running job
azd ai agent optimize status <job-id> --watch
# List all optimization runs
azd ai agent optimize list
# Cancel a running job
azd ai agent optimize cancel <job-id>
Clean up resources
When you finish experimenting, delete the provisioned resources:
azd down --force --purge
Tip
Why --purge? Foundry accounts use soft-delete by default. Without --purge, the resource name stays reserved for 48 hours, and reprovisioning with the same name fails.
Troubleshooting
| Problem | Cause | Fix |
|---|---|---|
| All scores are zero | Agent or eval model not deployed | Deploy the agent or eval model in your Foundry project, or use --eval-model to specify a deployed model |
azd provision fails with quota error |
Subscription lacks capacity | Try a different subscription or request a quota increase |