Fix insights with agent optimizer (preview)

Important

Items marked preview in this article are currently in preview. This preview is provided without a service-level agreement, and Microsoft doesn't recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.

Use Agent Optimizer in Microsoft Foundry to address recurring issues identified by insights from production traces. Select insights to fix, create an optimization run, and evaluate candidate fixes before promoting a new agent version.

This portal workflow currently supports prompt agents that don't use the Microsoft Copilot harness. Hosted agents use Foundry Toolkit or Azure Developer CLI workflows instead.

Prerequisites

Create an optimization run

Select insights and send them to agent optimizer to test possible fixes. The current agent version is the baseline. An optimization run doesn't change your active agent unless you review and promote a candidate.

Warning

If your agent's tools can modify production systems, optimization runs can cause unintended or irreversible changes. For example, an order-deletion tool can delete an order from the connected production database during evaluation. This risk applies to both regular optimization and optimization from insights, even if you don't promote a candidate.

Before starting a run, ensure tools use read-only access or isolated test resources. Isolate the resources and permissions the tools use, not just their execution environment.

Tool calls during evaluation can also incur charges or trigger rate limiting.

  1. In the Foundry portal, open your project, select Build > Agents, and then select your agent.

  2. On the Insights tab, review the insight descriptions, linked traces, and proposed fixes. Then select Select insights.

    Screenshot of the Insights tab with Select insights above the list and Fix insight in the details pane.

  3. Select the checkbox for each insight that you want agent optimizer to address, and then select Fix insights.

    Screenshot of two selected insights with the Fix insights action above the Insights list.

  4. In Create optimization run, confirm the following settings:

    Setting Description
    Agent The agent used as the baseline.
    Version The agent version that candidates are compared with.
    Optimization model The model that proposes candidate changes.
    Max candidates The maximum number of candidates to create and evaluate.
    Evaluation model The model that evaluates the baseline and candidates.
  5. Select Create to start the optimization run.

    Screenshot of the Create optimization run dialog with baseline, candidate, optimization model, maximum number of candidates, and evaluation model settings.

Understand the results

Use Insights improved as the primary outcome for insights optimization. The other views provide information about general quality and the optimization process.

Screenshot of a succeeded optimization run showing the recommended candidate, Insights improved chart, and candidate results.

Result or action How to interpret it
Insights improved Shows how many selected insights the candidate improves. Start here to assess whether the run addresses the issues you selected.
View evaluation Shows results from the general-quality rubric also used for regular optimization. For insights optimization, this evaluation acts as a regression check, not a measure of how many insights improve.
View overall score details Shows results for the baseline and candidates across different stages of the optimization pipeline. Use this view to investigate candidate performance and the optimization process in more detail.

The overall score details can show multiple candidates even when the main results view shows only one candidate.

A candidate can pass the general-quality check with up to 2% regression if it addresses at least one selected insight. Review both insight improvements and general-quality results before promoting a candidate.

Review and promote a candidate

Candidates can change agent instructions and descriptions for function-calling tools, but not the model or skills.

  1. When the run finishes, review the candidate changes and evaluation results in agent optimizer.
  2. Promote a candidate only after you confirm that it improves the selected behaviors without unacceptable regressions.

For more information about candidates, evaluation, and promotion, see agent optimizer overview.