evaluasi Microsoft Foundry

FoundryEvalsmenghubungkan API evaluasi Kerangka Kerja Agen ke layanan evaluasi terkelola Microsoft Foundry. Ini memberikan kualitas, keamanan, penggunaan alat, perilaku agen, dan evaluator rubrik, dengan laporan tersimpan yang tersedia di portal Foundry.

Untuk EvalItem, pemeriksaan lokal, evaluator kustom, dan strategi pemisahan percakapan, lihat Evaluasi agen.

Prasyarat

• Proyek Microsoft Foundry dan penyebaran model.
• Titik akhir Foundry yang dilingkupi proyek.
• Izin untuk mengirimkan evaluasi dan membaca laporan.

Mengevaluasi respons atau kueri pengujian

Konfigurasikan FoundryEvals, lalu evaluasi respons yang sudah dibuat atau biarkan EvaluateAsync jalankan agen untuk setiap kueri.

string endpoint = Environment.GetEnvironmentVariable("FOUNDRY_PROJECT_ENDPOINT") ?? throw new InvalidOperationException("FOUNDRY_PROJECT_ENDPOINT is not set.");
string deploymentName = Environment.GetEnvironmentVariable("FOUNDRY_MODEL") ?? "gpt-4o-mini";

// WARNING: DefaultAzureCredential is convenient for development but requires careful consideration in production.
// In production, consider using a specific credential (e.g., ManagedIdentityCredential) to avoid
// latency issues, unintended credential probing, and potential security risks from fallback mechanisms.
AIProjectClient projectClient = new(new Uri(endpoint), new DefaultAzureCredential());

AIAgent agent = projectClient.AsAIAgent(
    model: deploymentName,
    instructions: "You are a helpful assistant that provides clear, accurate answers.",
    name: "QualityTestAgent");

// Configure Foundry evaluators.
FoundryEvals foundryEvals = new(projectClient, deploymentName, FoundryEvals.Relevance, FoundryEvals.Coherence);

// --- Pattern 1: Run agent, then evaluate pre-existing responses ---
string[] queries = ["What is photosynthesis?", "Explain gravity in simple terms."];

AgentResponse[] responses = new AgentResponse[queries.Length];
for (int i = 0; i < queries.Length; i++)
{
    responses[i] = await agent.RunAsync(queries[i]);
}

AgentEvaluationResults results1 = await agent.EvaluateAsync(responses, queries, foundryEvals);

Console.WriteLine("=== Pattern 1: Evaluate pre-existing responses ===");
PrintResults(results1, queries);

// --- Pattern 2: Run + evaluate in one call ---
string[] queries2 = ["What causes rain?", "Why is the sky blue?"];
AgentEvaluationResults results2 = await agent.EvaluateAsync(queries2, foundryEvals);

Console.WriteLine("=== Pattern 2: Run + evaluate in one call ===");
PrintResults(results2, queries2);

Sampel .NET juga menunjukkan evaluator rubrik Foundry dan gerbang kualitas per dimensi.

Mengevaluasi agen

Berikan respons atau kueri pengujian yang ada ke evaluate_agent(). Hasilnya termasuk jumlah pass/fail dan URL laporan Foundry.

async def main() -> None:
    # 1. Set up the FoundryChatClient
    chat_client = FoundryChatClient(
        project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
        model=os.environ.get("FOUNDRY_MODEL", "gpt-4o"),
        credential=AzureCliCredential(),
    )

    # 2. Create an agent with tools
    agent = Agent(
        client=chat_client,
        name="travel-assistant",
        instructions=(
            "You are a helpful travel assistant. Use your tools to answer questions about weather and flights."
        ),
        tools=[get_weather, get_flight_price],
    )

    # 3. Create the evaluator — provider config goes here, once
    evals = FoundryEvals(client=chat_client)

    # =========================================================================
    # Pattern 1: evaluate_agent(responses=...) — evaluate a response you already have
    # =========================================================================
    print("=" * 60)
    print("Pattern 1: evaluate_agent(responses=...) — evaluate existing response")
    print("=" * 60)

    query = "How much does a flight from Seattle to Paris cost?"
    response = await agent.run(query)
    print(f"Agent said: {response.text[:100]}...")

    # Pass agent= so tool definitions are extracted, queries= for the eval item context
    results = await evaluate_agent(
        agent=agent,
        responses=response,
        queries=[query],
        evaluators=FoundryEvals(
            client=chat_client,
            evaluators=[FoundryEvals.RELEVANCE, FoundryEvals.TOOL_CALL_ACCURACY],
        ),
    )

    for r in results:
        print(f"Status: {r.status}")
        print(f"Results: {r.passed}/{r.total} passed")
        print(f"Portal: {r.report_url}")
        if r.all_passed:
            print("[PASS] All passed")
        else:
            print(f"[FAIL] {r.failed} failed")

Sampel tambahan mencakup evaluasi jejak, evaluasi panggilan alat, evaluasi multi-giliran, evaluasi alur kerja, penyedia campuran, dan rubrik Foundry kustom.

Nota

Microsoft Integrasi evaluasi Foundry saat ini tidak tersedia untuk Agent Framework Go. Lihat repositori Agent Framework Go untuk status terbaru.

Gerbang berkualitas

Sematkan himpunan data, penyebaran model, versi evaluator, dan versi rubrik saat hasil harus sebanding di seluruh eksekusi. Gunakan pembantu pernyataan hasil untuk menggagalkan CI saat regresi metrik yang diperlukan.

Langkah berikutnya

Evaluasi agen