FoundryEvals將 Agent Framework 的評估 API 與 Microsoft Foundry 的管理式評估服務連結起來。 它提供品質、安全、工具使用、代理行為及評分標準評估器，並可在 Foundry 入口網站中儲存報告。
關於 EvalItem、 局部檢查、自訂評估器及對話分割策略，請參見 代理評估。
先決條件
- 一個 Microsoft Foundry 專案與模型部署。
- 一個專案範圍的 Foundry 端點。
- 允許提交評估及閱讀報告。
評估回答或測驗問題
設定 FoundryEvals，然後評估已產生的回應，或讓 EvaluateAsync 代理執行每個查詢。
string endpoint = Environment.GetEnvironmentVariable("FOUNDRY_PROJECT_ENDPOINT") ?? throw new InvalidOperationException("FOUNDRY_PROJECT_ENDPOINT is not set.");
string deploymentName = Environment.GetEnvironmentVariable("FOUNDRY_MODEL") ?? "gpt-4o-mini";
// WARNING: DefaultAzureCredential is convenient for development but requires careful consideration in production.
// In production, consider using a specific credential (e.g., ManagedIdentityCredential) to avoid
// latency issues, unintended credential probing, and potential security risks from fallback mechanisms.
AIProjectClient projectClient = new(new Uri(endpoint), new DefaultAzureCredential());
AIAgent agent = projectClient.AsAIAgent(
model: deploymentName,
instructions: "You are a helpful assistant that provides clear, accurate answers.",
name: "QualityTestAgent");
// Configure Foundry evaluators.
FoundryEvals foundryEvals = new(projectClient, deploymentName, FoundryEvals.Relevance, FoundryEvals.Coherence);
// --- Pattern 1: Run agent, then evaluate pre-existing responses ---
string[] queries = ["What is photosynthesis?", "Explain gravity in simple terms."];
AgentResponse[] responses = new AgentResponse[queries.Length];
for (int i = 0; i < queries.Length; i++)
{
responses[i] = await agent.RunAsync(queries[i]);
}
AgentEvaluationResults results1 = await agent.EvaluateAsync(responses, queries, foundryEvals);
Console.WriteLine("=== Pattern 1: Evaluate pre-existing responses ===");
PrintResults(results1, queries);
// --- Pattern 2: Run + evaluate in one call ---
string[] queries2 = ["What causes rain?", "Why is the sky blue?"];
AgentEvaluationResults results2 = await agent.EvaluateAsync(queries2, foundryEvals);
Console.WriteLine("=== Pattern 2: Run + evaluate in one call ===");
PrintResults(results2, queries2);
.NET 樣本同時展示了 Foundry 評分標準評估器及每維度品質閘門。
評估一位代理人
將現有的回應或測試查詢傳遞給 evaluate_agent()。 結果包括通過/不通過計數及 Foundry 報告網址。
async def main() -> None:
# 1. Set up the FoundryChatClient
chat_client = FoundryChatClient(
project_endpoint=os.environ["FOUNDRY_PROJECT_ENDPOINT"],
model=os.environ.get("FOUNDRY_MODEL", "gpt-4o"),
credential=AzureCliCredential(),
)
# 2. Create an agent with tools
agent = Agent(
client=chat_client,
name="travel-assistant",
instructions=(
"You are a helpful travel assistant. Use your tools to answer questions about weather and flights."
),
tools=[get_weather, get_flight_price],
)
# 3. Create the evaluator — provider config goes here, once
evals = FoundryEvals(client=chat_client)
# =========================================================================
# Pattern 1: evaluate_agent(responses=...) — evaluate a response you already have
# =========================================================================
print("=" * 60)
print("Pattern 1: evaluate_agent(responses=...) — evaluate existing response")
print("=" * 60)
query = "How much does a flight from Seattle to Paris cost?"
response = await agent.run(query)
print(f"Agent said: {response.text[:100]}...")
# Pass agent= so tool definitions are extracted, queries= for the eval item context
results = await evaluate_agent(
agent=agent,
responses=response,
queries=[query],
evaluators=FoundryEvals(
client=chat_client,
evaluators=[FoundryEvals.RELEVANCE, FoundryEvals.TOOL_CALL_ACCURACY],
),
)
for r in results:
print(f"Status: {r.status}")
print(f"Results: {r.passed}/{r.total} passed")
print(f"Portal: {r.report_url}")
if r.all_passed:
print("[PASS] All passed")
else:
print(f"[FAIL] {r.failed} failed")
額外範例涵蓋追蹤評估、工具呼叫評估、多輪評估、工作流程評估、混合供應商及客製化 Foundry 評分標準。
Note
目前 Microsoft Foundry 的評估整合尚未支援 Agent Framework Go。 最新狀態請參閱 Agent Framework Go 倉庫 。
質量閘門
當結果必須在各次跑中可比較時，則要標註資料集、模型部署、評估器版本和評分標準版本。 當需要的指標回歸時，使用結果斷言輔助工具來使 CI 失敗。