Plan outcomes and success metrics

Completed

Before you make any platform or tool decisions, every agent design starts with the same foundational question: what should this agent accomplish, and how will you know when it's working? When you answer these questions upfront, you shape every architectural decision that follows.

Define the agent's goal and scope

Every agent design starts with a business outcome: the specific result the agent should measurably improve. Outcomes are concrete: reduced cycle time, higher accuracy, better consistency, or lower cost-to-serve. An outcome isn't a feature list or a technology description. It's a before-and-after statement about what changes when the agent works as intended.

Consider a customer service team where support representatives currently spend two business days answering routine inquiries. The primary outcome for an AI agent might be: reduce time-to-first-response to under two hours for routine requests, while maintaining policy-aligned accuracy on drafted responses. A secondary outcome might be a reduction in manual touch rate: the percentage of requests that require a human to write a full response from scratch. Both are concrete, measurable, and expressed as before-after statements.

Paired with the business outcome is the agent's scope: the set of tasks and user intents the agent handles. Scope answers the question "what does this agent do?" and, equally important, "what doesn't it do?"

For a customer service agent, in-scope tasks might include categorizing incoming requests, retrieving relevant policy and product information, drafting responses for common request types, and flagging requests that require human review. Out-of-scope tasks might include processing refunds, modifying orders, or handling legal escalations. For an HR onboarding agent, the scope boundary might fall between answering policy questions (in scope) and approving time-off requests or handling termination conversations (out of scope). The out-of-scope list is as important as the in-scope list: it defines the agent's boundaries and directly shapes the escalation rules built into it.

Getting scope right is a balancing act. Scope it too narrowly, and users may not find value in using the agent. Scope it too broadly, and you risk making the agent less effective at its primary tasks. A good rule of thumb: start with a narrow, well-defined scope, prove value, then expand deliberately.

Set boundaries and human oversight

Defining what the agent can do is only half the picture. Equally important is defining when the agent should stop and how control passes to a human.

Handoff conditions are the specific circumstances that trigger escalation: low confidence on a response, detected sensitive content (such as a legal complaint, safety concern, or mention of a named individual), or a request category the agent hasn't been designed to handle. Common examples:

• A customer service agent escalates when sentiment falls below a defined threshold or a request mentions a safety incident or regulatory question.
• A finance automation agent escalates when an invoice value exceeds a threshold, a vendor record doesn't match, or a confidence score falls below the team's quality bar.

The specific conditions differ by use case, but the design principle is the same: escalation behavior is explicit, not emergent.

Beyond handoff conditions, the team needs to decide how supervision works during the initial rollout. For high-stakes actions — like sending a response on behalf of the company — human review before send is a standard starting point. As the agent demonstrates reliability across a given request type, you can expand automation for that category while keeping tighter oversight on others. This graduated approach is safer than granting full autonomy from day one and adjusting only after something goes wrong.

Document these decisions explicitly: which request types require human approval before the agent acts, which are fully automated, and which are off-limits entirely. This boundary map becomes part of the solution design and a direct input to governance configuration later in the planning process.

Specify functional and safety requirements

Functional requirements are non-negotiable expectations for how the agent behaves. They're standards the agent must meet in every interaction, not aspirational goals. For an agent handling customer or employee communications, common functional requirements often include:

• Responses must be grounded in approved knowledge sources (product documentation, return policies, shipping guidelines). The agent doesn't fabricate answers.
• Every response that draws on a knowledge source includes a citation so a reviewer can verify it.
• Response latency stays within an acceptable window. A customer-facing agent that consistently takes 30 seconds to respond creates a worse experience than no agent at all.
• Tone remains professional and consistent with the company's communication standards.

Safety requirements address content the agent must never produce:

• Personally identifiable information it wasn't provided
• Medical or legal advice
• Statements that could be read as unauthorized commitments
• Content that disparages individuals

Write these requirements before evaluating any platform or tool. They aren't implementation details. They're acceptance criteria. Any architecture that can't satisfy them isn't a viable option, regardless of other advantages.

Establish success metrics with baselines

With the goal, scope, boundaries, and requirements defined, the team can express success numerically. Success metrics fall into two categories: quality metrics that measure whether the agent is doing its job well, and business value metrics that measure whether it's delivering the intended impact.

Quality metrics vary by scenario but typically include:

• Resolution or completion rate: the percentage of in-scope requests fully handled by the agent without escalation
• Accuracy or policy-alignment rate: the percentage of outputs assessed as correct by a human reviewer or automated evaluator
• Escalation correctness: the percentage of escalations confirmed as appropriate by reviewers

Business value metrics connect those quality outcomes to organizational impact. Microsoft's measurement guidance for AI agents organizes business value into four value drivers:

• Efficiency value: time saved, cycle time reduction, cost-per-interaction. The most accessible value driver and typically the first to show improvement. A customer service team might track time-to-first-response and manual touch rate against a current-state baseline.
• Quality value: error reduction, consistency improvement, rework avoidance. Often the largest driver in regulated industries, where output inconsistency carries compliance risk.
• Revenue value: conversion improvement, retention gains, or new revenue enabled. Connecting agent telemetry to CRM and sales data makes this case concrete.
• Strategic value: decision velocity, employee confidence, competitive positioning. Increasingly quantifiable through satisfaction signals and adoption trends.

For agents built with Copilot Studio, Microsoft provides a standard formula for quantifying efficiency value called Agent Assisted Hours. Regardless of platform, the principle holds: the license isn't the investment — adoption is. An agent that reaches only a fraction of its intended users returns only a fraction of its projected value, which is why routine adoption rate (the share of eligible users who engage with the agent regularly, not just those enrolled) is a leading indicator to track from the first day in production.

Each metric needs a before-state baseline — an honest snapshot of where things stand today. Without baselines, there's no way to demonstrate improvement. If the existing process has no formal measurement, as is common with manual workflows, establish one before or during the initial rollout, even if early numbers are rough estimates. Capturing three numbers before launch gives the measurement story a concrete foundation: time per task, error rate, and cost per interaction.

Treat success metrics as a design constraint

Success metrics don't just measure outcomes after the fact. Defined early, they shape the design itself.

When the team knows it needs 90% policy-aligned accuracy on drafted responses, that constraint guides decisions about knowledge source quality, grounding strategy, and the response review process. If the design can't meet the constraint, those decisions need to change — not the constraint. Building first and measuring later creates a quiet pressure to rationalize whatever was built, rather than evaluate it honestly.

This is the key principle: if the team can't write down what good looks like and describe how they'll know when it's achieved, the design isn't done.

Reflect: Pick an agent you're considering — or a manual process it might replace. What does success look like in concrete terms? What data exists today that you could use as a baseline?