Define your agent's purpose and key scenarios

Before starting evaluation design, make sure you have a clear and shared understanding of what your agent does and the business problems it solves. Without this foundation, your evaluation efforts might measure the wrong things.

• Core mission: Start by articulating the agent's core mission in concrete terms. What specific tasks should the agent accomplish? What user needs does it address? What does a successful interaction look like from both the user's and the business's perspective?
• Scope and boundaries: Document the agent's intended scope and boundaries. Be explicit about what the agent should handle, what it should gracefully decline, and what edge cases fall outside its purpose.
• Key scenarios: Identify the key scenarios the agent must handle to fulfill its purpose. These essential situations and use cases define what the agent needs to do. Be specific—vague scenarios lead to vague evaluations.

Employee Self-Service Agent

Core mission: Help employees get quick, accurate answers to HR questions without submitting tickets or waiting for human responses. Provide personalized guidance based on company policies and route complex or sensitive issues to appropriate human support.

In scope: Paid Time Off (PTO) policies and balances, benefits enrollment questions, training requirements, public holidays by location, general policy explanations, routing to HR for complex issues.

Out of scope: Performance reviews, salary negotiations, legal or compliance issues, medical emergencies, personnel decisions, compensation disputes.

Key scenarios:

• Answer HR policy questions: Respond accurately to questions about PTO, benefits, and leave policies using official company knowledge sources.
• Provide location or context-specific information: Tailor responses based on the employee's region, job level, or tenure.
• Route to human support appropriately: Recognize when a question requires human judgment and hand off gracefully.
• Protect sensitive information: Refuse requests for other employees' data or confidential information.

Next step

Learn how to design the right quality signals so your agent's evaluation reflects what truly matters.

Derive quality signals