Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Use evaluations to check that your Hosted Skill returns expected responses, calls the right tools,
and maintains context across a conversation. The @azure/functions-hosted-skills-evaluator package
provides a custom executor built on top of Vally, so you can use Vally's evaluation format and
graders to test your Hosted Skill locally or against a staging Function App. The evaluator executes
your Hosted Skill and captures responses and tool calls; Vally's graders assess that behavior.
The executor connects to your app's existing synchronous chat endpoint and captures responses and tool calls for grading. These evaluations focus on your Hosted Skill's responses and tool use, rather than Functions availability, scaling, or trigger delivery. For guidance on designing evaluations and choosing graders, see the Vally documentation. Review the Hosted Skill evaluation sample for a runnable receipt Hosted Skill and deterministic eval specification.
Prerequisites
- Node.js 22.12 or later, including npm
- a Hosted Skill with
builtin_endpoints.chat_api: true. To create one, follow the Hosted Skills quickstart. - a Function App running under Core Tools or deployed to staging
Install the evaluator
In a terminal, navigate to your Function App's root directory and install the published
@azure/functions-hosted-skills-evaluator
package and Vally CLI from npm.
Install the evaluation tools as development dependencies:
npm install --save-dev @azure/functions-hosted-skills-evaluator @microsoft/vally-cli
Write an eval specification
Create eval.yaml in your Function App's root directory using Vally's native format. Include the
following executor configuration under defaults to connect Vally to your Hosted Skill:
defaults:
executor:
name: azure-functions-hosted-skill
config:
endpointUrlEnv: AGENT_EVAL_TARGET_URL
auth:
type: anonymous
defaults.executor.name: Set toazure-functions-hosted-skill, the executor registered by the evaluator npm package.defaults.executor.config.endpointUrlEnv: The name of an environment variable that contains your Hosted Skill's complete synchronous chat endpoint URL, not the URL itself. In this example, the executor reads the URL fromAGENT_EVAL_TARGET_URLin the environment where the Vally CLI runs. Set that variable before running evaluations. You can use the same specification for local and staging targets by changing the variable's value.defaults.executor.config.auth: How the executor authenticates to the endpoint. Useanonymousfor an endpoint that doesn't require authentication. For protected endpoints, see Authenticate a staging target.
Add your evaluation's stimuli (test inputs) and graders (checks) as described in Vally's Writing Eval Specs and Choose a Grader guides. For a complete specification with test inputs, graders, and expected outputs, see Vally's expanded example and the Hosted Skill sample evaluation specification.
Handle external dependencies
Evaluations use the tools, connectors, and MCP servers configured in the target Function App. If your Hosted Skill calls an external service, that call reaches the configured service during evaluation; the evaluator doesn't automatically mock external dependencies.
For repeatable tests, configure your test app to use deterministic tool implementations or mock MCP
servers that return fixed data, as the Hosted Skill evaluation sample
does with its receipt tool. Configure these dependencies in the Function App, not in Vally's
agent_environment settings, which don't reconfigure the target app. To test real integrations,
use sandbox services or dedicated test accounts to avoid changing production data.
Run locally
Start the Function App, set its complete chat URL, and invoke the locally installed CLI from a second terminal in your Function App's root directory:
$env:AGENT_EVAL_TARGET_URL = "http://localhost:7071/agents/receipt/chat"
npx vally eval `
--eval-spec eval.yaml `
--executor-plugin @azure/functions-hosted-skills-evaluator `
--require-pass `
--junit `
--output-dir artifacts/agent-evaluation
The --executor-plugin option loads the installed npm package by name.
--require-pass is essential for CI: without it, a completed evaluation with failing grader verdicts exits successfully. Configuration, execution, and tooling failures still exit nonzero. --junit writes CI-compatible test output alongside Vally's result artifacts.
For example pass/fail results and grader evidence, see Vally's Understanding the output. For help investigating unexpected results, see Debugging Evals.
Authenticate a staging target
Don't put keys or bearer tokens in an eval specification. The executor supports three fail-closed authentication configurations.
Anonymous endpoints need no extra fields:
auth:
type: anonymous
For a Functions or host key, reference an environment variable:
auth:
type: function-key
keyEnv: AGENT_EVAL_FUNCTION_KEY
$env:AGENT_EVAL_FUNCTION_KEY = "<CI secret>"
For Microsoft Entra authentication, provide the accepted API scope directly or through scopeEnv:
auth:
type: entra
scopeEnv: AGENT_EVAL_ENTRA_SCOPE
$env:AGENT_EVAL_ENTRA_SCOPE = "api://<application-id>/.default"
Entra authentication uses DefaultAzureCredential. Use managed identity in Azure-hosted CI where
possible. Literal function keys, arbitrary headers, bearer tokens, and endpoint query strings are
rejected by configuration validation.
Runtime evidence and failure semantics
The executor captures conversation turns, session and model IDs, tool calls and results, parallel-tool batches, and elapsed time. It doesn't report token usage, cost, reasoning, skill activation, subagent, or workspace evidence.
Transport errors, authentication failures, timeouts, malformed responses, and session ID mismatches
are execution failures, not low quality scores. A tool result marked success can still contain an
unsuccessful plain-text response, so use graders to check important result content.
Don't run concurrent turns with the same explicit session ID; the runtime doesn't coordinate their ordering across workers.