Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Important
Items marked preview in this article are currently in preview. This preview is provided without a service-level agreement, and Microsoft doesn't recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.
A golden dataset is a small, curated set of evaluation examples that your team trusts as a benchmark for agent quality. Reuse it to compare agent versions, catch regressions, and measure improvements.
Don't make every dataset golden. Keep broad production samples, exploratory data, and regression cases separate from the benchmark that represents your quality bar.
| Aspect | Regular dataset | Golden dataset |
|---|---|---|
| Purpose | Evaluation, experimentation, or testing | Trusted baseline for comparing versions |
| Curation | Automated or lightly reviewed | Deliberately reviewed |
| Expected outputs | Optional | Expected for important examples |
| Size | Any | Small and high-signal |
| Maintenance | Depends on purpose | Continuously maintained |
Prerequisites
- An evaluation dataset created from annotated traces. See Create evaluation datasets from annotations.
Promote a dataset to golden
Your team decides when a dataset is ready to be golden. Consider the following factors:
- How much you trust the annotations and expected outputs.
- Whether the scenarios still reflect how your agent is used.
- Whether any sensitive content should be removed.
When you're ready, add a tag with the key golden to the dataset. The tag records your team's decision; it doesn't validate the examples.
To write a golden dataset from the SDK, add the golden tag in the output configuration of the data generation job. The tag value can be any string, such as a version label or true. This excerpt replaces output_configuration in the SDK example in Create evaluation datasets from annotations:
from azure.ai.projects.models import (
DataGenerationJobOutputWriteMode,
EvaluationDataGenerationJobOutputConfiguration,
)
output_configuration = EvaluationDataGenerationJobOutputConfiguration(
name="refund-agent-golden",
tags={"golden": "true"},
write_mode=DataGenerationJobOutputWriteMode.MERGE,
)
Organize datasets with tags
Use dataset tags to describe purpose and status, and to find datasets when you create evaluations. For example: golden, regression-test, production, deprecated, or customer-critical.
Version and maintain the dataset
- Record the version: Note which dataset version each evaluation uses. Compare agent versions against the same dataset version, so a change in the dataset isn't mistaken for a change in the agent.
- Add new failures deliberately: When production reveals a new failure or important behavior, review it, validate its expected output, and add it. Don't promote every production example.
- Re-evaluate after changes: When you update the benchmark, re-evaluate the agent versions you compare.
- Track coverage: Watch how often evaluations use golden datasets, and what share of examples have expected outputs.