Curate golden datasets (preview)

Important

Items marked preview in this article are currently in preview. This preview is provided without a service-level agreement, and Microsoft doesn't recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.

A golden dataset is a small, curated set of evaluation examples that your team trusts as a benchmark for agent quality. Reuse it to compare agent versions, catch regressions, and measure improvements.

Don't make every dataset golden. Keep broad production samples, exploratory data, and regression cases separate from the benchmark that represents your quality bar.

Aspect Regular dataset Golden dataset
Purpose Evaluation, experimentation, or testing Trusted baseline for comparing versions
Curation Automated or lightly reviewed Deliberately reviewed
Expected outputs Optional Expected for important examples
Size Any Small and high-signal
Maintenance Depends on purpose Continuously maintained

Prerequisites

Promote a dataset to golden

Your team decides when a dataset is ready to be golden. Consider the following factors:

  • How much you trust the annotations and expected outputs.
  • Whether the scenarios still reflect how your agent is used.
  • Whether any sensitive content should be removed.

When you're ready, add a tag with the key golden to the dataset. The tag records your team's decision; it doesn't validate the examples.

To write a golden dataset from the SDK, add the golden tag in the output configuration of the data generation job. The tag value can be any string, such as a version label or true. This excerpt replaces output_configuration in the SDK example in Create evaluation datasets from annotations:

from azure.ai.projects.models import (
    DataGenerationJobOutputWriteMode,
    EvaluationDataGenerationJobOutputConfiguration,
)

output_configuration = EvaluationDataGenerationJobOutputConfiguration(
    name="refund-agent-golden",
    tags={"golden": "true"},
    write_mode=DataGenerationJobOutputWriteMode.MERGE,
)

Organize datasets with tags

Use dataset tags to describe purpose and status, and to find datasets when you create evaluations. For example: golden, regression-test, production, deprecated, or customer-critical.

Version and maintain the dataset

  • Record the version: Note which dataset version each evaluation uses. Compare agent versions against the same dataset version, so a change in the dataset isn't mistaken for a change in the agent.
  • Add new failures deliberately: When production reveals a new failure or important behavior, review it, validate its expected output, and add it. Don't promote every production example.
  • Re-evaluate after changes: When you update the benchmark, re-evaluate the agent versions you compare.
  • Track coverage: Watch how often evaluations use golden datasets, and what share of examples have expected outputs.