Edit

winml eval

Evaluate ONNX or native Hugging Face PyTorch models.

When to use this

Use winml eval to measure how accurately a model performs on real data — especially after quantization, where comparing the quantized model against the floating-point baseline reveals any accuracy regression introduced by precision reduction.

Synopsis

winml eval [options]

Flags

Flag Short Type Default Description
--model -m TEXT — HuggingFace model ID, or path to a local .onnx file. Required (unless --model-id is provided directly).
--model-id TEXT — HuggingFace model ID used for preprocessor and config resolution when -m points to an ONNX file.
--task TEXT auto-detected Task name (e.g., image-classification). Auto-detected from --model-id when not provided. Required when -m is an ONNX file and the task cannot be inferred.
--precision TEXT auto Precision used when building the model from a HuggingFace ID. One of auto, fp32, fp16, int8, int16, or a mixed w{x}a{y} spec (e.g., w8a16). fp16/fp32 skip quantization. Ignored when -m is a pre-built .onnx file — the precision is already baked in.
--device choice auto Target device. Choices: auto, npu, gpu, cpu. auto selects the best available device. Combined with --precision, this drives the build when -m is a HuggingFace ID.
--device-luid TEXT — Select a physical adapter within the resolved EP/device pair using its LUID from winml sys (0xHHHHHHHH_0xLLLLLLLL, case-insensitive). Supported by the winml-ort runtime.
--ep / --execution-provider TEXT — Target ONNX Runtime execution provider when finer control than --device is needed. Full names (e.g., QNNExecutionProvider, OpenVINOExecutionProvider, VitisAIExecutionProvider) and aliases (qnn, ov/openvino, vitis/vitisai) are accepted.
--shape-config PATH — JSON shape overrides used while auto-generating a Hugging Face export config, for example {"height": 480, "width": 480}. Applies only when -m is a HuggingFace ID that eval builds; ignored for pre-built .onnx inputs.
--input-specs PATH — JSON input tensor specs to merge into the Hugging Face export config. Symbolic string dimensions infer dynamic axes. Ignored for pre-built .onnx inputs.
--export-config PATH — JSON ONNX export config overrides (opset version, constant folding, etc.) to merge into the Hugging Face export config. Ignored for pre-built .onnx inputs.
--dynamic-axes PATH — JSON dynamic axes mapping for Hugging Face ONNX export, for example {"input_ids": {"0": "batch", "1": "sequence"}}. Ignored for pre-built .onnx inputs.
--runtime winml-ort\|pytorch winml-ort Evaluation runtime. winml-ort exports Hugging Face checkpoints to ONNX; pytorch evaluates the original checkpoint and supports auto, cpu, or CUDA-backed gpu devices.
--dataset TEXT task default HuggingFace dataset path (e.g., imagenet-1k, nyu-mll/glue). If omitted, a default dataset is selected based on the task.
--dataset-name TEXT — Dataset configuration name for multi-config datasets.
--dataset-revision TEXT — Git revision (branch, tag, or commit) of the dataset to load. Use refs/convert/parquet for HF datasets that are only served via the parquet mirror.
--dataset-script TEXT — Path to a Python script that builds the evaluation dataset locally. Requires --trust-remote-code.
--trust-remote-code / --no-trust-remote-code flag false Allow executing custom code from model repositories or dataset scripts. Required with --dataset-script. Use only with trusted sources.
--samples INTEGER 100 Number of dataset samples to evaluate.
--split TEXT validation Dataset split to use (e.g., validation, test, train).
--shuffle / --no-shuffle flag shuffle Shuffle the dataset before sampling. Disable with --no-shuffle for reproducible sample ordering.
--streaming / --no-streaming flag false Stream the dataset from the Hub instead of downloading the full split. Useful for large datasets.
--column TEXT (multiple) — Column mapping as key=value pairs (e.g., --column input_column=image). Can be specified multiple times.
--label-mapping PATH — Path to a JSON file mapping dataset label names to the integer class IDs the model emits: {"label_name": id}.
--output -o PATH — Output JSON file path for the evaluation results.
--overwrite flag false Replace an existing output report.
--schema flag false Print the expected dataset schema for the given --task and exit. Does not run evaluation.
--mode onnx\|compare onnx Evaluation mode. onnx evaluates the candidate model on a dataset. compare runs the candidate and a reference on identical random inputs and reports per-tensor similarity metrics — no dataset required. The reference is the HuggingFace model from --model-id by default, or an ONNX file when --reference is given.
--input-data PATH — Path to a .npz file of real input tensors to compare with instead of randomly generated ones (used with --mode compare). Keys must match the candidate model's input names. The leading axis of each array is the sample axis, so an archive shaped (N, ...) yields N samples (mean/std/min/max are computed across them); all inputs must share the same N. Each run is shaped to the candidate's batch size — a dynamic batch runs one row per sample, a static batch B chunks the axis into N // B batches (trailing rows are dropped with a warning). Note this differs from winml perf --input-data, which runs the whole archive as a single batch.
--reference TEXT — Reference .onnx file to compare against (used with --mode compare). --model-id and --task are not required in this mode.
--reference-device cpu\|gpu\|npu\|auto cpu Device used for the reference ONNX model. Only valid with --reference.
--reference-device-luid TEXT — Select a physical adapter for the reference ONNX model using its LUID from winml sys. Only valid with --reference.
--reference-ep TEXT — Explicit execution provider used for the reference ONNX model, for example dml. Only valid with --reference.

How it works

Choose the evaluation mode to match the question. Dataset evaluation measures task performance against labels; compare mode measures output similarity against a reference using identical tensors. Neither automatically establishes that a model meets your application's acceptance criteria.

For a prebuilt ONNX input, build-time precision and export options don't transform the graph. Create each candidate first, then evaluate it with the same preprocessing, sample selection, and reference.

winml eval loads the model and runs the evaluation pipeline via the internal evaluate function, then pulls the requested number of samples from a HuggingFace dataset. By default, Hugging Face model IDs and local checkpoints use the winml-ort runtime: they are exported to ONNX and evaluated through WinML. With --runtime pytorch, the task-resolved PyTorch model and stored dtype are preserved and the same dataset preprocessing, evaluator, and metrics run directly against that model. PyTorch auto selects CUDA when available and otherwise CPU; gpu requires CUDA. The JSON report identifies the effective runtime as winml-ort or pytorch for these workflows.

Python callers can pass an existing model directly with evaluate(config, pytorch_model=model). An explicit config.model_id selects the tokenizer or processor; otherwise evaluation infers it from model.config._name_or_path and reports an error if neither source is available.

PyTorch text-generation evaluation adapts the checkpoint to the existing causal-LM evaluator contract and reports perplexity without ONNX export. Pre-built ONNX files, composite role=path models, GenAI bundles, compare mode, references, and tensor input archives remain on their existing WinML paths. --runtime pytorch rejects those forms, along with ONNX build, export, EP, precision, quantization, optimization, analysis, and cache-related options.

Examples

Evaluate a HuggingFace model using the task-default dataset:

winml eval -m microsoft/resnet-50

Evaluate the original Hugging Face PyTorch checkpoint on CPU without ONNX export:

winml eval -m microsoft/resnet-50 --runtime pytorch --device cpu

Task:     image-classification
Dataset:  timm/mini-imagenet (test, 100 samples)
Device:   auto

Accuracy: 76.00%

Results saved to: microsoft_resnet-50_eval.json

Evaluate a pre-exported ONNX file, providing the source model ID for preprocessing:

winml eval -m model.onnx --model-id microsoft/resnet-50 --dataset timm/mini-imagenet

Evaluate a BERT model on the MRPC paraphrase task with column remapping:

winml eval -m Intel/bert-base-uncased-mrpc --dataset nyu-mll/glue --dataset-name mrpc --column input_column=sentence1 --column second_input_column=sentence2 --samples 500

Compare an ONNX candidate against its HuggingFace reference on real input tensors instead of random ones by passing a .npz archive whose keys match the candidate's input names. The leading axis of each array is the sample axis, so an archive shaped (N, ...) runs N samples:

winml eval --mode compare -m model.onnx --model-id microsoft/resnet-50 --input-data inputs.npz

Compare two ONNX files directly (e.g. an fp32 baseline vs a quantized build), reporting per-output tensor-similarity metrics on identical random inputs — no --model-id or dataset needed:

winml eval --mode compare -m quantized.onnx --reference baseline.onnx

Compare a candidate running on DML with an ONNX reference running on CPU:

winml eval --mode compare -m candidate.onnx --device gpu --ep dml --reference baseline.onnx

Select a specific DML GPU for the candidate using its LUID from winml sys:

winml eval --mode compare -m candidate.onnx --device gpu --ep dml `
    --device-luid 0x00000000_0x00012C8B --reference baseline.onnx

The LUID pins only the candidate model. The reference continues to use --reference-device, --reference-device-luid, and --reference-ep. If a selected adapter cannot initialize, eval fails instead of silently retrying that model on CPU.

Pin the candidate and reference to different physical GPUs:

winml eval --mode compare -m candidate.onnx --device gpu --ep dml `
    --device-luid 0x00000000_0x00012C8B --reference baseline.onnx `
    --reference-device gpu --reference-ep dml `
    --reference-device-luid 0x00000000_0x00015A31

Check what dataset columns are expected before running, then remap them to match your dataset:

winml eval --schema --task text-classification

Input schema for text-classification models
==================================================

--column option schema

Evaluating needs a dataset with the following columns:
  input_column
      input text (default: text)
  label_column
      class label (ClassLabel or integer) (default: label)
  second_input_column
      second text for sentence-pair tasks (optional) (default: None)

Override any default with --column:
  --column input_column=<your_text_column>
  --column label_column=<your_label_column>
  --column second_input_column=<your_pair_column>

The GLUE SST-2 dataset uses sentence instead of the default text column, so remap it with a single --column override:

winml eval -m distilbert/distilbert-base-uncased-finetuned-sst-2-english --dataset nyu-mll/glue --dataset-name sst2 --column input_column=sentence --samples 500

Evaluate against a custom dataset whose label names differ from the model's class IDs. The --label-mapping flag points to a JSON file whose keys are the label name strings as they appear in the dataset and whose values are the integer class IDs the model emits. For example, ResNet-50 outputs ImageNet-1k class IDs (0–999), so if your custom dataset uses readable strings like "tabby cat" or "golden retriever", labels.json translates each dataset label to the corresponding ImageNet ID the model predicts:

{
  "tabby cat": 281,
  "Egyptian cat": 285,
  "golden retriever": 207
}

winml eval -m microsoft/resnet-50 --dataset my-org/my-pets-dataset --label-mapping labels.json -o results/resnet_eval.json

Evaluate a composite model from pre-exported ONNX files. Some tasks (e.g., image-to-text, encoder-decoder, dual-encoder) split the model across multiple ONNX files, one per role. Pass -m once per role as <role>=<path>.onnx and supply --model-id so the preprocessor and tokenizer can be resolved. Run winml eval --schema --task image-to-text to see the expected roles for a task:

winml eval -m encoder=encoder.onnx -m decoder=decoder.onnx --model-id microsoft/trocr-base-printed

Model build cache

Evaluation reuses persistent model build artifacts by default. Pass --no-use-cache for a fresh build in a temporary directory, or --rebuild for a fresh build that replaces the persistent cache entry.

For a pre-built ONNX input, cache controls apply only when --no-skip-build is set. The CLI warns when an explicit cache control has no effect because no model build runs. GenAI's runtime _compiled/ artifacts are a separate cache and are not currently governed by these model build cache controls. Explicit build or cache controls on a GenAI bundle produce a warning that distinguishes the model-build pipeline from the runtime compilation cache.

Common pitfalls

  • Dataset evaluation of an ONNX file requires --model-id. When evaluating a dataset with a .onnx path in -m, supply --model-id to resolve the preprocessor and model configuration. Without it, dataset evaluation exits with a usage error. Comparing two ONNX files with --mode compare --reference .\baseline.onnx doesn't require --model-id or --task.
  • The task-default dataset may not match every model. A default dataset cannot fit every model. Classification and detection models in particular need a dataset whose label space and domain match what the model was trained on — using the default may produce misleadingly low scores, missing-label errors, or a dataset-schema error. Always pass --dataset (and --label-mapping if needed) when evaluating a model whose label space or domain differs from the task default.
  • Some dataset requires Hub credentials for gated datasets. Some datasets (e.g., imagenet-1k) require a HuggingFace account with accepted terms of use. Log in with huggingface-cli login before running eval on gated data.
  • --shuffle is on by default. The random 100-sample slice changes between runs unless you pass --no-shuffle. Use --no-shuffle when comparing two model variants to ensure they see identical samples.
  • --streaming skips the local cache. Streaming mode avoids downloading the full split but prevents random shuffling on large datasets. For reproducible evaluation, download the split once and omit --streaming.
  • Export overrides only apply when eval builds from a HuggingFace ID. --shape-config, --input-specs, --export-config, and --dynamic-axes shape the ONNX export that eval generates when -m is a HuggingFace model ID. When -m is a pre-built .onnx file, there is no export step, so these flags are ignored and the command prints a warning.
  • The PyTorch runtime accepts only Hugging Face checkpoints. --runtime pytorch cannot be combined with ONNX files, composite models, GenAI bundles, compare/reference/input-data modes, EP or adapter LUID selection, or ONNX build/export controls. Use --device cpu, --device gpu with CUDA, or --device auto.
  • Column names vary across datasets. If the evaluator raises a missing-column error, run winml eval --schema --task <task> to inspect the expected schema and use --column to remap dataset field names to the expected names.

See also

  • Last updated on