Language
Edit

Choose how a Task runs your model

Important

Specifying a model for a Task is experimental, not supported for use in production environments, and requires Windows ML 2.9 experimental or later.

Instead of building a pipeline, you can specify a model for a Task. The call that loads the model decides which engine runs it. Windows ML never moves a model from one path to another, so a model that a path doesn't support fails instead of falling back.

Choose a model source and execution path

In each row, your app passes the loaded model to the Task with SetModel, or with SetComposition for exported graphs. The load call decides which engine runs the model:

Load call Model How the Task runs it
LoadModelFromPath A checkpoint folder with config.json, such as a Hugging Face folder of safetensors weights Windows ML compiles the model into programs for the CGC backend and runs it on the GPU target your app picked.
LoadModelFromFile or LoadModelFromPath A GGUF file The same as a checkpoint folder: the CGC backend runs it, not llama.cpp. A file whose architecture or weight types Windows ML doesn't support fails with HRESULT_FROM_WIN32(ERROR_NOT_SUPPORTED).
IWinMLLlamaModelLoader::LoadModelFromFile A GGUF file llama.cpp runs it, for text generation only.
LoadModelFromFile for each exported graph, then SetComposition Exported .onnx graphs, CGC artifacts, or a GGUF decoder Each graph runs on its pipeline backend, composed as your description says.

Note

This table is about passing a model to a Task. A Runtime pipeline that you build with AddModelStage doesn't take GGUF models; GGUF models run only through the Task APIs.

Important

When Windows ML runs a model on the GPU with the CGC backend, try the recommended models first. Other models might not load or might not run well yet.

Text generation from published weights supports dense decoder models as folders of unquantized safetensors weights or as GGUF files, but models outside the recommended list aren't validated yet. To run a GGUF file that Windows ML doesn't support, load it with the llama.cpp loader.

Run published weights

Load the published weights, create a text generation configuration, and call SetModel with a GPU execution target. You pass no tokenizer: Windows ML creates the model's own tokenizer from its files, and you create the Task with that tokenizer.

// Temporary: WinMLRuntime and the other wrapper types in the WinMLSamples.Runtime
// namespaces aren't in the Windows ML package yet. Copy the wrapper files from
// https://github.com/microsoft/WindowsML/tree/main/Samples/Runtime/shared/csharp
// into your project. A future update will include the C# APIs natively.
using WinMLSamples.Runtime;
using WinMLSamples.Runtime.Tasks;

using WinMLRuntime runtime = WinMLRuntime.Create();
using ExecutionTarget target = runtime.CreateExecutionTarget(
    ExecutionTargetKind.Gpu, ExecutionTargetPreference.Performance);

// A checkpoint folder loads with LoadModelFromPath; a single GGUF file
// loads with LoadModelFromFile.
using Model model = runtime.LoadModelFromPath(@"C:\models\qwen2.5-0.5b-instruct");

using WinMLTasks tasks = WinMLTasks.Create(runtime);
using TextGenerationConfiguration configuration =
    tasks.TextGeneration.CreateTextGenerationConfiguration();

// No SetTokenizer: SetModel creates the model's own tokenizer from its files.
using (ModelTextGenerationConfiguration modelConfiguration =
    configuration.As<ModelTextGenerationConfiguration>())
{
    modelConfiguration.SetModel(target, model, new TextGenerationModelOptions());
}

// The Task must use the configuration's tokenizer.
using Tokenizer tokenizer = configuration.GetTokenizer();
using TextGenerationTask task = tasks.TextGeneration.CreateTextGenerationTask(tokenizer);

// A TextGenerator keeps the session on a thread of its own and streams tokens.
await using TextGenerator generator = await TextGenerator.CreateAsync(tasks, task, configuration);

// Stream a reply from the generator.

After binding, every pattern produces the same text generation session, with the same streaming, cancellation, token counts, and finish reasons. A chat completion Task can use it. In C++, you query the configuration for IWinMLModelTextGenerationConfiguration and call SetModel.

Call SetLogicalCapacity after SetModel to set the largest sequence, in tokens, that a session holds. A smaller capacity uses less GPU memory. Without it, a session holds the model's default sequence length.

Cache compiled programs

The first session of a Task that uses the CGC backend compiles programs for the GPU, which can take many seconds. Set a program cache directory in the model options (compiledProgramDirectory, or --program-cache in the samples) to keep the compiled programs for later processes.

  • By default, the cache keeps only the programs, a few megabytes for each model. A later session prepares the weights from the model's files again.
  • WINML_COMPILED_PROGRAM_STORAGE_PROGRAMS_AND_WEIGHTS (--cache-weights in the samples) also keeps the prepared weights. A warm start is faster, but each program keeps its own copy, so the cache can take several times the model's size on disk.
  • A model that can't reuse compiled programs, such as a Whisper checkpoint, refuses the directory with ERROR_NOT_SUPPORTED. The Task then compiles for each session without a cache.

Run an exported model

When you exported your own graphs, load each one with LoadModelFromFile, set the tokenizer with SetTokenizer, and describe the graphs with SetComposition. You describe the inputs, outputs, and state, such as the key/value cache pair of a decoder, instead of wiring each stage. Each session composes the described models from Runtime stages, so you get a Task in a few lines.

Run a GGUF file with llama.cpp

llama.cpp is an engine-specific choice. Windows ML uses it for a model only when your app loads the model with IWinMLLlamaModelLoader (Runtime.load_llama_model in Python), and only for text generation.

  1. Query the runtime for IWinMLLlamaModelLoader. It's present when the Windows ML package includes the llama.cpp engine.
  2. Load the model, and optionally a smaller draft model, with the loader's LoadModelFromFile.
  3. Call SetModel, passing the draft model in the model options and selecting the DRAFT_MODEL speculative method. SetModel reads the tokenizer from the GGUF file.

A draft model must share the model's tokenizer and be loaded through the same loader.

Deploy the llama.cpp backend

The llama.cpp backend is app-local: you deploy its payload with your app. Without the payload, SetModel succeeds but the first CreateSession fails with HRESULT_FROM_WIN32(ERROR_FILE_NOT_FOUND).

  • C++: reference the Microsoft.Windows.AI.MachineLearning.LibLlama.Core NuGet package. It copies the llama.cpp adapter, the llama.cpp core, and the CPU module next to your app. In Windows ML 2.9, the package is on the experimental channel like the main package and carries llama.cpp build 11847.
  • Python: install the windowsml-llama-core wheel (2.9.42a0), which must match the windowsml wheel version.

The package includes only a CPU backend, so a llama.cpp session runs on the CPU. For GPU acceleration, build a ggml backend, such as CUDA or Vulkan, for the same llama.cpp build and backend API version as the package, and place it next to your app. Without a GPU backend, CreateSession for a GPU target fails with HRESULT_FROM_WIN32(ERROR_NOT_SUPPORTED). Rebuild a GPU backend after you update the package.

GGUF files that use the W3_G32 tensor type of earlier Windows ML releases no longer load, so requantize them with a standard llama.cpp type.

For a walkthrough of both steps, see the GGUF models guide and the llama.cpp tools.

Speculative decoding

Speculative decoding lets a faster draft source propose several tokens that the model verifies in one step. With greedy decoding, the text stays the same, apart from a rare near tie between two tokens. For text generation from published weights:

Method Draft model Behavior
NONE None Sequential decoding.
MODEL Optional Drafts with a companion drafter that you load as the draft model, such as a DFlash 2 checkpoint. Without one, it uses the multi-token prediction layers that the model carries.
PROMPT_LOOKUP None Drafts tokens that followed an earlier occurrence of the end of the sequence.
DRAFT_MODEL Required for llama.cpp A separate, smaller model. Published weights don't support this method.

Windows ML never looks for a draft model on its own. A configuration can report whether a method is supported before you create a session, and the terminal result reports the drafted and accepted token counts.

Next steps