Language
Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Important
Specifying a model for a Task is experimental, not supported for use in production environments, and requires Windows ML 2.9 experimental or later.
Instead of building a pipeline, you can specify a model for a Task. The call that loads the model decides which engine runs it. Windows ML never moves a model from one path to another, so a model that a path doesn't support fails instead of falling back.
Choose a model source and execution path
In each row, your app passes the loaded model to the Task with SetModel, or with SetComposition for exported graphs. The load call decides which engine runs the model:
| Load call | Model | How the Task runs it |
|---|---|---|
LoadModelFromPath |
A checkpoint folder with config.json, such as a Hugging Face folder of safetensors weights |
Windows ML compiles the model into programs for the CGC backend and runs it on the GPU target your app picked. |
LoadModelFromFile or LoadModelFromPath |
A GGUF file | The same as a checkpoint folder: the CGC backend runs it, not llama.cpp. A file whose architecture or weight types Windows ML doesn't support fails with HRESULT_FROM_WIN32(ERROR_NOT_SUPPORTED). |
IWinMLLlamaModelLoader::LoadModelFromFile |
A GGUF file | llama.cpp runs it, for text generation only. |
LoadModelFromFile for each exported graph, then SetComposition |
Exported .onnx graphs, CGC artifacts, or a GGUF decoder |
Each graph runs on its pipeline backend, composed as your description says. |
Note
This table is about passing a model to a Task. A Runtime pipeline that you build with AddModelStage doesn't take GGUF models; GGUF models run only through the Task APIs.
Important
When Windows ML runs a model on the GPU with the CGC backend, try the recommended models first. Other models might not load or might not run well yet.
Text generation from published weights supports dense decoder models as folders of unquantized safetensors weights or as GGUF files, but models outside the recommended list aren't validated yet. To run a GGUF file that Windows ML doesn't support, load it with the llama.cpp loader.
Run published weights
Load the published weights, create a text generation configuration, and call SetModel with a GPU execution target. You pass no tokenizer: Windows ML creates the model's own tokenizer from its files, and you create the Task with that tokenizer.
// Temporary: WinMLRuntime and the other wrapper types in the WinMLSamples.Runtime
// namespaces aren't in the Windows ML package yet. Copy the wrapper files from
// https://github.com/microsoft/WindowsML/tree/main/Samples/Runtime/shared/csharp
// into your project. A future update will include the C# APIs natively.
using WinMLSamples.Runtime;
using WinMLSamples.Runtime.Tasks;
using WinMLRuntime runtime = WinMLRuntime.Create();
using ExecutionTarget target = runtime.CreateExecutionTarget(
ExecutionTargetKind.Gpu, ExecutionTargetPreference.Performance);
// A checkpoint folder loads with LoadModelFromPath; a single GGUF file
// loads with LoadModelFromFile.
using Model model = runtime.LoadModelFromPath(@"C:\models\qwen2.5-0.5b-instruct");
using WinMLTasks tasks = WinMLTasks.Create(runtime);
using TextGenerationConfiguration configuration =
tasks.TextGeneration.CreateTextGenerationConfiguration();
// No SetTokenizer: SetModel creates the model's own tokenizer from its files.
using (ModelTextGenerationConfiguration modelConfiguration =
configuration.As<ModelTextGenerationConfiguration>())
{
modelConfiguration.SetModel(target, model, new TextGenerationModelOptions());
}
// The Task must use the configuration's tokenizer.
using Tokenizer tokenizer = configuration.GetTokenizer();
using TextGenerationTask task = tasks.TextGeneration.CreateTextGenerationTask(tokenizer);
// A TextGenerator keeps the session on a thread of its own and streams tokens.
await using TextGenerator generator = await TextGenerator.CreateAsync(tasks, task, configuration);
// Stream a reply from the generator.
After binding, every pattern produces the same text generation session, with the same streaming, cancellation, token counts, and finish reasons. A chat completion Task can use it. In C++, you query the configuration for IWinMLModelTextGenerationConfiguration and call SetModel.
Call SetLogicalCapacity after SetModel to set the largest sequence, in tokens, that a session holds. A smaller capacity uses less GPU memory. Without it, a session holds the model's default sequence length.
Cache compiled programs
The first session of a Task that uses the CGC backend compiles programs for the GPU, which can take many seconds. Set a program cache directory in the model options (compiledProgramDirectory, or --program-cache in the samples) to keep the compiled programs for later processes.
- By default, the cache keeps only the programs, a few megabytes for each model. A later session prepares the weights from the model's files again.
WINML_COMPILED_PROGRAM_STORAGE_PROGRAMS_AND_WEIGHTS(--cache-weightsin the samples) also keeps the prepared weights. A warm start is faster, but each program keeps its own copy, so the cache can take several times the model's size on disk.- A model that can't reuse compiled programs, such as a Whisper checkpoint, refuses the directory with
ERROR_NOT_SUPPORTED. The Task then compiles for each session without a cache.
Run an exported model
When you exported your own graphs, load each one with LoadModelFromFile, set the tokenizer with SetTokenizer, and describe the graphs with SetComposition. You describe the inputs, outputs, and state, such as the key/value cache pair of a decoder, instead of wiring each stage. Each session composes the described models from Runtime stages, so you get a Task in a few lines.
Run a GGUF file with llama.cpp
llama.cpp is an engine-specific choice. Windows ML uses it for a model only when your app loads the model with IWinMLLlamaModelLoader (Runtime.load_llama_model in Python), and only for text generation.
- Query the runtime for
IWinMLLlamaModelLoader. It's present when the Windows ML package includes the llama.cpp engine. - Load the model, and optionally a smaller draft model, with the loader's
LoadModelFromFile. - Call
SetModel, passing the draft model in the model options and selecting theDRAFT_MODELspeculative method.SetModelreads the tokenizer from the GGUF file.
A draft model must share the model's tokenizer and be loaded through the same loader.
Deploy the llama.cpp backend
The llama.cpp backend is app-local: you deploy its payload with your app. Without the payload, SetModel succeeds but the first CreateSession fails with HRESULT_FROM_WIN32(ERROR_FILE_NOT_FOUND).
- C++: reference the
Microsoft.Windows.AI.MachineLearning.LibLlama.CoreNuGet package. It copies the llama.cpp adapter, the llama.cpp core, and the CPU module next to your app. In Windows ML 2.9, the package is on the experimental channel like the main package and carries llama.cpp build 11847. - Python: install the
windowsml-llama-corewheel (2.9.42a0), which must match thewindowsmlwheel version.
The package includes only a CPU backend, so a llama.cpp session runs on the CPU. For GPU acceleration, build a ggml backend, such as CUDA or Vulkan, for the same llama.cpp build and backend API version as the package, and place it next to your app. Without a GPU backend, CreateSession for a GPU target fails with HRESULT_FROM_WIN32(ERROR_NOT_SUPPORTED). Rebuild a GPU backend after you update the package.
GGUF files that use the W3_G32 tensor type of earlier Windows ML releases no longer load, so requantize them with a standard llama.cpp type.
For a walkthrough of both steps, see the GGUF models guide and the llama.cpp tools.
Speculative decoding
Speculative decoding lets a faster draft source propose several tokens that the model verifies in one step. With greedy decoding, the text stays the same, apart from a rare near tie between two tokens. For text generation from published weights:
| Method | Draft model | Behavior |
|---|---|---|
NONE |
None | Sequential decoding. |
MODEL |
Optional | Drafts with a companion drafter that you load as the draft model, such as a DFlash 2 checkpoint. Without one, it uses the multi-token prediction layers that the model carries. |
PROMPT_LOOKUP |
None | Drafts tokens that followed an earlier occurrence of the end of the sequence. |
DRAFT_MODEL |
Required for llama.cpp | A separate, smaller model. Published weights don't support this method. |
Windows ML never looks for a draft model on its own. A configuration can report whether a method is supported before you create a session, and the terminal result reports the drafted and accepted token counts.