Language
Edit

Windows ML Task APIs overview

Important

The Windows ML Task APIs are experimental and not supported for use in production environments. Apps that try these APIs shouldn't be published to the Microsoft Store.

The Windows ML Task APIs enable common AI tasks, such as text generation, chat conversations, speech recognition, and answering typed questions, over a model that your app supplies. A Task handles tokenization, decoding, streaming, and cancellation, so you don't write the token loop yourself. Windows doesn't ship a model for these APIs: you choose, package, and distribute the model with your app.

The Task APIs build on the Windows ML Runtime API. Windows ML 2.9 experimental lets you specify a model for a Task, instead of building a pipeline. For models, see Find or train models.

Tip

If you want on-device AI models that Microsoft manages, see ready-to-use AI models and APIs instead.

Available tasks

Task What it does Article
Text generation Runs the token loop over a language model with sampling, streaming, cancellation, and optional speculative decoding. Generate text
Chat completion Applies the model's chat template to a list of messages and streams the reply. Chat completion
Speech recognition Transcribes audio from a speech model and streams the transcript. Recognize speech
Decision Answers a typed question about a text with a probability for each option, and generates no text. Make decisions

Choose how your model runs

How your app loads the model decides how Windows ML runs it. Windows ML doesn't choose among engines for you and never falls back from one to another. For details, see Choose how a Task runs your model.

If you have Your app Windows ML
Published weights, as a checkpoint folder or a GGUF file Loads the weights and calls SetModel. Compiles the model into programs for the GPU with the CGC backend and creates its tokenizer from the model's files. The model must be one that Windows ML supports.
Graphs you exported yourself Loads the graphs, sets a tokenizer, and describes the graphs with SetComposition. Runs each graph on its pipeline backend, composed as you describe, so you don't wire each stage.
A GGUF file that you want llama.cpp to run Loads the file with IWinMLLlamaModelLoader and calls SetModel. Runs it on llama.cpp, for text generation only. Use it for a model that the CGC backend doesn't support, or for speculative decoding with a draft model.

For full control over stages, tensor bindings, and backend options, you can still build the pipelines yourself and configure the Task by hand.

Requirements

  • Windows ML 2.9 experimental or later to specify a model for a Task. Earlier packages don't include these APIs, and changing only the package version number doesn't enable them.
  • A GPU with Direct3D 12 for models that run on the CGC backend. These models run on the GPU only, with no CPU fallback. For now, use the recommended models.
  • C++, C#, or Python. C# needs a temporary wrapper that you copy into your project. A future update will include the C# APIs natively. See Set up a project.

Samples

The Windows ML samples repository includes samples that specify a model for a Task, for text generation, composed text generation, llama.cpp text generation, speech recognition, and decisions, in C++ and Python. C# samples cover text generation, chat completion, speech recognition, and decisions. See Windows ML samples.

Next steps