Language
Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Important
The Chat Completion Task is experimental and not supported for use in production environments.
The Chat Completion Task composes a tokenizer that exposes a structured conversation formatter with a text generation session. You send a list of role-tagged messages. The Task applies the model's own chat template, starts generation, and streams the reply as content, reasoning, and tool-call events.
Your app owns the conversation. It appends each reply to its message list and sends the whole list on the next turn. The text generation session can come from any way of choosing how a Task runs your model, and the chat session, events, and result work the same for each.
Complete a conversation
The following examples assume that you already have a text generation Task, configuration, and tokenizer. The chat session wraps the text generation session, and the tokens that session holds carry over from one turn to the next.
Create a ChatGenerator from the text generation Task, configuration, and tokenizer. It creates the text generation session and the chat session over it, and keeps both on a thread of their own, because a session belongs to the thread that created it. It streams replies as IAsyncEnumerable<T>:
// Temporary: ChatGenerator and the other wrapper types in the WinMLSamples.Runtime
// namespaces aren't in the Windows ML package yet. Copy the wrapper files from
// https://github.com/microsoft/WindowsML/tree/main/Samples/Runtime/shared/csharp
// into your project. A future update will include the C# APIs natively.
using WinMLSamples.Runtime;
using WinMLSamples.Runtime.Tasks;
// The chat session wraps the text generation session, and the tokens that
// session holds carry over from one turn to the next.
await using ChatGenerator chat = await ChatGenerator.CreateAsync(tasks, task, configuration, tokenizer);
Each turn sends the full history:
// Send the conversation, stream the reply, and return it so that
// the caller can add it to the history.
static async Task<string> CompleteTurnAsync(
ChatGenerator chat,
TextGenerationOptions options,
List<ConversationMessage> history,
CancellationToken cancellationToken)
{
ConversationRequest request = new()
{
FormatterMode = ConversationFormatterMode.ModelDefault,
Messages = [.. history],
AddGenerationPrompt = true,
};
// Content events carry the reply as the model generates it.
await foreach (ChatCompletionUpdate update in chat.GenerateAsync(request, options, cancellationToken))
{
if (update.Type == ConversationOutputEventType.ContentDelta)
{
Console.Write(update.Text);
}
else if (update.Type == ConversationOutputEventType.Error)
{
throw WinMLException.GetExceptionForHResult(update.ErrorCode);
}
}
// The outcome has the reply, the token counts, and how many prompt tokens
// the session reused from the previous turn.
ChatCompletionOutcome outcome = chat.LastOutcome!;
Console.WriteLine(
$"\n Tokens: prompt={outcome.PromptTokenCount} " +
$"(reused={outcome.PromptCacheUse.CachedPromptTokenCount}) " +
$"completion={outcome.CompletionTokenCount}");
return outcome.Content;
}
Call it once for each question, and add the reply to the history after each turn:
string[] questions =
[
"Name three primary colors.",
"Which of those is the color of the sky?",
];
using TextGenerationOptions options = tasks.TextGeneration.CreateTextGenerationOptions(48);
List<ConversationMessage> history = [];
foreach (string question in questions)
{
history.Add(new ConversationMessage { Role = "user", Content = question });
Console.Write($"User: {question}\nAssistant: ");
string reply = await CompleteTurnAsync(chat, options, history, CancellationToken.None);
history.Add(new ConversationMessage { Role = "assistant", Content = reply });
}
// Closing the sessions reports a failure that disposing them would not.
await chat.CloseAsync();
Cancelling the CancellationToken stops generation in the Runtime.
Reuse the conversation
When the formatted prompt starts with the token sequence that the session retained, generation continues with the new suffix instead of evaluating the whole prompt again. The result content is the same either way. Only the amount of prompt work differs. The result reports the full prompt count, the retained-prefix count, and the tokens evaluated, so you can see the reuse:
User: Name three primary colors.
Assistant: Three primary colors are red, blue, and yellow.
Tokens: prompt=34 (reused=0) completion=11
User: Which of those is the color of the sky?
Assistant: The color of the sky is blue.
Tokens: prompt=65 (reused=45) completion=8
Results and cancellation
The terminal result reports the final content, reasoning, tool calls, prompt and completion token counts, the finish reason, and the underlying text generation result. Pass an IWinMLCancellationSource to Generate and call Cancel from any thread to stop generation.
Sessions and streams belong to the thread that created them. Calls from another thread return RPC_E_WRONG_THREAD.
Image input
A chat session can pass images to a model that supports them. Set a media encoder with SetMediaEncoder, then call GenerateWithMedia with bindings that match each image content part to a tensor.