Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Windows Copilot Runtime provides a variety of AI-powered features available via APIs, allowing you to utilize AI capabilities without the need to find, run, or optimize your own Machine Learning (ML) model. The models that power Windows Copilot Runtime on Copilot+ PCs run locally and continuously in the background.
When utilizing AI features, we recommend that you review: Developing Responsible Generative AI Applications and Features on Windows.
Windows Copilot Runtime APIs
Windows Copilot Runtime includes the following features and AI-backed APIs powered by models running locally on the Windows device. These APIs will ship in the Windows App SDK, and are currently only available in the latest experimental channel release of the Windows App SDK.
To get started trying available APIs, see Set up your development environment to build Windows Copilot Runtime APIs, this guidance includes code to check whether the required models are available on the user's device.
The Windows App SDK experimental channel includes APIs and features in early stages of development. All APIs in the experimental channel are subject to extensive revisions and breaking changes and may be removed from subsequent releases at any time. Experimental features are not supported for use in production environments and apps that use them cannot be published to the Microsoft Store.
Phi Silica
Similar to OpenAI's GPT Large Language Model (LLM) that powers ChatGPT, Phi is a Small Language Model (SLM) developed by Microsoft Research to perform language-processing tasks on a local device. Phi Silica is specifically designed for Windows devices with a Neural Processing Unit (NPU), allowing text generation and conversation features to run in a high performance, hardware-accelerated way directly on the device. Phi Silica is not available in China.
Text Recognition
The Text Recognition APIs enable the recognition of text in an image and the conversion of different types of documents (such as scanned paper documents, PDF files, or images captured by a digital camera) into editable and searchable data on a local device.
Get started with Text Recognition
Image Super Resolution
The Image Super Resolution APIs enable image sharpening and scaling.
Get started with Image Super Resolution
Image Segmentation
The Image Segmentation APIs enable segmentation of images.
Get started with Image Segmentation
Image Description
The Image Description APIs describes images in natural language. (Image Description features are not available in China.)
Get started with Image Description
Content Moderation
Content Moderation: Learn how Windows Copilot Runtime moderates content and how to adjust sensitivity filters.
Additional AI features
Studio Effects: Windows devices with compatible Neural Processing Units (NPUs) integrate Studio Effects into the built-in device camera and microphone settings. Apply special effects that utilize AI, including: Background Blur, Eye Contact correction, Automatic Framing, Portrait Light correction, Creative Filters, or Voice Focus for filtering out background noise.
Recall (Not currently supported as an API): Recall enables users to quickly find things from their past activity, such as documents, images, websites and more. Developers can enrich the user's Recall experience with their app by adding contextual information to the underlying vector database with the User Activity API. This integration will help users pick up where they left off in your app, improving app engagement and user's seamless flow between Windows and your app.
Live Caption Translations (Not yet supported) help everyone on Windows, including those who are deaf or hard of hearing, better understand audio by viewing captions of spoken content (even when the audio content is in a language different from the system's preferred language).
Integrate AI in enterprise apps using Windows Copilot Runtime APIs
Watch the demo session Integrate AI in Enterprise apps using Windows Copilot Runtime APIs from the November 2024 Ignite Conference.
Additional resources
- Get started with AI on Windows: Windows Copilot Runtime implements a Text Content Moderation API to flag and filter out potentially harmful content. Learn more about this feature and how to adjust the filter sensitivity.
- Developing Responsible Generative AI Applications and Features on Windows: Guidance for responsibly developing apps that incorporate AI.
- AI on Windows Sample Gallery: A collection of samples that demonstrate a variety of ways to enhance your Windows apps using AI.