Edit

Share via


Windows Copilot Runtime overview

Important

Self-contained apps are not supported.

Windows Copilot Runtime provides a variety of AI-powered features available via APIs, allowing you to utilize AI capabilities without the need to find, run, or optimize your own Machine Learning (ML) model. The models that power Windows Copilot Runtime on Copilot+ PCs run locally and continuously in the background.

When utilizing AI features, we recommend that you review: Developing Responsible Generative AI Applications and Features on Windows.

Windows Copilot Runtime features and APIs for Windows apps

Windows Copilot Runtime includes the following features and AI-backed APIs powered by models running locally on the Windows device. **These APIs are currently only available in the latest experimental channel release of the Windows App SDK.

To get started trying available APIs, see Set up your development environment to build Windows Copilot Runtime APIs, this guidance includes code to check whether the required models are available on the user's device.

The Windows App SDK experimental channel includes APIs and features in early stages of development. All APIs in the experimental channel are subject to extensive revisions and breaking changes and may be removed from subsequent releases at any time. Experimental features are not supported for use in production environments and apps that use them cannot be published to the Microsoft Store.

  • Phi Silica: The Phi Silica APIs will ship in the Windows App SDK. Similar to OpenAI's GPT Large Language Model (LLM) that powers ChatGPT, Phi is a Small Language Model (SLM) developed by Microsoft Research to perform language-processing tasks on a local device. Phi Silica is specifically designed for Windows devices with a Neural Processing Unit (NPU), allowing text generation and conversation features to run in a high performance, hardware-accelerated way directly on the device. Phi Silica is not available in mainland China.

  • Text Recognition with OCR: The Text Recognition APIs (also referred to as Optical Character Recognition or OCR) will ship in the Windows App SDK. These APIs enable the recognition of text in an image and the conversion of different types of documents (such as scanned paper documents, PDF files, or images captured by a digital camera) into editable and searchable data on a local device.

  • Imaging APIs: The AI-enhanced Imaging APIs will ship in the Windows App SDK. These APIs perform a variety of actions such as intelligently scaling images and identifying objects within images. Image Description features are not available in mainland China.

  • Studio Effects: Windows devices with compatible Neural Processing Units (NPUs) integrate Studio Effects into the built-in device camera and microphone settings. Apply special effects that utilize AI, including: Background Blur, Eye Contact correction, Automatic Framing, Portrait Light correction, Creative Filters, or Voice Focus for filtering out background noise.

  • Recall (Not currently supported as an API): Recall enables users to quickly find things from their past activity, such as documents, images, websites and more. Developers can enrich the user's Recall experience with their app by adding contextual information to the underlying vector database with the User Activity API. This integration will help users pick up where they left off in your app, improving app engagement and user's seamless flow between Windows and your app.

  • Live Caption Translations (Not currently supported) help everyone on Windows, including those who are deaf or hard of hearing, better understand audio by viewing captions of spoken content (even when the audio content is in a language different from the system's preferred language).

  • Content Moderation: Learn how Windows Copilot Runtime moderates content and how to adjust sensitivity filters.

Integrate AI in enterprise apps using Windows Copilot Runtime APIs

Watch the demo session Integrate AI in Enterprise apps using Windows Copilot Runtime APIs from the November 2024 Ignite Conference.

Additional resources