Edit

Share via

Windows Copilot Runtime Overview

  • Article

Microsoft Copilot stack on Windows democratizes the ability to experiment, build, and reach people with breakthrough AI experiences, putting the developer in control. As a developer, you can integrate your apps with AI-powered Windows experiences such as Recall and Studio Effects, leverage new APIs powered by on-device models through the Windows Copilot Library, discover Machine Learning (ML) models to fine-tune with your own customized data using AI Toolkit in Visual Studio Code, integrate your own ML models using frameworks like ONNX Runtime, PyTorch, or WebNN, and access hardware-acceleration for better performance and scale through DirectML.

Windows Copilot Runtime and Windows Copilot Library

Windows Copilot Runtime introduces new ways of interacting with the operating system that utilize AI, such as Phi Silica, the Small Language Model (SLM) created by Microsoft Research that is able to offer many of the same capabilities found in Large Language Models (LLMs), but more compact and efficient so that it can run locally on Windows.

There are new innovations that use AI to improve and redefine the Windows search experience as well. Using a "Semantic Index", Windows now offers AI-assisted discovery to power Recall, an API that helps Windows users retrieve past activities and continue where they left off, as well as an AI-assisted search in File Explorer to help speed up your ability to find the files you’re looking for based on entering whatever clues you remember.

While some AI innovations are baked right into using Windows, others are available for App Developers to integrate into their app features. These new ways of integrating AI in your app form the Windows Copilot Library, a list of ready-to-use AI-backed features and APIs, including:

  • Studio Effects: Enhances the camera and audio capabilities of Windows devices with AI-powered background effects, eye contact correction, automatic framing, voice focus, blur, lighting, and creative filters run on the device NPU to maintain fast performance speeds.
  • Recall: Makes past activities on your Windows device searchable, so that you can pick up where you left off, whether that was using an app, editing a document, or responding to an email.
  • Phi Silica: Enables your app connect with the on-device Phi model for natural language processing tasks (chat, math, code, reasoning) using the Windows App SDK.
  • Text Recognition: Optical Character Recognition, or OCR, enables the extraction of text from images and documents. Imagine tasks like converting a PDF, paper document, or picture of a classroom white board into editable digital text.
  • Live Caption Translations: Helps everyone on Windows, including those who are deaf or hard of hearing, better understand audio by viewing captions of spoken content, even when the audio content is in a language different from the system's preferred language.

Developers will be able to access these APIs as part of the Windows App SDK.

In addition to the ready-to-use AI-backed APIs in Windows Copilot Library, we have guidance on how to enhance your app using Machine Learning (ML) models. This covers topics like:

Leading with responsibility and examples

We have also created resources to support our developers seeking to integrate AI into your Windows applications by providing a Samples Gallery, a guide for Responsible AI use, and some high-level FAQs that can help with unpacking some of the terminology and concepts.

Get started adding models to your Windows app