Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
The ability to build intelligent AI experiences on, and with, Windows is developing rapidly. Windows AI Foundry offers AI-backed features and APIs on Windows 11 PCs. These features are in active development and run locally in the background at all times.
Windows AI Foundry includes several components that can enable unique AI experiences:
Windows AI APIs: you can use built-in Windows AI APIs to integrate AI services for users on Copilot+ PCs. A few of these ready-to-use AI features that you can tap into from your Windows app include:
- Phi Silica: a local, ready-to-use language model (text generation)
- AI Imaging: use AI to scale and sharpen an image (Image Super Resolution), generate text that describes an image (Image Description), identify objects within an image (Image Segmentation), and remove objects from an image (Object Erase).
- Text Recognition: use AI to detect and extract text within images and convert it into machine readable character streams
Foundry Local: popular OSS models that you can leverage and pull into your app
Windows ML: enables AI inferencing with your own ONNX model
Dev Tools: tooling such as Visual Studio AI Toolkit and AI Dev Gallery that will help you be successful building AI capabilities
How can you use AI in your Windows app?
A few ways that Windows apps can leverage Machine Learning (ML) models to enhance their functionality and user experience with AI, include:
- Apps can use Generative AI models to understand complex topics to summarize, rewrite, report on, or expand.
- Apps can use models that transform free-form content into a structured format that your app can understand.
- Apps can use Semantic Search models that allow users to search for content by meaning and quickly find related content.
- Apps can use natural language processing models to reason over complex natural language requirements, and plan and execute actions to accomplish the user's ask.
- Apps can use image manipulation models to intelligently modify images, erase or add subjects, upscale, or generate new content.
- Apps can use predictive diagnostic models to help identify and predict issues and help guide the user or do it for them.
Using Windows AI APIs versus bringing your own models
Use Windows AI APIs
When a local AI model is the right solution, you can use Windows AI APIs to integrate AI services for users on Copilot+ PCs. These APIs are built-in on your PC and enable unique AI-powered features with relatively little overhead.
Train your own model
If you have the ability to train your own model using your own private data with platforms like TensorFlow or PyTorch. You can integrate that custom model into your Windows application by running it locally on the device hardware using ONNX Runtime and AI Toolkit for Visual Studio Code.
AI Toolkit for Visual Studio Code is a VS Code Extension that enables you to download and run AI models locally, including access to hardware-acceleration for better performance and scale through DirectML. The AI Tookit can also help you with:
- Testing models in an intuitive playground or in your application with a REST API.
- Fine-tuning your AI model, both locally or in the cloud (on a virtual machine) to create new skills, improve reliability of responses, set the tone and format of the response.
- Fine-tuning popular small-language models (SLMs), like Phi-3 and Mistral.
- Deploy your AI feature either to the cloud or with an application that runs on a device.
- Leverage hardware acceleration for better performance with AI features using DirectML. DirectML is a low-level API that enables your Windows device hardware to accelerate the performance of ML models using the device GPU or NPU. Pairing DirectML with the ONNX Runtime is typically the most straightforward way for developers to bring hardware-accelerated AI to their users at scale. Learn more: DirectML Overview.
- Quantize and validate a model for use on NPU by using the model conversion capabilities
You may also want to look into these model fine-tuning concepts to adjust a pre-trained model to better fit your data.
Using Cloud AI Models
If using local AI features isn't the right path for you, using Cloud AI models and resources can be a solution.
Other AI Features
App Actions on Windows: create actions for your app enabling new and unique AI capabilities for consumers
Recall utilizes AI to help you find anything you've seen on your PC. Click to Do is an AI-supported feature connect actions to the content (text or images) found by Recall.
Windows Studio Effects utilizes AI to apply special effects to the device camera
Use Responsible AI practices
Whenever you are incorporating AI features in your Windows app, we highly recommend following the Developing Responsible Generative AI Applications and Features on Windows guidance.