Editja

Choose between cloud-based and local AI models

For app developers seeking to integrate AI features, Microsoft Windows offers a comprehensive and flexible platform that supports both local, on-device processing and scalable, cloud-based solutions.

Choosing between cloud-based and local AI models depends on your specific needs and priorities. Many production apps use a hybrid strategy: try a local Windows AI API or local model first, then fall back to a cloud endpoint when the model isn't installed, the device isn't supported, the user doesn't consent to a model download, or the task requires a larger model.

Factors to consider include:

  • Data privacy, compliance, and security
  • Resource availability
  • Accessibility and collaboration
  • Cost
  • Maintenance and updates
  • Performance & latency
  • Scalability
  • Connectivity requirements
  • Model size and complexity
  • Tooling and the associated ecosystem
  • Customization and control

Key decision factors for app developers

  • Data privacy, compliance, and security

    • Local, on-premises: Since data remains on the device, running a model locally can offer benefits regarding security and privacy, with the responsibility of data security resting on the user. The developer holds responsibility for managing updates, ensuring compatibility, and monitoring security vulnerabilities.

    • Cloud: Cloud providers offer robust security measures, but data needs to be transferred to the cloud, which might raise data privacy concerns for the business or app service maintainer in some cases. Sending data to the cloud also must comply with data protection regulations, such as GDPR or HIPAA, depending on the nature of the data and the region in which the app operates. Cloud providers typically handle security updates and maintenance, but users must ensure that they are using secure APIs and following best practices for data handling.

  • Resource availability

    • Local, on-premises: Running a model depends on the resources available on the device being used, including the CPU, GPU, NPU, memory, and storage capacity. This can be limiting if the device does not have high computational power or sufficient storage. Small Language Models (SLMs), like Phi, are more suitable for local use on a device. Copilot+ PCs offer built-in models with ready-to-use AI features supported by Microsoft Foundry on Windows.

    • Cloud: Cloud platforms, such as Azure AI Services, offer scalable resources. You can use as much computational power or storage as you need and only pay for what you use. Large Language Models (LLMs), like the OpenAI language models, require more resources, but are also more powerful.

  • Accessibility and collaboration

    • Local, on-premises: The model and data are accessible only on the device unless shared manually. This has the potential to make collaboration on model data more challenging.

    • Cloud: The model and data can be accessed from anywhere with internet connectivity. This may be better for collaboration scenarios.

  • Cost

    • Local, on-premises: There is no additional cost beyond the initial investment in the device hardware.

    • Cloud: While cloud platforms operate on a pay-as-you-go model, costs can accumulate based on the resources used and the duration of usage.

  • Maintenance and Updates

    • Local, on-premises: The user is responsible for maintaining the system and installing updates.

    • Cloud: Maintenance, system updates, and new feature updates are handled by the cloud service provider, reducing maintenance overhead for the user.

  • Performance & Latency

    • Local, on-premises: Running a model locally can reduce latency since data does not need to be sent over the network. However, performance is limited by the device's hardware capabilities.

    • Cloud: Cloud-based models can leverage powerful hardware, but they may introduce latency due to network communication. The performance can vary based on the user's internet connection and the cloud service's response time.

  • Scalability

    • Local, on-premises: Scaling a model on a local device may require significant hardware upgrades or the addition of more devices, which can be costly and time-consuming.

    • Cloud: Cloud platforms offer easy scalability, allowing you to quickly adjust resources based on demand without the need for physical hardware changes.

  • Connectivity Requirements

    • Local, on-premises: A local device does not require an internet connection to run a model, which can be beneficial in environments with limited connectivity.

    • Cloud: Cloud-based models require a stable internet connection for access and may be affected by network issues.

  • Runtime availability and fallback

    • Local, on-premises: Local AI features can depend on hardware, Windows version, Windows App SDK version, region, and whether an optional model is installed. Before invoking a local AI feature, check its readiness state and guide the user through any required model download or consent flow. For Windows AI APIs, use the API-specific readiness pattern documented in Get started with Windows AI APIs and the individual API pages.

    • Cloud: A cloud endpoint can provide a fallback when a local model isn't ready or supported on the current device. Decide whether fallback is automatic, user-controlled, or disabled for privacy-sensitive scenarios. Make sure the UI explains when data leaves the device.

  • Model Size and Complexity

    • Local, on-premises: Local devices may have limitations on the size and complexity of models that can be run due to hardware constraints. Smaller models, such as Phi, are more suitable for local execution.

    • Cloud: Cloud platforms can handle larger and more complex models, such as those provided by OpenAI, due to their scalable infrastructure.

  • Tooling and the associated ecosystem

    • Local, on-premises: Local AI solutions, such as Microsoft Foundry on Windows, Windows ML, and Foundry Local, integrate with Windows App SDK and ONNX Runtime, allowing developers to embed models directly into desktop or edge apps with minimal external dependencies.

    • Cloud: Cloud AI solutions, such as Microsoft Foundry, Azure AI Services, and Azure OpenAI Service, provide a comprehensive set of APIs and SDKs for building AI applications. These services are designed to integrate seamlessly with Azure DevOps, GitHub Copilot, Semantic Kernel, and other Azure services, enabling end-to-end orchestration, model deployment, and monitoring at scale.

  • Customization and Control

    • Local, on-premises: Local models can be used out-of-the-box, without requiring a high level of expertise. Microsoft Foundry on Windows offers models like Phi Silica that are ready to use. Alternatively, Windows ML allows developers to run custom models, such as those trained with ONNX Runtime, directly on Windows devices. This provides a high level of control over the model and its behavior, allowing for fine-tuning and optimization based on specific use cases. Foundry Local also allows developers to run models locally on Windows devices, providing a high level of control over the model and its behavior.

    • Cloud: Cloud-based models also offer both ready-to-use and customizable options, allowing developers to leverage pre-trained capabilities while still tailoring the model to their specific needs. Microsoft Foundry is a unified Azure platform-as-a-service offering for enterprise AI operations, model builders, and application development. This foundation combines production-grade infrastructure with friendly interfaces, enabling developers to focus on building applications rather than managing infrastructure.

Cloud AI Samples

If a cloud-based solution works better for your Windows app scenario, you may be interested in some of the tutorials below.

Many APIs are available for accessing cloud-based models to power AI features in your Windows app, whether those models are customized or ready-to-use. Using a cloud-based model can allow your app to remain streamlined by delegating resource-intensive tasks to the cloud. A few resources to help you add cloud-based AI-backed APIs offered by Microsoft or OpenAI include:

  • Add OpenAI chat completions to your WinUI 3 / Windows App SDK desktop app: A tutorial on how to integrate the cloud-based OpenAI ChatGPT completion capabilities into a WinUI 3 / Windows App SDK desktop app.

  • Add DALL-E to your WinUI 3 / Windows App SDK desktop app: A tutorial on how to integrate the cloud-based OpenAI DALL-E image generation capabilities into a WinUI 3 / Windows App SDK desktop app.

  • Azure OpenAI Service: If you want your Windows app to access OpenAI models, such as GPT-4, GPT-4 Turbo with Vision, GPT-3.5-Turbo, DALLE-3 or the Embeddings model series, with the added security and enterprise capabilities of Azure, you can find guidance in this Azure OpenAI documentation.

  • Azure AI Services: Azure offers an entire suite of AI services available through REST APIs and client library SDKs in popular development languages. For more information, see each service's documentation. These cloud-based services help developers and organizations rapidly create intelligent, cutting-edge, market-ready, and responsible applications with out-of-the-box and prebuilt and customizable APIs and models. Example applications include natural language processing for conversations, search, monitoring, translation, speech, vision, and decision-making.

Design a hybrid local/cloud path

Use a hybrid path when your app should take advantage of local inference when available, but still provide a useful experience on unsupported devices or before a local model is ready.

  1. Choose the local capability first. Start with a Windows AI API when it matches your scenario, or use Windows ML / Foundry Local when you need to run a specific model.
  2. Check readiness before use. At startup or before the feature is shown, query the local feature's readiness state. If the API reports that the feature isn't supported, isn't installed, or needs a download, don't invoke it blindly.
  3. Ask for consent when a local model must be downloaded. Some local models are optional and can be several GB. Explain the download size, why the model is needed, and whether the app can continue without it.
  4. Define a cloud fallback. If the local model isn't ready, call a cloud endpoint only when the user and organization allow data to leave the device. Keep the local and cloud paths behind the same app-level abstraction so the rest of the app doesn't depend on where inference runs.
  5. Keep behavior observable. Log which path was used, readiness failures, model-download outcomes, and cloud fallback errors without logging prompts, tokens, or sensitive content unless your organization has approved that data handling.

Hybrid fallback decision flow

Use this flow as a starting point:

  1. When the user starts an AI feature, check whether a local AI feature fits the task.
  2. If no local feature fits the task, use cloud AI or another service.
  3. If a local feature fits the task, check whether the feature is ready on the current device.
  4. If the local feature is ready, run local inference.
  5. If the local feature needs a model download, ask for user consent before starting the download.
  6. If the user consents, download or prepare the model, then retry local inference.
  7. If the user doesn't consent, or if the local feature is unsupported, check whether cloud fallback is allowed for the scenario.
  8. If cloud fallback is allowed, use the cloud path.
  9. If cloud fallback isn't allowed, explain the device, model, or policy requirement and disable or hide the feature.

For local readiness details, see Windows AI APIs, Get started with Windows AI APIs, Foundry Local, and Windows ML.

  • Last updated on