Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Windows ML runs machine learning models on your device by pairing the ONNX Runtime with hardware-tuned execution providers (EPs) that target the CPU, GPU, or Neural Processing Unit (NPU). When you run models this way, you often need to minimize power consumption, memory usage, and background resource overhead while maintaining acceptable performance. This need is especially important for continuously running workloads, where background CPU usage, memory pressure, and battery drain directly affect the user experience.
This article shows you how to apply the efficiency mechanism that matches where the work runs. For CPU workloads, the OS scheduler and power manager are the primary levers. For NPU workloads, the most portable lever is to express efficiency intent through the ONNX Runtime so the selected execution provider can map it to device-specific controls.
CPU workloads
Use EcoQoS for background inference that doesn't contribute to the foreground user experience and doesn't require optimal performance.
Don't use EcoQoS for foreground or performance-critical inference.
Opt the process, or preferably the specific inference threads, into EcoQoS by enabling the power-throttling "execution speed" policy.
Use SetThreadInformation with ThreadPowerThrottling (THREAD_POWER_THROTTLING_STATE), or SetProcessInformation with ProcessPowerThrottling (PROCESS_POWER_THROTTLING_STATE).
For a thread, set THREAD_POWER_THROTTLING_EXECUTION_SPEED in both the ControlMask and StateMask members.
For a process, set PROCESS_POWER_THROTTLING_EXECUTION_SPEED in both members.
This setting gives Windows an explicit hint to favor more power-efficient CPU scheduling and performance states for that work.
Note
Apply EcoQoS to an entire process or to individual threads. Use process-level EcoQoS when all work in the process can prioritize efficiency. If only specific work should remain efficiency-oriented, apply EcoQoS at the thread level to just those inference threads rather than the entire process.
NPU workloads
Note
Windows ML execution providers acquired through the EP catalog require Windows 11, version 24H2 (build 26100) or later. NPU inference also requires compatible NPU hardware, a compatible driver, and a supported NPU execution provider.
Use ONNX Runtime's SetEpDynamicOptions() to set ep.dynamic.workload_type=Efficient on the session, so the selected EP can map the signal to device-specific controls such as scheduling priority and power-saver modes.
Currently, only NPU execution providers act on the Efficient signal.
Efficient mode can increase latency when higher-priority accelerator workloads are active.
Use it for background work that doesn't have strict latency requirements.
You set ep.dynamic.workload_type per ONNX Runtime inference session. If you have multiple sessions, configure each session's efficiency mode independently.
Avoid using hardware-vendor-specific performance knobs (for example, QNN htp_performance_mode) directly.
These knobs bypass the OS's intent-based scheduling and power management, and can conflict with system-level policies.
Prefer ep.dynamic.workload_type to express efficiency intent so the OS and EP can coordinate appropriately.
Verify efficiency settings
Check the result of every API call before running inference.
If SetThreadInformation or SetProcessInformation returns zero, call GetLastError and handle the failure.
For process-level EcoQoS, call GetProcessInformation with ProcessPowerThrottling and verify that PROCESS_POWER_THROTTLING_EXECUTION_SPEED is present in both the ControlMask and StateMask members.
For thread-level EcoQoS, check that SetThreadInformation succeeds because GetThreadInformation doesn't support querying ThreadPowerThrottling.
Check the status returned by SetEpDynamicOptions() and treat an error as a configuration failure.
To validate the effect of an efficiency setting, measure inference latency, processor or accelerator utilization, and power consumption under a representative workload.
Memory optimization
If your app is sensitive to memory footprint (for example, an always-on security product), several approaches are available depending on the scenario.
| Approach | What it does | Reference |
|---|---|---|
| Use quantized models (INT8/INT4) | Reduces per-weight storage (for example, FP32 is 4 bytes per weight, INT8 is 1 byte per weight). Actual savings depend on model structure and quantization method. | ONNX Runtime quantization |
Disable the CPU memory arena (DisableCpuMemArena()) |
Disables the arena entirely on CPU, so allocations go directly to the system allocator. Trades throughput for lower peak memory. | DisableCpuMemArena |
Execution provider-specific performance options
For finer-grained control beyond ep.dynamic.workload_type, use EP-specific options exposed through SetEpDynamicOptions() or session configuration (for example, QNN's htp_performance_mode or OpenVINO's PERFORMANCE_HINT).
Prefer the common ep.dynamic.workload_type mechanism when possible, because vendor-specific knobs bypass the OS's intent-based scheduling and power management and can conflict with system-level policies.
Respond to system memory pressure
For processes that host multiple models or need to respond to system memory pressure, consider the following options:
CreateMemoryResourceNotification: Monitor system-wide memory state. On a low-memory notification, query your memory usage to determine what you can evict—for example, release idle sessions, drop cached arenas, or unload models that aren't actively running inference.MEMORY_PRIORITY_INFORMATION: Set lower memory priority on idle model sessions by usingSetProcessInformationwithProcessMemoryPriorityorSetThreadInformationwithThreadMemoryPriority. The OS reclaims pages from lower-priority work first under memory pressure. This approach is useful in multi-model scenarios where only one model is active at a time. It doesn't reduce allocations, but it improves system behavior under pressure.
MEMORY_PRIORITY_INFORMATION doesn't reduce actual memory consumption. It influences how aggressively the OS trims working-set pages.