Implementing resource controls for Windows containers
Applies to: Windows Server 2022, Windows Server 2019
There are several resource controls that can be implemented on a per-container and per-resource basis. By default, containers run are subject to typical Windows resource management, which in general is fair-share based but though configuration of these controls a developer or administrator can limit or influence resource usage. Resources that can be controlled include: CPU/Processor, Memory/RAM, Disk/Storage and Networking/Throughput.
Windows containers utilize job objects to group and track processes associated with each container. Resource controls are implemented on the parent job object associated with the container.
In the case of Hyper-V isolation resource controls are applied both to the virtual machine as well as to the job object of the container running inside the virtual machine automatically, this ensures that even if a process running in the container bypassed or escaped the job objects controls the virtual machine would ensure it was not able to exceed the defined resource controls.
Resources
For each resource this section provides a mapping between the Docker command line interface as an example of how the resource control might be used (it might be configured by an orchestrator or other tooling) to the corresponding Windows host compute service (HCS) API as well as generally how the resource control has been implemented by Windows (please note that this description is high-level and that the underlying implementation is subject to change).
Memory
| Resource | Location |
|---|---|
| Docker interface | --memory |
| HCS interface | MemoryMaximumInMB |
| Shared Kernel | JOB_OBJECT_LIMIT_JOB_MEMORY |
| Hyper-V isolation | Virtual machine memory |
Note
For Hyper-V isolation in Windows Server 2016, when using a memory cap, you'll see the container initially allocate the cap amount of memory, then start to return it to the container host. Later versions of Windows Server (1709 or later) have optimized this process.
CPU (count)
| Resource | Location |
|---|---|
| Docker interface | --cpus |
| HCS interface | ProcessorCount |
| Shared Kernel | Simulated with JOB_OBJECT_CPU_RATE_CONTROL_HARD_CAP* |
| Hyper-V isolation | Number of virtual processors exposed |
CPU (percent)
| Resource | Location |
|---|---|
| Docker interface | --cpu-percent |
| HCS interface | ProcessorMaximum |
| Shared Kernel | JOB_OBJECT_CPU_RATE_CONTROL_HARD_CAP |
| Hyper-V isolation | Hypervisor limits on virtual processors |
CPU (shares)
| Resource | Location |
|---|---|
| Docker interface | --cpu-shares |
| HCS interface | ProcessorWeight |
| Shared Kernel | JOB_OBJECT_CPU_RATE_CONTROL_WEIGHT_BASED |
| Hyper-V isolation | Hypervisor virtual processors weights |
Storage (image)
| Resource | Location |
|---|---|
| Docker interface | --io-maxbandwidth/--io-maxiops |
| HCS interface | StorageIOPSMaximum and StorageBandwidthMaximum |
| Shared Kernel | JOBOBJECT_IO_RATE_CONTROL_INFORMATION |
| Hyper-V isolation | JOBOBJECT_IO_RATE_CONTROL_INFORMATION |
Storage (volumes)
| Resource | Location |
|---|---|
| Docker interface | --storage-opt size= |
| HCS interface | StorageSandboxSize |
| Shared Kernel | JOBOBJECT_IO_RATE_CONTROL_INFORMATION |
| Hyper-V isolation | JOBOBJECT_IO_RATE_CONTROL_INFORMATION |
Additional notes or details
Memory requirements
Windows containers run some system process in each container typically those which provide per-container functionality like user management, networking etc… and while much of the memory required by these processes is shared amongst containers the memory cap must be high enough to allow for them. A table is provided in the system requirements document for each base image type and with and without Hyper-V isolation.
CPU Shares (without Hyper-V isolation)
When using CPU shares the underlying implementation (when not using Hyper-V isolation) configures the JOBOBJECT_CPU_RATE_CONTROL_INFORMATION, specifically setting the control flag to JOB_OBJECT_CPU_RATE_CONTROL_WEIGHT_BASED and providing an appropriate Weight. The valid weight ranges of the job object are 1 – 9 with a default of 5 which is lower fidelity than the host compute services values of 1 – 10000. As examples a share weight of 7500 would result in a weight of 7 or a share weight of 2500 would result in a value of 2.