Use GPUs with clustered VMs

Applies to: Windows Server 2025 (preview)

Important

GPU with clustered VMs in Windows Server 2025 is in PREVIEW. This information relates to a prerelease product that may be substantially modified before it's released. Microsoft makes no warranties, expressed or implied, with respect to the information provided here.

Applies to: Azure Stack HCI, versions 23H2 and 22H2

Note

The recommended way to create and manage VMs on Azure Stack HCI 23H2 is using the Azure Arc control plane. Use the mechanism described below to manage your VMs only if you need functionality that is not available in Azure Arc VMs.

You can include GPUs in your clusters to provide GPU acceleration to workloads running in clustered VMs. GPU acceleration is provided via Discrete Device Assignment (DDA), which allows you to dedicate one or more physical GPUs to a VM. Clustered VMs can take advantage of GPU acceleration, and clustering capabilities such as high availability via failover. Live migration of virtual machines (VMs) isn't currently supported, but VMs can be automatically restarted and placed where GPU resources are available if there's a failure.

In this article, learn how to use graphics processing units (GPUs) with clustered VMs to provide GPU acceleration to workloads in the clustered VMs. This article guides you through preparing the cluster, assigning a GPU to a cluster VM, and failing over that VM using Windows Admin Center and PowerShell.

Prerequisites

There are several requirements and things to consider before you begin to use GPUs with clustered VMs:

• You need an Azure Stack HCI cluster running Azure Stack HCI, version 21H2 or later.

• You need a Windows Server Failover cluster running Windows Server 2025 or later.

• You also need GPUs installed in every server of the cluster.

• Review and follow the instructions from your GPU manufacturer to install the necessary drivers and software on each server in the cluster.

• Depending on your hardware vendor, you might also need to configure any GPU licensing requirements.

• You need a machine with Windows Admin Center installed. This machine could be one of your cluster nodes.

• Create a VM to assign the GPU to. Prepare that VM for DDA by setting its cache behavior, stop action, and memory-mapped I/O (MMIO) properties according to the instructions in Deploy graphics devices using Discrete Device Assignment.

• Prepare the GPUs in each server by installing security mitigation drivers on each server, disabling the GPUs, and dismounting them from the host. To learn more about this process, see Deploy graphics devices by using Discrete Device Assignment.

• Create a VM to assign the GPU to. Prepare that VM for DDA by setting its cache behavior, stop action, and memory-mapped I/O (MMIO) properties according to the instructions in Deploy graphics devices using Discrete Device Assignment.

• Prepare the GPUs in each server by installing security mitigation drivers on each server, disabling the GPUs, and dismounting them from the host. To learn more about this process, see Deploy graphics devices by using Discrete Device Assignment.

Note

The Azure Stack HCI Catalog does not yet indicate GPU compatibility or certification information.

Prepare the cluster

When the prerequisites are complete, you can prepare the cluster to use GPUs with clustered VMs.

Preparing the cluster involves creating a resource pool that contains the GPUs that are available for assignment to VMs. The cluster uses this pool to determine VM placement for any started or moved VMs that are assigned to the GPU resource pool.

Windows Admin Center

Using Windows Admin Center, follow these steps to prepare the cluster to use GPUs with clustered VMs.

To prepare the cluster and assign a VM to a GPU resource pool:

1. Launch Windows Admin Center and make sure the GPUs extension is already installed.

2. Select Cluster Manager from the top dropdown menu and connect to your cluster.

3. From the Settings menu, select Extensions > GPUs.

4. On the Tools menu, under Extensions, select GPUs to open the tool.

   

5. On tool's main page, select the GPU pools tab, and then select Create GPU pool.

   

6. On the New GPU pool page, specify the following and then select Save:

   1. Server name
   2. GPU pool name
   3. GPUs that you want to add to the pool

   

   After the process completes, you'll receive a success prompt that shows the name of the new GPU pool and the host server.

PowerShell

Follow these steps to prepare the cluster to use GPUs with clustered VMs using PowerShell.

1. Create a new empty resource pool on each server containing the clustered GPU resources. Make sure to provide the same pool name on each server.

   In PowerShell, run the following cmdlet as an administrator:

    New-VMResourcePool -ResourcePoolType PciExpress -Name "GpuChildPool"
   

2. Add the dismounted GPUs from each server to the resource pool that you created in the previous step.

   In PowerShell, run the following commands:

    $gpu = Get-VMHostAssignableDevice
   
    Add-VMHostAssignableDevice -HostAssignableDevice $gpu -ResourcePoolName "GpuChildPool"
   

You now have a cluster-wide resource pool (named GpuChildPool) that is populated with assignable GPUs. The cluster uses this pool to determine VM placement for any started or moved VMs that are assigned to the GPU resource pool.

Assign a VM to a GPU resource pool

You can now assign a VM to a GPU resource pool. You can assign one or more VMs to a clustered GPU resource pool, and remove a VM from a clustered GPU resource pool.

Windows Admin Center

Follow these steps to assign an existing VM to a GPU resource pool using Windows Admin Center.

Note

You also need to install drivers from your GPU manufacturer inside the VM so that apps in the VM can take advantage of the GPU assigned to them.

1. On the Assign VM to GPU pool page, specify the following, then select Assign:

   1. Server name
   2. GPU pool name
   3. Virtual machine that you want to assign the GPU to from the GPU pool.

   You can also define advanced setting values for memory-mapped IO (MMIO) spaces to determine resource requirements for a single GPU.

   

   After the process completes, you'll receive a confirmation prompt that shows you successfully assigned the GPU from the GPU resource pool to the VM, which displays under Assigned VMs.

   

To unassign a VM from a GPU resource pool:

1. On the GPU pools tab, select the GPU that you want to unassign, and then select Unassign VM.

2. On the Unassign VM from GPU pool page, in the Virtual machines list box, specify the name of the VM, and then select Unassign.

   

   After the process completes, you receive a success prompt that the VM has been unassigned from the GPU pool, and under Assignment status the GPU shows Available (Not assigned).

PowerShell

Follow these steps to assign an existing VM to a GPU resource pool using PowerShell.

1. Configure the cluster VM resource’s default offline action as force-shutdown rather than save. Make sure to replace <vmname> with the name of the VM that you want to assign to the GPU resource pool.

   In PowerShell, run the following cmdlet:

    Get-ClusterResource -name <vmname> | Set-ClusterParameter -Name "OfflineAction" -Value 3
   

2. Assign the resource pool that you created earlier to the VM. Assigning the resource pool declares to the cluster that the VM requires an assigned device from the GpuChildPool pool when it's started or moved.

   In PowerShell, run the following cmdlet:

    Get-ClusterResource -name <vmname> | Add-VMAssignableDevice -ResourcePoolName "GpuChildPool"
   

   Note

   If you want to add more than one GPU to the VM, first verify that the resource pool has more than one assignable GPU available, and then run the previous command again.

You can also remove an assigned GPU from a VM. To do so, in PowerShell, run the following command. Make sure to replace <vmname> with the name of the VM that you want to assign to the GPU resource pool.

$vm = Get-VMAssignableDevice -VMName <vmname> | Where-Object { $_.ResourcePoolName -eq "GpuChildPool" }

$vm | Remove-VMAssignableDevice

When you start the VM, the cluster ensures that the VM is placed on a server with available GPU resources from this cluster-wide pool. The cluster also assigns the GPU to the VM through DDA, which allows the GPU to be accessed from workloads inside the VM.

Fail over a VM with an assigned GPU

To test the cluster’s ability to keep your GPU workload available, perform a drain operation on the server where the VM is running with an assigned GPU. To drain the server, follow the instructions in Failover cluster maintenance procedures. The cluster restarts the VM on another server in the cluster, as long as another server has sufficient available GPU resources in the pool that you created.

To test the cluster’s ability to keep your GPU workload available, perform a drain operation on the server where the VM is running with an assigned GPU. To drain the server, follow the instructions in Failover cluster maintenance procedures. The cluster restarts the VM on another server in the cluster, as long as another server has sufficient available GPU resources in the pool that you created.

Related content

For more information on using GPUs with your clustered VMs, see:

• Manage VMs with Windows Admin Center
• Plan for deploying devices by using Discrete Device Assignment

For more information on using GPUs with your VMs and GPU partitioning, see:

• Manage VMs with Windows Admin Center
• Plan for deploying devices by using Discrete Device Assignment