Code 12: Driver Error in Windows Hyperv VM for NVIDIA GPUs

Question

I’m facing an issue with NVIDIA GPUs in my Windows virtual machine (VM) using Hyper-V with Discrete Device Assignment (DDA).

I’ve added two GPUs to my Hyper-V Windows VM, both visible in Device Manager under Display Adapters. However, one GPU is working perfectly, while the other is encountering a Code 12 error.

Here’s what I’ve tried:

1. GPU Assignment with DDA:
   • I successfully added two GPUs to my Hyper-V Windows VM using the Discrete Device Assignment (DDA) method.

       PS C:\Users\Administrator> Dismount-VMHostAssignableDevice -LocationPath "PCIROOT(16)#PCI(0200)#PCI(0000)" -force
       PS C:\Users\Administrator> Dismount-VMHostAssignableDevice -LocationPath "PCIROOT(C9)#PCI(0200)#PCI(0000)" -force
       PS C:\Users\Administrator>Add-VMAssignableDevice -LocationPath "PCIROOT(36)#PCI(0000)#PCI(0000)" -VMName Windows10
       PS C:\Users\Administrator>Add-VMAssignableDevice -LocationPath "PCIROOT(C9)#PCI(0200)#PCI(0000)" -VMName Windows10
     

2. NVIDIA Driver Installation:
   • After adding the GPUs, I installed the NVIDIA drivers on the VM.
3. Device Manager Status:
   • Both GPUs are visible in Device Manager under Display Adapters.
4. Code 12 Error:
   • One GPU is working fine.
   • The second GPU, however, displays a warning and is not functioning properly.
   • The error message states: “This device cannot find enough free resources that it can use. (Code 12) If you want to use this device, you will need to disable one of the other devices on this system.”

5. Disable/Enable Test:

• When I disable the GPU1 adapter, GPU2 works fine.
• Enabling GPU1 and disabling GPU2 and vice versa also confirms the individual functionality of each GPU.

Question:
How can I overcome the Code 12 error and utilize both NVIDIA GPUs simultaneously in my Hyper-V Windows VM?

Any insights, experiences, or tips from the community would be greatly appreciated.

Thanks in advance!

Answer 1

Hi Sam,

Hope you’re doing well.

This error is usually caused by other devices in the system taking up GPU resources. When two GPUs are enabled at the same time, they will try to use the same resources, causing conflicts. You can try the following methods to resolve this issue:

1. Use multi-GPU parallel training: If you are using a deep learning framework for training, you may consider using multi-GPU parallel training. This will allow you to use multiple GPUs simultaneously, thus speeding up training. Note that this requires your framework to support multi-GPU training.
2. Disable other devices: Follow the instructions in the error message to disable other devices in the system so that sufficient resources are available to the GPU. Please note that this may affect your system performance, so proceed with caution.
3. Update GPU driver: Make sure your GPU driver is up to date. You can download the latest drivers from the GPU manufacturer's website.

If none of the above resolves the issue, try contacting the GPU manufacturer's support team for further assistance.

Hope my answer can help you!

Best Regards

Answer 2

Did you end up fixing this issue? if so can you please share it?

Answer 3

Hi Mohammed,

Thanks for your reply.

Your issue does not occur in my environment, I just give solutions based on your situation.

Best Regards

Answer 4

Hi Xu Gu

Thanks for the reply and your suggestion.

The aforementioned issues were successfully resolved by configuring MMIO space, as outlined in the official Microsoft document: [Microsoft Official Document]

GPUs, particularly, require additional MMIO space for the VM to access the memory of that device. While each VM starts with 128 MB of low MMIO space and 512 MB of high MMIO space by default, certain devices or multiple devices may require more space, potentially exceeding these values.

Subsequently, I reconfigured the VM following the instructions in the Microsoft Official Document: [VM Preparation for Graphics Devices]

Solution :

• Enabled Write-Combining on the CPU using the cmdlet: Set-VM -GuestControlledCacheTypes $true -VMName VMName
• Configured the 32-bit MMIO space with the cmdlet: Set-VM -LowMemoryMappedIoSpace 3Gb -VMName VMName
• Configured greater than 32-bit MMIO space with the cmdlet: Set-VM -HighMemoryMappedIoSpace 33280Mb -VMName VMName

Dismount the GPUs

To dismount the GPU devices from the host:

• Located the device's location path
• Copied the device's location path
• Disabled the GPU in Device Manager

Dismounted the GPU devices from the host partition using the cmdlet:

Dismount-VMHostAssignableDevice -Force -LocationPath $locationPath1

Dismount-VMHostAssignableDevice -Force -LocationPath $locationPath2

Assigned the GPU devices to the VM using the cmdlet:

Add-VMAssignableDevice -LocationPath $locationPath1 -VMName VMName

Add-VMAssignableDevice -LocationPath $locationPath2 -VMName VMName

The configuration of the VM for DDA has been successfully completed.

Both GPUs are now recognized in my Linux Hyper-V VM:

root@DEB-HYPERV-6fabb3a422fb6e499b57dd2e11a7aa59:~# lspci
0000:00:00.0 Host bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge (AGP disabled) (rev 03)
0000:00:07.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 01)
0000:00:07.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)
0000:00:07.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 02)
0000:00:08.0 VGA compatible controller: Microsoft Corporation Hyper-V virtual VGA
0000:00:0a.0 Ethernet controller: Digital Equipment Corporation DECchip 21140 [FasterNet] (rev 20)
076a:00:00.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1)
e95f:00:00.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1)
root@DEB-HYPERV-6fabb3a422fb6e499b57dd2e11a7aa59:~#

Answer 5

Hi Mohammed Alkhatib1,

If you are facing the same issues, please refer to the solution that I have just posted in the above thread.