Trying to passthru my RTX 2080 ti from FreeBSD to a Windows 11 vm (it generates a resource conflict and requires additional installation)

Question

Hello to everyone.

What I would like to understand if why,when I try to passthru my RTX 2080 ti from FreeBSD to Windows 11,it won't do it,causing the error 12. It says that it generates a resource conflict and requires additional installation). I'm trying to debug the error. Below you see the IOMMU group of my Nvidia Geforce RTX 2080 ti passed in a Windows 11 / bhyve VM and of Windows 11 installed physically. I'm not able to understand if there are inconsistencies.

VM :

Group: 30.0.15.1123 - Device: NVIDIA GeForce RTX 2080 Ti (Device ID: PCI\VEN_10DE&DEV_1E04&SUBSYS_250319DA&REV_A1\3&61AAA01&0&48)

Group: 4.39.0.0 - Device: NVIDIA Virtual Audio Device (Wave Extensible) (WDM) (Device ID: ROOT\UNNAMED_DEVICE\0000)

Group: 1.46.831.832 - Device: NVIDIA USB Type-C Port Policy Controller (Device ID: PCI\VEN_10DE&DEV_1AD7&SUBSYS_250319DA&REV_A1\3&61AAA01&0&4B)

Group: 10.0.22000.2360 - Device: Controller host NVIDIA USB 3.10 eXtensible - 1.10 (Microsoft) (Device ID: PCI\VEN_10DE&DEV_1AD6&SUBSYS_250319DA&REV_A1\3&61AAA01&0&4A)

Physical installation :

Group: 31.0.15.5222 - Device: NVIDIA GeForce RTX 2080 Ti (Device ID: PCI\VEN_10DE&DEV_1E04&SUBSYS_250319DA&REV_A1\4&94764DC&0&0009)

Group: 4.49.0.0 - Device: NVIDIA Virtual Audio Device (Wave Extensible) (WDM) (Device ID: ROOT\UNNAMED_DEVICE\0000)

Group: 1.50.831.832 - Device: NVIDIA USB Type-C Port Policy Controller (Device ID: PCI\VEN_10DE&DEV_1AD7&SUBSYS_250319DA&REV_A1\4&94764DC&0&0309)

Group: 10.0.22000.2360 - Device: NVIDIA USB 3.10 eXtensible Host Controller - 1.10 (Microsoft) (Device ID: PCI\VEN_10DE&DEV_1AD6&SUBSYS_250319DA&REV_A1\4&94764DC&0&0209)

As an experiment I've passed only this device from FreeBSD to WIndows 11 :

pptdevs="2/0/0"

I have excluded "2/0/1 ; 2/0/2 ; 2/0/3". I want to show you what means these addresses :

02:00.0 VGA compatible controller: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] (rev a1)

02:00.1 Audio device: NVIDIA Corporation TU102 High Definition Audio Controller (rev a1)

02:00.2 USB controller: NVIDIA Corporation TU102 USB 3.1 Host Controller (rev a1)

02:00.3 Serial bus controller: NVIDIA Corporation TU102 USB Type-C UCSI Controller (rev a1)

What happened when I launched the vm using these parameters ?

bhyve -S -c sockets=2,cores=2,threads=2 -m 4G -w -H -A \

-s 0,hostbridge \

-s 1,ahci-hd,/mnt/$vmdisk0'p2'/bhyve/img/Windows/Windows11.img,bootindex=1 \

-s 9:0,passthru,2/0/0 \

-s 11,hda,play=/dev/dsp,rec=/dev/dsp \

-s 13,virtio-net,tap18 \

-s 29,fbuf,tcp=0.0.0.0:5918,w=1600,h=950,wait \

-s 30,xhci,tablet \

-s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \ vm0:18 < /dev/null & sleep 2 && vncviewer 0:18

this error :

Assertion failed: (!err), function hda_init, file /usr/corvin-src-releng-140/usr.sbin/bhyve/pci_hda.c, line 353.

The error is caused by this parameter :

-s 11,hda,play=/dev/dsp,rec=/dev/dsp \

I have excluded it and Windows 11 booted. Now,inside it I still see the error 12. It says : This device :

PCI\VEN_10DE&DEV_1E04&SUBSYS_250319DA&REV_A1\3&61aaa01&0&48

generates a resource conflict and requires additional installation.

If I pass to the vm only -s 9:0,passthru,2/0/0 \ ;

it does not work anyway,because AFAIK,MAYBE there is one and only IOMMU group,so I should pass everything ? But if I pass everything,the error 12 is there anyway. I would like to understand if there is a problem with the IOMMU thing,if there is maybe there is the needing to develop a patch like this one :

GitHub - benbaker76/linux-acs-override for FreeBSD.

For sure in the past experiments I have passed all the 4 devices in the group. Unfortunately I've got the same error. What I need to know is what can I do to know with which other device the VGA compatible controller conflicts. What's the procedure to understand what are the interrupts that conflict and to whom they belong.

For an experienced developer should not be so hard to understand which devices conflict. I want to ask some help to understand how to debug the error. If we will be able to understand which devices conflict,maybe I can explain this to the bhyve developer with which I'm in contact and he can develop a patch. So,we will be able to use every modern nvidia gpus inside a Windows VM with bhyve under FreeBSD ! That's a nice goal,isn't it ? So,please ask me more informations that you need to understand the origin of the error.

Answer 1

Hi Mario,

Hope you're doing well.

1. Verify that all devices you're trying to pass through to the VM are in the same IOMMU group. If they are not, you may need to adjust kernel parameters or BIOS settings to ensure proper grouping.
2. Check the kernel logs (dmesg output) for any relevant error messages or warnings related to IOMMU, PCIe, or device passthrough. This may provide clues about what is causing the resource conflict.
3. Try passing through individual devices one by one and observe if any specific device triggers the error. This can help identify which device is causing the conflict. Experiment with different combinations of devices to see if there's a particular combination that works without triggering error 12.
4. If your devices are not grouped together in the same IOMMU group, you may need to use the ACS (Access Control Services) override patch to isolate devices into separate groups. This can sometimes resolve resource conflicts.

Best Regards,

Ian Xue

---

If the Answer is helpful, please click "Accept Answer" and upvote it.