Hi, I'm developing a Windows PCIe GPU driver. The device exposes a MMIO BAR region. In kernel mode, the driver can access the BAR registers correctly.

For example:

• Register offset: 0x1f380
• Kernel-mode read value: 0x7666

However, when I map the BAR into a user-mode application and read the same register, the result is always:0xffffffff

What's interesting is:

• The exact same driver + user-mode application works correctly on a desktop PC
• The issue only happens on a laptop connected through an external PCIe expansion setup

---

Environment

Working system

Desktop PC:

• Same GPU device
• Same driver
• Same user-mode application
• User-mode MMIO works correctly
• But the GPU is installed directly in the motherboard's PCIe slot

Failing system

Laptop:

• External PCIe expansion setup connect to Laptop by Thunderbolt
• The Gpu is installed in the External PCIe expansion adapter.
• User-mode MMIO reads return 0xffffffff
• Kernel-mode MMIO still works correctly on the laptop.

---

Kernel-mode MMIO access

The driver maps BAR resources using normal MMIO access:

READ_REGISTER_ULONG(...)

Kernel-mode register reads work correctly on both systems.

---

User-mode BAR mapping method

The driver maps device BAR pages into user-mode using:

MmMapLockedPagesSpecifyCache(
    mdl,
    UserMode,
    MmNonCached,
    NULL,
    FALSE,
    NormalPagePriority
);

Important details:

• Cache type is MmNonCached
• The mapped pages are directly backed by PCIe BAR PFNs
• User-mode app accesses registers via volatile pointer dereference

Example:

volatile ULONG* reg =

    (volatile ULONG*)((char*)mapped_base + 0x1f380);

printf("%x\n", *reg);

Result on laptop:

ffffffff

---

Additional observations

Kernel-mode access still works

Even on the failing laptop system:

READ_REGISTER_ULONG(...)

returns the expected register value.Only user-mode mapped access fails.

---

Possible PCIe topology difference

The failing system is a laptop using an external PCIe setup connected to laptop Thunderbolt .

I'm wondering whether this could be related to:

• Thunderbolt PCIe tunneling
• IOMMU / Kernel DMA Protection
• PCIe bridge restrictions
• User-mode MMIO restrictions
• BAR forwarding behavior
• Prefetchable BAR behavior

---

Questions

1. Is directly mapping PCIe BAR PFNs into user-mode via MmMapLockedPagesSpecifyCache(UserMode) officially supported/reliable on modern Windows systems?
2. Are there known platform restrictions for user-mode MMIO on:
   • Thunderbolt
   • external PCIe bridges
   • laptops with Kernel DMA Protection enabled
3. Has anyone seen:
   • kernel MMIO working, but user-mode MMIO returning 0xffffffff
4. Is there a recommended alternative approach besides direct user-mode BAR mapping? (e.g. IOCTL-based register access)

---

Additional notes

• Kernel DMA Protection is disabled on the laptop, user-mode access still failed.
• Same BAR offset is being accessed in both kernel and user mode.
• Same driver binary and user application are used on both systems.

Any insight would be appreciated.

Hi @Erdong zhang ,

From the behavior you described, I would not treat this as a simple BAR offset or cache-type issue. Kernel-mode MMIO works on the same BAR and offset, while the user-mode mapping returns 0xFFFFFFFF only on the Thunderbolt / external PCIe setup. That tells me the issue is specific to the user-mode mapping path rather than the BAR resource itself.

Is directly mapping PCIe BAR PFNs into user-mode via MmMapLockedPagesSpecifyCache(UserMode) officially supported/reliable on modern Windows systems?

MmMapLockedPagesSpecifyCache(..., UserMode, ...) can create a user-mode mapping, but the documentation notes that for the AccessMode parameter, "Almost all drivers should use KernelMode." Microsoft's register access guidance also points toward keeping register access inside the driver rather than exposing mapped register space directly to user mode. I could not find Microsoft documentation presenting direct user-mode BAR mapping as a recommended general-access model for device registers, so even though it works on a desktop PCIe slot, I would not rely on it as a portable design.

Are there known platform restrictions for user-mode MMIO on Thunderbolt, external PCIe bridges, laptops with Kernel DMA Protection enabled?

Thunderbolt introduces a different PCIe topology involving tunneled PCIe, bridges, hotplug security, and potentially different IOMMU handling compared to a native motherboard PCIe slot. Windows does treat externally exposed PCIe hierarchies differently, which could explain why the user-mode mapping behaves differently on this topology. You also mentioned you already disabled Kernel DMA Protection and the problem persisted, which lowers confidence that Kernel DMA Protection alone is the cause but other firmware-level or IOMMU behavior may still be present underneath even when the Windows-level setting is off.

Has anyone seen kernel MMIO working, but user-mode MMIO returning 0xffffffff?

0xFFFFFFFF from an MMIO read is commonly associated with the transaction not completing successfully, for example, the read not reaching the device or not being forwarded correctly upstream. Since kernel-mode reads still work on the same offset, this makes a completely invalid BAR or nonfunctional device less likely and points to the two access paths not being equivalent on this platform.

Worth noting: your kernel path uses READ_REGISTER_ULONG(...), which includes memory barrier semantics. Your user-mode path uses a volatile pointer dereference, which is not semantically identical. That said, if this were purely a barrier issue I would expect stale or intermittent values, not a hard constant 0xFFFFFFFF so the more likely explanation is still the transaction not getting through on the user-mode path.

Is there a recommended alternative approach besides direct user-mode BAR mapping?

Yes, keep the BAR mapped in kernel mode and expose only the required operations through IOCTLs. The application sends an IOCTL with the register offset and operation type, the driver validates the offset, performs READ_REGISTER_ULONG / WRITE_REGISTER_ULONG, and returns the result. Microsoft's driver security best practices guidance treats unconstrained device memory access from user mode as unsafe and says the driver should validate and constrain the address and size.

I would suggest:

• Keep MMIO access in the kernel driver.
• Expose only the needed register operations through IOCTLs.
• Validate register offset and size in the driver.
• Avoid exposing the whole BAR directly to user mode.

This should be more reliable across different PCIe topologies and avoids depending on platform- and topology-specific behavior of direct user-mode BAR mappings.

Hope my explanation answers your question! If you find it helpful to you so far, I would greatly appreciate it if you could follow the instructions here so others with the same problem can benefit as well. Thank you.

Hi @Tom Tran (WICLOUD CORPORATION)

I found the reason after four day‘s debug. I read the bar register and get expected value in driver initialization flow, But I got 0xffffffff when read the bar register in driver EvtDeviceFileCreateprocess. Not only happened in user space. The reason is memory bar is disabled, the pcie command register is 0 which means memory bar is disabled. I try to reenable memory bar by write the command register to 6 in EvtDeviceFileCreateprocess, then my gpgpu can work normally.

I don’t know who disabled the gpgpu Memory BAR.

Hi @Erdong zhang ,

Thanks for sharing your finding. That explains the 0xFFFFFFFF result much more clearly.

If the PCI Command register is 0, then memory-space decoding for the BAR is disabled, so MMIO reads from that BAR can fail or return 0xFFFFFFFF. The fact that setting the Command register to 6 makes the device work again points to the BAR/resource itself being valid, but the device appears to be losing the required PCI command state somewhere after the earlier initialization path.

At this point, I would use that as a useful confirmation and check when the Command register changes. For example, you can log the Command register value in:

EvtDevicePrepareHardware
EvtDeviceD0Entry
EvtDeviceD0Exit
EvtDeviceFileCreate

If the value changes after a power transition, reset, or Thunderbolt/external PCIe hotplug-related state change, then the restore/reinitialization logic may fit better in the PnP/power path, especially EvtDeviceD0Entry, instead of only when the application opens the device. Microsoft’s EvtDeviceD0Entry documentation describes this callback as the place where a driver performs operations needed when the device enters the working D0 state: EVT_WDF_DEVICE_D0_ENTRY callback function.

So I think your finding identifies the immediate cause: the BAR read fails because PCI memory decoding is disabled at that point. The remaining part is to identify what clears the PCI Command register, and tracing it through the device start/power callbacks should help narrow that down.

1. Directly mapping PCIe BAR PFNs into user mode with MmMapLockedPagesSpecifyCache(..., UserMode, ...) is not a recommended or supported design pattern for a production Windows driver. It effectively gives a user‑mode process raw access to device (and potentially physical) memory, which is explicitly called out as unsafe and something that must be tightly constrained and validated. The driver security guidance shows that mapping device/physical memory into user mode must be limited to a very narrow, validated range to avoid abuse, and is presented as something to avoid or heavily constrain rather than a general mechanism for user‑mode MMIO. The example in the driver security best‑practices documentation demonstrates that arbitrary mapping of physical/device memory into user mode via MmMapIoSpace + MDL + MmMapLockedPagesSpecifyCache(..., UserMode, ...) is considered an “unsafe and improper method” and must be replaced with a constrained, validated mapping only for the exact expected address/size.
2. Platform restrictions on user‑mode MMIO are real and can differ between systems and topologies:
   • Windows places strict controls on how PCI configuration and device resources are accessed. For PCI devices, only the documented mechanisms (such as BUS_INTERFACE_STANDARD and IRP_MN_READ_CONFIG / IRP_MN_WRITE_CONFIG) are supported for configuration space access; other methods can be blocked and even cause a bug check on modern systems with virtualization‑based security and Secure Devices (SDEV) ACPI tables.
   • While the documentation in the provided context focuses on configuration space rather than BAR MMIO, it shows that Windows can and does enforce topology‑ and security‑dependent restrictions on how PCI resources are accessed, especially when virtualization‑based security or secure‑device paths are involved.
   • Thunderbolt, external PCIe bridges, and laptop platforms with IOMMU / kernel DMA protection are exactly the kinds of environments where firmware/ACPI and Windows may impose additional restrictions on how device memory is exposed to user mode, even when kernel‑mode MMIO continues to work.
   The behavior observed (kernel MMIO OK, user‑mode mapping reading 0xFFFFFFFF) is consistent with a platform/firmware or Windows security policy that allows the kernel to access the BAR but does not fully expose that BAR mapping to user‑mode page tables for that topology.
3. The pattern “kernel MMIO works, user‑mode MMIO returns 0xFFFFFFFF” is consistent with:
   • The device or bridge not forwarding the transaction from the user‑mode mapping (e.g., due to how the host bridge / Thunderbolt controller / IOMMU maps or filters that address range for user processes).
   • The mapping being treated as invalid or read‑as‑ones by the platform when accessed from user mode, even though the same physical BAR is reachable from kernel mode.
   The driver security guidance explicitly warns against exposing arbitrary physical/device memory to user mode and shows how such mappings must be constrained and validated. On some platforms (such as a desktop with a directly attached PCIe slot) the mapping may appear to work, while on more complex or security‑hardened topologies (Thunderbolt, external PCIe) the same pattern can fail.
4. Recommended alternative approach:
   • Instead of mapping BAR pages directly into user mode, use an IOCTL‑based interface where the kernel‑mode driver performs the MMIO on behalf of user mode. This keeps all device memory access in kernel mode, where Windows expects high‑privilege operations to occur, and avoids topology‑dependent user‑mode MMIO behavior.
   • If user‑mode access to device functionality is required, consider a user‑mode driver model (UMDF) where appropriate, but still avoid exposing raw BAR PFNs directly to arbitrary user processes.
   • If any mapping to user mode is absolutely necessary, follow the security guidance: validate the requested physical address and size against a strict expected range and refuse anything outside that range. This does not remove platform‑specific limitations, but it aligns with best practices and reduces the risk of the mapping being treated as unsafe.

Given the documented stance on mapping kernel/physical/device memory into user mode and the fact that PCI access methods are tightly controlled on modern Windows (especially with advanced security and ACPI/VBS features), the robust solution is to keep MMIO in kernel mode and expose a controlled IOCTL interface rather than relying on direct user‑mode BAR mappings, particularly on Thunderbolt/external PCIe laptop configurations.

---

References:

• Best practices for constraining high privileged behavior in kernel mode drivers
• Accessing PCI device configuration space