Performance Issues with os.scandir on Windows 11 Compared to Windows 10

Question

Why is the execution of os.scandir() in Python significantly slower on a Windows 11 system with a high-end configuration (i9 14900HX, 32GB DDR5, 4TB SSD) compared to a Windows 10 system (i7 11800H, 64GB DDR4, 4TB SSD)?

On the previous Windows 10 setup, using 16 threads to scan 900GB of files with os.scandir() took about 0.5 seconds. Now on Windows 11, using the same code, SSD, and 16 processes, the scanning duration has increased to several tens of minutes.

Additionally, building the flash-attn wheel with pip took about three hours on the Windows 10 system, maintaining full CPU utilization. However, on the new Windows 11 system, this process has already taken 8 hours with CPU utilization around 20%, and only two threads are being created.

What is causing these performance issues, and how can they be resolved? There seems to be a lack of solutions available online, and if this cannot be fixed, reverting to Windows 10 may be considered. The current system configuration includes an i9 14900HX, RTX 4070, 32GB DDR5, 4TB SSD, Windows version 23H2, build 22631.4460, Chinese Home Edition.

Answer 1

Hello,

Performance degradation like this isn’t typical and might be due to various factors that may be related to the operating system environment, configuration, or an underlying issue. Here are some potential causes and troubleshooting steps you can consider:

Windows Settings and Configuration:

Check for any power-saving settings on your Windows 11 system that could be limiting your CPU performance. Make sure you’re using a high-performance power plan. Try disabling or configuring Windows Defender or other antivirus software

Verify that you are using the same version of Python and all related packages as you were on Windows 10. Python itself or any of the packages could have received updates that impact performance differently on Windows 11.

You mentioned using a Home Edition and a specific build number. Check if you are on the latest version and whether there are any pending updates that might address performance issues in Windows 11. Sometimes, early builds of a new OS can have performance known issue that are fixed in later updates.

Hardware Drivers:

Ensure that all drivers, particularly the chipset and storage controller drivers, are up to date on your Windows 11 system. Manufacturers may release Windows 11-specific drivers that could optimize performance.

Performance Monitoring:

Use Windows Performance Monitor and Resource Monitor to check system resource utilization and what might be causing the bottleneck during your Python script execution.

Best Regards,

Hania Lian

============================================

If the Answer is helpful, please click "Accept Answer" and upvote it.