Windows Server 2016 NFS client exhibits inferior write performance compared to Linux

Question

In identical hardware environments (25GbE network, 32GB RAM, 32-core CPU) with RockyLinux 8.9 running NFSD as backend, FIO tests were conducted using 64 jobs with 1MB block size and direct I/O. While both Windows and Linux NFS clients achieved similar read throughput (~3GB/s), the Windows NFS client's write throughput (600MB/s) was significantly lower than its Linux counterpart (3GB/s).

Here are the FIO test scripts for Windows and Linux respectively:

[sec3_1M_write_iodepth1]

directory=Y:\192.168.27.93

size=128M

bs=1M

rw=write

rwmixread=70

iodepth=1

numjobs=64

thread=1

direct=1

overwrite=1

ioengine=windowsaio

time_based=1

runtime=180

norandommap=1

randrepeat=0

ramp_time=5

log_avg_msec=500

stonewall=1

group_reporting=1

cpus_allowed_policy=split

filename_format=$jobnum.$filenum

create_serialize=0

write_bw_log=C:\Users\Administrator\AppData\Local\fio/dev0_192.168.27.93/logs/3_1M_write_iodepth1_bw.log

write_lat_log=C:\Users\Administrator\AppData\Local\fio/dev0_192.168.27.93/logs/3_1M_write_iodepth1_lat.log

write_iops_log=C:\Users\Administrator\AppData\Local\fio/dev0_192.168.27.93/logs/3_1M_write_iodepth1_iops.log

[sec3_1M_write_iodepth1]

directory=/mnt/fio/192.168.27.93

size=128M

bs=1M

rw=write

rwmixread=70

iodepth=1

numjobs=64

thread=1

direct=1

overwrite=1

ioengine=psync

time_based=1

runtime=180

norandommap=1

randrepeat=0

ramp_time=5

log_avg_msec=500

stonewall=1

group_reporting=1

cpus_allowed_policy=split

filename_format=$jobnum.$filenum

create_serialize=0

write_bw_log=/tmp/fio/dev0_192.168.27.93/logs/3_1M_write_iodepth1_bw.log

write_lat_log=/tmp/fio/dev0_192.168.27.93/logs/3_1M_write_iodepth1_lat.log

write_iops_log=/tmp/fio/dev0_192.168.27.93/logs/3_1M_write_iodepth1_iops.log

Answer 1

Hello,

If the issue persists after these steps, it's advisable to reach out to both Microsoft and Olfeo support teams for further technical assistance.

The lower write performance of the Windows Server 2016 NFS client compared to the Linux client can be attributed to several key factors. Here's a detailed explanation:

1. NFS Protocol Implementation Differences:

Windows NFS client tends to perform synchronous writes, prioritizing data integrity at the cost of write throughput.

Linux NFS client supports asynchronous writes and benefits from deeper integration with the kernel, leading to better performance in high-concurrency scenarios.

Solution: Check Windows NFS client registry settings (HKLM\SOFTWARE\Microsoft\ClientForNFS) and enable asynchronous writes (e.g., EnableWriteBuffering).

1. IO Engine Configuration:

The Windows test script uses windowsaio as the IO engine, which may introduce higher latency in multi-threaded writes.

The Linux script uses psync, which is better suited for large, sequential writes.

Solution: On Windows, test with sync or psync IO engines to see if write performance improves.

1. Network Stack and Throughput Differences:

The Windows network stack can introduce additional overhead in high-throughput, high-concurrency scenarios.

Solution:

Enable TCP offloading on the network adapter.

Update network drivers and firmware.

Optimize network settings, including TCP window size and MTU, to fully utilize the 25GbE bandwidth.

1. Write Buffer Size:

Windows may use a smaller default write buffer, causing performance bottlenecks.

Solution: Increase the WriteBufferSize parameter in the Windows NFS client configuration.

1. Testing Environment and Filesystem Optimization:

Differences in test script configurations and underlying filesystem settings may impact performance.

Solution: Ensure consistent test script parameters and optimize the underlying filesystem on both Windows and Linux.

1. Operating System Limitations:

The NFS client implementation in Windows Server 2016 may have limitations affecting its write performance.

Solution: Consider upgrading to Windows Server 2022, which may include performance improvements for the NFS client.