Remote connections speed limited to about 1.5 MB / s

Question

I've been struggling in my company setup with a strange problem. After diagnosis, the problem seems to be caused by Windows 10.

My setup

• Windows 10 Pro machine serving as a file repository for workstations in LAN and for remote connections (let's call it a server).
• Asus ZPA-U8 server board, Xeon E5 20603, 24GB RAM, 500GB SSD for system + 2 x RAID 0 (2TB)
• Gigabit network with Netgear GS716T switch and Funbox 6 router
• Internet connection with public IP: Orange Fiber 1000 Mbps / 300 Mbps

Usage scenario

• We are a graphics studio, so the repository usually contains large graphic files (500MB +)
• Local workstations (also on Windows 10) connect to the file repository within LAN, where we easily achieve transfers of 100 MB / s (both directions).
• For remote employees, I have tested various access options, but on none of them I am able to go beyond the limit of about 1.5 MB / s (both directions). Even though, the Internet connection provides much higher speed.

What remote connection options have I tested?

• FTP installed as a Windows service using the Filezilla server
• HTTP installed as a Windows service using XAMPP
• Access to the local network via VPN Hamachi
• FTP server on Linux virtual machine (VirtualBox with Turnkey Linux)
• VPN on Linux virtual machine (VirtualBox with Turnkey Linux and OpenVPN)
• Owncloud server on Linux virtual machine (VirtualBox with ownCloud by Univention)

Regardless of the connection method (FTP, VPN, WebDAV, HTTP), I'm not able to go beyond the limit of about 1.5 MB / s. I tested on 3 different computers, and each of them had a much faster internet connection.

However, when I open two separate FTP connections, I can download two different files with speeds ranging from 1.2 - 1.5 MB / s (per connection). This may indicate that problem is somehow connected to TCP connection limit.

Previously, we had internet from a cable network (1000Mbps / 60Mbps), and there was the same problem with the speed limit.

I also tested all of those remote connection points from LAN, and I was getting much faster speeds (closer do LAN).

How did I diagnose that the restriction could be caused by Windows 10?

• I'm able to achieve faster speeds while connecting from the server to other services (via FTP, HTTP, WebDAV, VPN). The speed depends on the service/server I'm connecting to, but It can be 10x faster. When testing with my hosting company I can upload over 6 Mb/s.
• SpeedTest also shows much faster transfer rates.

I tried to look for answers in different forums and testing various Windows settings (including network card, protocols etc.), but with no luck.

Dose anyone have an idea what might be causing this problem?
I'm trying to figure this out before completely switching to Linux.

Answer 1

Thanks to help from @Gary Nebbett my issue was resolved.

I can archive faster transfers after enabling "Receive Window Auto-Tuning Level" both on server and client side.

It can be done by running this command in Command Line (as administrator)

netsh interface tcp set global autotuninglevel=normal  

Answer 2

Hello @Tomasz Maciąg ,

That looks like the older syntax (PktMon still seems to be a "work in progress" from Microsoft). This documentation seems to be more up-to-date - use whatever documentation best matches your version of Windows (the "pktmon help" command is perhaps the most reliable source of information)

I would suggest two changes:

1. Don't use a filter. The file transfer mechanism may well use TCP but if you are using a VPN then the TCP packets may be sent in IP packets with the ESP (Encapsulating Security Payload) protocol type. Let's see everything.
2. Don't specify "-p 0" which captures all of the data in each packet; I think that the default (without "-p 0") is to capture just the first 128 bytes) and that is normally enough for performance analysis, although 256 bytes probably makes some analysis easier (-p 256); the full packet is needed when one wants to investigate a problem with the implementation of an application layer protocol like SMB.

Don't worry too much about the first capture - once we have a rough idea of what might be happening we can refine the parameters and conditions of subsequent traces and tests.

Gary

Answer 3

@Gary Nebbett

I did two traces during downloading about 10MB file from serwer. First was done using FTP (filezilla server, client connected in passive mode over UDP). Second was done using HTTP over TCP (xampp apache server, client download from browser). I wasn't using VPN, but I had to use Remote Desktop to run those commands on serwer (I'm working remotely).

• trace-download-FTP-over-UDP.etl (deleted)
• trace-download-HTTP-over-TCP.etl (deleted)

Answer 4

Hello @Tomasz Maciąg ,

Thanks for that.

You should remove the downloads now. At least one of them contains "PI" (Personal Information).

There is much that is remarkable about the trace data. A graph of the throughput is amazingly linear/regular:

The average throughput for the two tests is also remarkably similar: 12.441 Mbps and 12.488 Mbps

It also looks as though some "Large Receive Offload" (LRO) setting is enabled in the network adapter. Most incoming IP packets appear in the trace data with a length of 0 but a payload of many thousand bytes. This means that the trace data is not reflecting the true activity "on the wire".

My suggestion now is to try the following:

• Just use the Microsoft-Windows-TCPIP provider, which does not capture any application data (no risk of revealing PI) but does reveal the inner thinking of the TCP stack.
• Repeat the two download tests and add an upload test. The trace of the "internal" thinking of a sender is much more revealing (how it measures network congestion and responds to it).

The command to start a trace is now:

logman start Gary -ets -p Microsoft-Windows-TCPIP -o why.etl

The command to stop the trace is:

logman stop Gary -ets

Gary

Update: I just realized that the trace was probably made on the "server", so that was a much more common Large Send Offload (LSO). There are still many oddities that need to be thought about, but I wanted to reply quickly because of the PI.