Packet Delay causing TCP retransmission on Hyper-V virtual machine

DiMi85 21 Reputation points
2020-08-25T09:18:29.35+00:00

Hello,
Sorry in advance if this getting a bit verbose but I was not able to shorten this up without dropping some maybe useful Information’s
I have a strange behaviour with Hyper-V Networking. It looks like a layer 2 problem but I am not entirely sure.
We have many programmable logic controller communicating over ADS Protocol (ADS is a proprietary transport protocol sitting on top of TCP from the PLC vendor) with a Windows Service running on a virtual machine. I am getting lots of connection Interrupts if the service runs on the virtual machine. If the service runs on a normal industrial PC everything is fine.
So I am building a Lab Setup to troubleshoot the problem. I scaled it down to one PLC and a NUC with Intel Atom barely able to running Windows 10 that I use as the server. To my surprise everything is working fine. But if I switch the NUC with a HP ProLiant DL360 Gen10 Server running Windows Server 2019 Stnd. With Hyper-V and a virtual machine also Windows Server 2019 (fully patched), the connection interrupts are back. After this I think it is not a performance issue.
This is my capture setup:
20089-ds-plc-kommunikation-messung.png
I Use a TAP to capture the traffic on the wire. To capture the traffic on the VM I use a second VM capturing the packets from a SPAN Port on the virtual switch.
Here is a sample of a connection timeout of 0.3 sec:
20147-grafik.png
The PLC is sending a burst of Data. The last packet of this burst is 3685 my TAP was not able to measure the time between these frame as you can see.
On the receiving side its look like this:
20109-grafik.png
The frames are coming in fast, but the last frame here 3689 has a 0.3 second delay. So all frames where on the same time on the wire according to the TAP capture. But the last frame show up late on the virtual switch. The PLC of course didn’t get an ACK for the package and sends it again (Frame 3690 on both captures)

If I disable VMQ everything runs fine. But I am not comfortable disabling a feature that should increase network performance. Also in most environments I have no control over the host server and telling the IT Admin to disable settings on a NIC I need good arguments to do so.

What is best practise in this case? Disable VMQ? Or am I missing something here?

Thank you for your help

Windows Server 2019
Windows Server 2019
A Microsoft server operating system that supports enterprise-level management updated to data storage.
3,457 questions
Hyper-V
Hyper-V
A Windows technology providing a hypervisor-based virtualization solution enabling customers to consolidate workloads onto a single server.
2,538 questions
Windows Server Infrastructure
Windows Server Infrastructure
Windows Server: A family of Microsoft server operating systems that support enterprise-level management, data storage, applications, and communications.Infrastructure: A Microsoft solution area focused on providing organizations with a cloud solution that supports their real-world needs and meets evolving regulatory requirements.
513 questions
0 comments
Accepted answer
  1. Candy Luo 12,656 Reputation points Microsoft Vendor
    2020-08-26T08:19:18.14+00:00

    Hi ,

    Thanks for your updating.

    Since you are using 1Gbit Network adapter, you don't need to enable VMQ or RSS on your NIC. Enable VMQ won't improve your network performance much when you're using a 1GB NIC.

    So just disable VMQ and as you found, everything will work fine.

    ---Please Accept as answer if the reply is helpful---

    Best Regards,

    Candy

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Candy Luo 12,656 Reputation points Microsoft Vendor
    2020-08-26T02:51:36.887+00:00

    Hi ,

    >>If I disable VMQ everything runs fine.

    Before going further, I would appreciate your help in clarifying the following situations:

    How did you disable VMQ? From host side or VM side?

    Did you enable VMQ on host's physical adapter ? If yes, how did you configure VMQ? With powershell command or in the Advanced tab of the network adapter's properties page?

    If you are using Enable-NetAdapterVmq command to enable use of VMQ on physical adapter, did you specify CPU range with -BaseProcessorNumber and -MaxProcessorNumber options?

    In addition, as far as I know, RSS should be disabled if VMQ is enabled. You might disable RSS (enable VMQ) and then see if everything works fine.

    Best Regards,

    Candy