Massive Storage Bottleneck INSIDE Hyper-V VMs

Question

We run 2 Hyper-V Hosts (Server 2016) and the Storage INSIDE EVERY VirtualMachine behaves extremely slow.
The Storage itself is fast if used or benched directly on the Host. But as soon as you use a VM ontop of this Storage it slows down to a fraction INSIDE of this VM.

For Reference:
Speed of Storage measured an the parent partition: 18000 I/Os, 400 MB/s, 0,5ms Latency
Speed of Storage measured INSIDE a VM: 6 I/Os, 1-20 MB/s, 2400ms Latency, Storage-Queues bigger than 5 during these Tests

When do we notice this:
It got to our Attention that a new SQL-Server behaved slow. We are not sure if this is a new Problem or has been there since the two hosts are running (3 years) and we didnt notice because everything ran smooth
(DCs, RDP-Server, SQL, 3rd-Party Business Software...)
Then we started Perfmon.exe and some simple File-Copys (with Files larger then 1GB) as well as I/O-Test with 3rd-Party-SW (diskspd.exe)

Ressources:
CPU is nearly idle on HOST and VM during these Tests
Memory is more then enough free on HOST and VM
Storage-Queues bigger than 5 during these Tests - but only during these Tests.

What we checked so far:
No matter if the VM sits in a local attached HDD, SSD or a SAN, simple Disk, RAID1 or RAID10 - extremely slow storage. (ok, a SSD is faster than a HDD, but it also slows down to a fraction of its Speed in the parent partition)
No matter if you change the VM to the other Host
No matter if the Hyper-V Guest Extensions are activated or not
No matter if you switch off or even uninstall AV
No matter if we disable the SW-Firewall
No matter how much memory is assigned to the VMs (as already told the memory has been free to 80% on the VMs)
No Matter how many virtual CPUs are assignet to a VM
No matter if Pagefile is switched on or off inside the VM
No matter if dynamic or fixed VHDX are used for the VMs (marginal difference while benching)
No Matter if the VM is using "Windows 10", "Server 2016" or "Server 2019"
No Matter if we use a virgin, fresh installed VM with no extra SW but the Windows Updates
No Matter if the VM is member of a Domain or standalone
No Matter if Power-Management is disabled or not (in VM and Host)
No Matter if the Type of VM is "Gen1" or "Gen2"
No Matter if VM Ressource "Virtual Machine Reserve" is 0% or even 100%
No Matter if VM Ressource "Virtual Machine Limit(%)" is 100 or 1000%
No Matter if we disabled "Intel Virtualisation" (Xeon-CPUs) in BIOS of the Host
No Matter if we disabled "Intel VT-d" (Xeon-CPUs) in BIOS of the Host
No Matter if the VM is the only one which runs on the Host or if several others are running also
All hosts and VMs have the latest Updates...

We ran out of ideas where the source of this behaviour lies.

Any ideas or hints to narrow it down?

Serverwolf

Answer 1

Hello

Thank you for your question and reaching out.

I can understand you are facing issues with Storage performance in Hyper-V Environment.

I really appreciated that you have already gone thru some checks and posted the results.

1. First I will suggest you to check your NIC Teaming and Network card speed ( 1GB, 10GB ) as SAN \ iSCSI is purely depend on speed of your Network.
2. Please update your Hosts NIC card drivers from vendor website.
3. As you have mentioned you have already tried with Local Storage which is not part of the SAN \ iSCSI , So I believe that issue is within VM configuration or Applications installed on this VM.
4. Try create Pass-through Disk or LUN from storage and Map directly to this VM
   5 As you have also mentioned its slows in new SQL-Server :
   5.b Please check what is current Database size on this SQL server as SQL server is always high resources intensive.
   5.a Please try stop SQL services and any Business applications you have installed on VM and check storage performance.
   5.c Please also try measure storage performance using copy method as described in below article.

https://learn.microsoft.com/en-us/archive/blogs/josebda/using-file-copy-to-measure-storage-performance-why-its-not-a-good-idea-and-what-you-should-do-instead

https://learn.microsoft.com/en-us/windows-server/administration/performance-tuning/role/hyper-v-server/storage-io-performance

---

--If the reply is helpful, please Upvote and Accept as answer--

Answer 2

I'm having the same issue here and hope someone can help on that.

As mentioned the response time of the storage seen by the Hyper-v hosts is less then 4ms.

Storage is a 16Gb FC HPE MSA Storage.

The response time seen from inside the VM on the cluster is about 1-2 seconds.