GuestVm high disk response times compare to Hyper-V host 2019

Ivan Efremovski 1 Reputation point
2022-03-04T08:13:01.427+00:00

Hi,  Recently, we had a migration from our old Hyper-V Cluster WS 2012R2 to our new hosts (Dell PowerEdge R740) and the new 2019 Hyper-V cluster. We migrated the VMs with live compute+storage migration. Everything went without problems. After the migration, our customers started to report, that the VMs are very slow after the migration. Our initial analysis showed high disk response times (1000 - 2500ms) monitored from the VM guest. The host is not showing high response times on the CSV disks. All drivers, firmware, and windows updates are updated. We also raised the machine version level and integration services are up to date. The storage array is All-flash and connected through SAN.  We did a disk performance test with diskspd, between the host, and one virtual machine. All machine virtual disks are located on the same CSV disk. Here are the results:  diskspd -b64K -d30 -o4 -t8 -Su -r -w25 -L -Z1G -c20G  Host:  Results for timespan 1:  ------------------------------------------------------------------------  actual test time: 30.00s   thread count: 8   proc count: 48  ## CPU | Usage | User | Kernel | Idle  0| 24.11%| 0.83%| 23.28%| 75.89%   1| 22.60%| 0.63%| 21.98%| 77.40%   2| 22.50%| 0.68%| 21.82%| 77.50%   3| 22.71%| 0.89%| 21.82%| 77.29%   4| 22.40%| 0.68%| 21.72%| 77.60%   5| 24.43%| 0.78%| 23.65%| 75.57%   6| 24.17%| 1.20%| 22.97%| 75.83%   7| 22.19%| 0.68%| 21.51%| 77.81%   8| 0.83%| 0.21%| 0.63%| 99.17%   9| 0.26%| 0.21%| 0.05%| 99.74%   10| 0.31%| 0.05%| 0.26%| 99.69%   11| 0.31%| 0.00%| 0.31%| 99.69%   12| 0.16%| 0.10%| 0.05%| 99.84%   13| 0.26%| 0.16%| 0.10%| 99.74%   14| 0.73%| 0.10%| 0.63%| 99.27%   15| 0.68%| 0.42%| 0.26%| 99.32%   16| 1.25%| 0.05%| 1.20%| 98.75%   17| 0.05%| 0.00%| 0.05%| 99.95%   18| 0.99%| 0.16%| 0.83%| 99.01%   19| 0.10%| 0.00%| 0.10%| 99.90%   20| 1.30%| 0.21%| 1.09%| 98.70%   21| 0.83%| 0.00%| 0.83%| 99.17%   22| 0.78%| 0.16%| 0.63%| 99.22%   23| 0.99%| 0.21%| 0.78%| 99.01%   24| 31.04%| 0.73%| 30.31%| 68.96%   25| 0.47%| 0.21%| 0.26%| 99.53%   26| 7.60%| 0.68%| 6.93%| 92.40%   27| 1.09%| 0.26%| 0.83%| 98.91%   28| 4.11%| 0.52%| 3.59%| 95.89%   29| 0.42%| 0.26%| 0.16%| 99.58%   30| 4.06%| 0.52%| 3.54%| 95.94%   31| 0.42%| 0.16%| 0.26%| 99.58%   32| 1.35%| 0.31%| 1.04%| 98.65%   33| 0.73%| 0.16%| 0.57%| 99.27%   34| 3.39%| 0.57%| 2.81%| 96.61%   35| 2.81%| 0.21%| 2.60%| 97.19%   36| 1.46%| 0.57%| 0.89%| 98.54%   37| 0.73%| 0.05%| 0.68%| 99.27%   38| 0.89%| 0.57%| 0.31%| 99.11%   39| 1.20%| 0.78%| 0.42%| 98.80%   40| 3.80%| 0.16%| 3.65%| 96.20%   41| 1.72%| 0.73%| 0.99%| 98.28%   42| 6.51%| 0.42%| 6.09%| 93.49%   43| 0.26%| 0.05%| 0.21%| 99.74%   44| 20.73%| 0.16%| 20.57%| 79.27%   45| 2.81%| 0.31%| 2.50%| 97.19%   46| 7.92%| 2.03%| 5.89%| 92.08%  ## 47| 1.20%| 0.26%| 0.94%| 98.80%  avg.| 6.28%| 0.40%| 5.89%| 93.72%  Total IO  ## thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file  0 | 8321761280 | 126980 | 264.54 | 4232.63 | 0.944 | 0.650 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   1 | 8333426688 | 127158 | 264.91 | 4238.57 | 0.943 | 0.621 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   2 | 8330346496 | 127111 | 264.81 | 4237.00 | 0.943 | 0.626 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   3 | 8335917056 | 127196 | 264.99 | 4239.83 | 0.943 | 0.652 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   4 | 8300331008 | 126653 | 263.86 | 4221.73 | 0.947 | 0.690 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   5 | 8350793728 | 127423 | 265.46 | 4247.40 | 0.941 | 0.680 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   6 | 8323858432 | 127012 | 264.61 | 4233.70 | 0.944 | 0.647 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)  ## 7 | 8354398208 | 127478 | 265.58 | 4249.23 | 0.940 | 0.652 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)  total: 66650832896 | 1017011 | 2118.76 | 33900.10 | 0.943 | 0.653  Read IO  ## thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file  0 | 6231621632 | 95087 | 198.10 | 3169.54 | 0.905 | 0.672 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   1 | 6258753536 | 95501 | 198.96 | 3183.34 | 0.905 | 0.644 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   2 | 6234439680 | 95130 | 198.19 | 3170.98 | 0.904 | 0.642 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   3 | 6275465216 | 95756 | 199.49 | 3191.84 | 0.905 | 0.682 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   4 | 6228082688 | 95033 | 197.98 | 3167.74 | 0.908 | 0.716 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   5 | 6266224640 | 95615 | 199.20 | 3187.14 | 0.903 | 0.718 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   6 | 6245777408 | 95303 | 198.55 | 3176.74 | 0.903 | 0.661 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)  ## 7 | 6278217728 | 95798 | 199.58 | 3193.24 | 0.902 | 0.649 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)  total: 50018582528 | 763223 | 1590.04 | 25440.57 | 0.904 | 0.673  Write IO  ## thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file  0 | 2090139648 | 31893 | 66.44 | 1063.09 | 1.061 | 0.566 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   1 | 2074673152 | 31657 | 65.95 | 1055.23 | 1.056 | 0.531 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   2 | 2095906816 | 31981 | 66.63 | 1066.03 | 1.060 | 0.559 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   3 | 2060451840 | 31440 | 65.50 | 1047.99 | 1.058 | 0.536 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   4 | 2072248320 | 31620 | 65.87 | 1053.99 | 1.062 | 0.589 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   5 | 2084569088 | 31808 | 66.27 | 1060.26 | 1.056 | 0.536 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   6 | 2078081024 | 31709 | 66.06 | 1056.96 | 1.067 | 0.586 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)  ## 7 | 2076180480 | 31680 | 66.00 | 1055.99 | 1.056 | 0.651 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)  total: 16632250368 | 253788 | 528.72 | 8459.53 | 1.060 | 0.570  GuestVM (Windows server 2016):  Results for timespan 1:  ------------------------------------------------------------------------  actual test time: 30.00s   thread count: 8   proc count: 16  ## CPU | Usage | User | Kernel | Idle  0| 1.61%| 0.63%| 0.99%| 98.39%   1| 2.03%| 1.77%| 0.26%| 97.97%   2| 1.72%| 1.30%| 0.42%| 98.28%   3| 2.19%| 0.83%| 1.35%| 97.81%   4| 0.57%| 0.47%| 0.10%| 99.43%   5| 0.31%| 0.26%| 0.05%| 99.69%   6| 0.47%| 0.26%| 0.21%| 99.53%   7| 0.47%| 0.36%| 0.10%| 99.53%   8| 2.29%| 1.46%| 0.83%| 97.71%   9| 0.21%| 0.10%| 0.10%| 99.79%   10| 1.98%| 0.78%| 1.20%| 98.02%   11| 1.72%| 1.67%| 0.05%| 98.28%   12| 1.20%| 0.52%| 0.68%| 98.80%   13| 1.35%| 1.30%| 0.05%| 98.65%   14| 3.91%| 2.50%| 1.41%| 96.09%  ## 15| 0.16%| 0.00%| 0.16%| 99.84%  avg.| 1.39%| 0.89%| 0.50%| 98.61%  Total IO  ## thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file  0 | 11206656 | 171 | 0.36 | 5.70 | 688.358 | 945.686 | E:\iotest.dat (20GiB)   1 | 9764864 | 149 | 0.31 | 4.97 | 783.187 | 952.255 | E:\iotest.dat (20GiB)   2 | 9895936 | 151 | 0.31 | 5.03 | 772.906 | 921.165 | E:\iotest.dat (20GiB)   3 | 8519680 | 130 | 0.27 | 4.33 | 898.271 | 939.896 | E:\iotest.dat (20GiB)   4 | 11075584 | 169 | 0.35 | 5.63 | 699.540 | 896.609 | E:\iotest.dat (20GiB)   5 | 9830400 | 150 | 0.31 | 5.00 | 782.480 | 933.612 | E:\iotest.dat (20GiB)   6 | 10289152 | 157 | 0.33 | 5.23 | 738.098 | 929.830 | E:\iotest.dat (20GiB)  ## 7 | 11862016 | 181 | 0.38 | 6.03 | 630.648 | 852.503 | E:\iotest.dat (20GiB)  total: 82444288 | 1258 | 2.62 | 41.93 | 742.060 | 923.016  Read IO  ## thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file  0 | 7929856 | 121 | 0.25 | 4.03 | 389.060 | 723.862 | E:\iotest.dat (20GiB)   1 | 7208960 | 110 | 0.23 | 3.67 | 477.191 | 741.434 | E:\iotest.dat (20GiB)   2 | 7602176 | 116 | 0.24 | 3.87 | 579.250 | 849.748 | E:\iotest.dat (20GiB)   3 | 6225920 | 95 | 0.20 | 3.17 | 708.906 | 894.934 | E:\iotest.dat (20GiB)   4 | 8454144 | 129 | 0.27 | 4.30 | 486.887 | 771.755 | E:\iotest.dat (20GiB)   5 | 7340032 | 112 | 0.23 | 3.73 | 601.953 | 845.077 | E:\iotest.dat (20GiB)   6 | 7798784 | 119 | 0.25 | 3.97 | 600.108 | 840.095 | E:\iotest.dat (20GiB)  ## 7 | 9306112 | 142 | 0.30 | 4.73 | 537.521 | 846.196 | E:\iotest.dat (20GiB)  total: 61865984 | 944 | 1.97 | 31.47 | 542.452 | 819.194  Write IO  ## thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file  0 | 3276800 | 50 | 0.10 | 1.67 | 1412.658 | 1024.290 | E:\iotest.dat (20GiB)   1 | 2555904 | 39 | 0.08 | 1.30 | 1646.254 | 951.272 | E:\iotest.dat (20GiB)   2 | 2293760 | 35 | 0.07 | 1.17 | 1414.739 | 855.258 | E:\iotest.dat (20GiB)   3 | 2293760 | 35 | 0.07 | 1.17 | 1412.261 | 863.605 | E:\iotest.dat (20GiB)   4 | 2621440 | 40 | 0.08 | 1.33 | 1385.345 | 927.101 | E:\iotest.dat (20GiB)   5 | 2490368 | 38 | 0.08 | 1.27 | 1314.560 | 978.064 | E:\iotest.dat (20GiB)   6 | 2490368 | 38 | 0.08 | 1.27 | 1170.222 | 1056.216 | E:\iotest.dat (20GiB)  ## 7 | 2555904 | 39 | 0.08 | 1.30 | 969.727 | 786.905 | E:\iotest.dat (20GiB)  total: 20578304 | 314 | 0.65 | 10.47 | 1342.155 | 957.000  We cannot find the problem. Every comment, help, and solution is welcome.  BR,  Ivan 

Windows Server 2019
Windows Server 2019
A Microsoft server operating system that supports enterprise-level management updated to data storage.
3,507 questions
Hyper-V
Hyper-V
A Windows technology providing a hypervisor-based virtualization solution enabling customers to consolidate workloads onto a single server.
2,575 questions
Windows Server Storage
Windows Server Storage
Windows Server: A family of Microsoft server operating systems that support enterprise-level management, data storage, applications, and communications.Storage: The hardware and software system used to retain data for subsequent retrieval.
636 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. PMT 86 Reputation points
    2022-03-04T13:49:56.237+00:00

    Hello, @Ivan Efremovski

    Please observe storage performance during off hours and use robocopy to measure disk performance.

    --Thanks
    pm.tha.7@Stuff .com

    0 comments No comments