GuestVm high disk response times compare to Hyper-V host 2019

Question

Hi,  Recently, we had a migration from our old Hyper-V Cluster WS 2012R2 to our new hosts (Dell PowerEdge R740) and the new 2019 Hyper-V cluster. We migrated the VMs with live compute+storage migration. Everything went without problems. After the migration, our customers started to report, that the VMs are very slow after the migration. Our initial analysis showed high disk response times (1000 - 2500ms) monitored from the VM guest. The host is not showing high response times on the CSV disks. All drivers, firmware, and windows updates are updated. We also raised the machine version level and integration services are up to date. The storage array is All-flash and connected through SAN.  We did a disk performance test with diskspd, between the host, and one virtual machine. All machine virtual disks are located on the same CSV disk. Here are the results:  diskspd -b64K -d30 -o4 -t8 -Su -r -w25 -L -Z1G -c20G  Host:  Results for timespan 1:  ------------------------------------------------------------------------  actual test time: 30.00s   thread count: 8   proc count: 48  ## CPU | Usage | User | Kernel | Idle  0| 24.11%| 0.83%| 23.28%| 75.89%   1| 22.60%| 0.63%| 21.98%| 77.40%   2| 22.50%| 0.68%| 21.82%| 77.50%   3| 22.71%| 0.89%| 21.82%| 77.29%   4| 22.40%| 0.68%| 21.72%| 77.60%   5| 24.43%| 0.78%| 23.65%| 75.57%   6| 24.17%| 1.20%| 22.97%| 75.83%   7| 22.19%| 0.68%| 21.51%| 77.81%   8| 0.83%| 0.21%| 0.63%| 99.17%   9| 0.26%| 0.21%| 0.05%| 99.74%   10| 0.31%| 0.05%| 0.26%| 99.69%   11| 0.31%| 0.00%| 0.31%| 99.69%   12| 0.16%| 0.10%| 0.05%| 99.84%   13| 0.26%| 0.16%| 0.10%| 99.74%   14| 0.73%| 0.10%| 0.63%| 99.27%   15| 0.68%| 0.42%| 0.26%| 99.32%   16| 1.25%| 0.05%| 1.20%| 98.75%   17| 0.05%| 0.00%| 0.05%| 99.95%   18| 0.99%| 0.16%| 0.83%| 99.01%   19| 0.10%| 0.00%| 0.10%| 99.90%   20| 1.30%| 0.21%| 1.09%| 98.70%   21| 0.83%| 0.00%| 0.83%| 99.17%   22| 0.78%| 0.16%| 0.63%| 99.22%   23| 0.99%| 0.21%| 0.78%| 99.01%   24| 31.04%| 0.73%| 30.31%| 68.96%   25| 0.47%| 0.21%| 0.26%| 99.53%   26| 7.60%| 0.68%| 6.93%| 92.40%   27| 1.09%| 0.26%| 0.83%| 98.91%   28| 4.11%| 0.52%| 3.59%| 95.89%   29| 0.42%| 0.26%| 0.16%| 99.58%   30| 4.06%| 0.52%| 3.54%| 95.94%   31| 0.42%| 0.16%| 0.26%| 99.58%   32| 1.35%| 0.31%| 1.04%| 98.65%   33| 0.73%| 0.16%| 0.57%| 99.27%   34| 3.39%| 0.57%| 2.81%| 96.61%   35| 2.81%| 0.21%| 2.60%| 97.19%   36| 1.46%| 0.57%| 0.89%| 98.54%   37| 0.73%| 0.05%| 0.68%| 99.27%   38| 0.89%| 0.57%| 0.31%| 99.11%   39| 1.20%| 0.78%| 0.42%| 98.80%   40| 3.80%| 0.16%| 3.65%| 96.20%   41| 1.72%| 0.73%| 0.99%| 98.28%   42| 6.51%| 0.42%| 6.09%| 93.49%   43| 0.26%| 0.05%| 0.21%| 99.74%   44| 20.73%| 0.16%| 20.57%| 79.27%   45| 2.81%| 0.31%| 2.50%| 97.19%   46| 7.92%| 2.03%| 5.89%| 92.08%  ## 47| 1.20%| 0.26%| 0.94%| 98.80%  avg.| 6.28%| 0.40%| 5.89%| 93.72%  Total IO  ## thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file  0 | 8321761280 | 126980 | 264.54 | 4232.63 | 0.944 | 0.650 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   1 | 8333426688 | 127158 | 264.91 | 4238.57 | 0.943 | 0.621 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   2 | 8330346496 | 127111 | 264.81 | 4237.00 | 0.943 | 0.626 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   3 | 8335917056 | 127196 | 264.99 | 4239.83 | 0.943 | 0.652 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   4 | 8300331008 | 126653 | 263.86 | 4221.73 | 0.947 | 0.690 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   5 | 8350793728 | 127423 | 265.46 | 4247.40 | 0.941 | 0.680 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   6 | 8323858432 | 127012 | 264.61 | 4233.70 | 0.944 | 0.647 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)  ## 7 | 8354398208 | 127478 | 265.58 | 4249.23 | 0.940 | 0.652 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)  total: 66650832896 | 1017011 | 2118.76 | 33900.10 | 0.943 | 0.653  Read IO  ## thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file  0 | 6231621632 | 95087 | 198.10 | 3169.54 | 0.905 | 0.672 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   1 | 6258753536 | 95501 | 198.96 | 3183.34 | 0.905 | 0.644 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   2 | 6234439680 | 95130 | 198.19 | 3170.98 | 0.904 | 0.642 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   3 | 6275465216 | 95756 | 199.49 | 3191.84 | 0.905 | 0.682 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   4 | 6228082688 | 95033 | 197.98 | 3167.74 | 0.908 | 0.716 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   5 | 6266224640 | 95615 | 199.20 | 3187.14 | 0.903 | 0.718 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   6 | 6245777408 | 95303 | 198.55 | 3176.74 | 0.903 | 0.661 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)  ## 7 | 6278217728 | 95798 | 199.58 | 3193.24 | 0.902 | 0.649 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)  total: 50018582528 | 763223 | 1590.04 | 25440.57 | 0.904 | 0.673  Write IO  ## thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file  0 | 2090139648 | 31893 | 66.44 | 1063.09 | 1.061 | 0.566 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   1 | 2074673152 | 31657 | 65.95 | 1055.23 | 1.056 | 0.531 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   2 | 2095906816 | 31981 | 66.63 | 1066.03 | 1.060 | 0.559 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   3 | 2060451840 | 31440 | 65.50 | 1047.99 | 1.058 | 0.536 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   4 | 2072248320 | 31620 | 65.87 | 1053.99 | 1.062 | 0.589 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   5 | 2084569088 | 31808 | 66.27 | 1060.26 | 1.056 | 0.536 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)   6 | 2078081024 | 31709 | 66.06 | 1056.96 | 1.067 | 0.586 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)  ## 7 | 2076180480 | 31680 | 66.00 | 1055.99 | 1.056 | 0.651 | C:\ClusterStorage\Volume17\iotest.dat (20GiB)  total: 16632250368 | 253788 | 528.72 | 8459.53 | 1.060 | 0.570  GuestVM (Windows server 2016):  Results for timespan 1:  ------------------------------------------------------------------------  actual test time: 30.00s   thread count: 8   proc count: 16  ## CPU | Usage | User | Kernel | Idle  0| 1.61%| 0.63%| 0.99%| 98.39%   1| 2.03%| 1.77%| 0.26%| 97.97%   2| 1.72%| 1.30%| 0.42%| 98.28%   3| 2.19%| 0.83%| 1.35%| 97.81%   4| 0.57%| 0.47%| 0.10%| 99.43%   5| 0.31%| 0.26%| 0.05%| 99.69%   6| 0.47%| 0.26%| 0.21%| 99.53%   7| 0.47%| 0.36%| 0.10%| 99.53%   8| 2.29%| 1.46%| 0.83%| 97.71%   9| 0.21%| 0.10%| 0.10%| 99.79%   10| 1.98%| 0.78%| 1.20%| 98.02%   11| 1.72%| 1.67%| 0.05%| 98.28%   12| 1.20%| 0.52%| 0.68%| 98.80%   13| 1.35%| 1.30%| 0.05%| 98.65%   14| 3.91%| 2.50%| 1.41%| 96.09%  ## 15| 0.16%| 0.00%| 0.16%| 99.84%  avg.| 1.39%| 0.89%| 0.50%| 98.61%  Total IO  ## thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file  0 | 11206656 | 171 | 0.36 | 5.70 | 688.358 | 945.686 | E:\iotest.dat (20GiB)   1 | 9764864 | 149 | 0.31 | 4.97 | 783.187 | 952.255 | E:\iotest.dat (20GiB)   2 | 9895936 | 151 | 0.31 | 5.03 | 772.906 | 921.165 | E:\iotest.dat (20GiB)   3 | 8519680 | 130 | 0.27 | 4.33 | 898.271 | 939.896 | E:\iotest.dat (20GiB)   4 | 11075584 | 169 | 0.35 | 5.63 | 699.540 | 896.609 | E:\iotest.dat (20GiB)   5 | 9830400 | 150 | 0.31 | 5.00 | 782.480 | 933.612 | E:\iotest.dat (20GiB)   6 | 10289152 | 157 | 0.33 | 5.23 | 738.098 | 929.830 | E:\iotest.dat (20GiB)  ## 7 | 11862016 | 181 | 0.38 | 6.03 | 630.648 | 852.503 | E:\iotest.dat (20GiB)  total: 82444288 | 1258 | 2.62 | 41.93 | 742.060 | 923.016  Read IO  ## thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file  0 | 7929856 | 121 | 0.25 | 4.03 | 389.060 | 723.862 | E:\iotest.dat (20GiB)   1 | 7208960 | 110 | 0.23 | 3.67 | 477.191 | 741.434 | E:\iotest.dat (20GiB)   2 | 7602176 | 116 | 0.24 | 3.87 | 579.250 | 849.748 | E:\iotest.dat (20GiB)   3 | 6225920 | 95 | 0.20 | 3.17 | 708.906 | 894.934 | E:\iotest.dat (20GiB)   4 | 8454144 | 129 | 0.27 | 4.30 | 486.887 | 771.755 | E:\iotest.dat (20GiB)   5 | 7340032 | 112 | 0.23 | 3.73 | 601.953 | 845.077 | E:\iotest.dat (20GiB)   6 | 7798784 | 119 | 0.25 | 3.97 | 600.108 | 840.095 | E:\iotest.dat (20GiB)  ## 7 | 9306112 | 142 | 0.30 | 4.73 | 537.521 | 846.196 | E:\iotest.dat (20GiB)  total: 61865984 | 944 | 1.97 | 31.47 | 542.452 | 819.194  Write IO  ## thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file  0 | 3276800 | 50 | 0.10 | 1.67 | 1412.658 | 1024.290 | E:\iotest.dat (20GiB)   1 | 2555904 | 39 | 0.08 | 1.30 | 1646.254 | 951.272 | E:\iotest.dat (20GiB)   2 | 2293760 | 35 | 0.07 | 1.17 | 1414.739 | 855.258 | E:\iotest.dat (20GiB)   3 | 2293760 | 35 | 0.07 | 1.17 | 1412.261 | 863.605 | E:\iotest.dat (20GiB)   4 | 2621440 | 40 | 0.08 | 1.33 | 1385.345 | 927.101 | E:\iotest.dat (20GiB)   5 | 2490368 | 38 | 0.08 | 1.27 | 1314.560 | 978.064 | E:\iotest.dat (20GiB)   6 | 2490368 | 38 | 0.08 | 1.27 | 1170.222 | 1056.216 | E:\iotest.dat (20GiB)  ## 7 | 2555904 | 39 | 0.08 | 1.30 | 969.727 | 786.905 | E:\iotest.dat (20GiB)  total: 20578304 | 314 | 0.65 | 10.47 | 1342.155 | 957.000  We cannot find the problem. Every comment, help, and solution is welcome.  BR,  Ivan 

Answer 1

Hello, @Ivan Efremovski

Please observe storage performance during off hours and use robocopy to measure disk performance.

--Thanks
pm.tha.7@Stuff .com