Windows Server 2019
A Microsoft server operating system that supports enterprise-level management updated to data storage.
3,900 questions
This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
Hi, Recently, we had a migration from our old Hyper-V Cluster WS 2012R2 to our new hosts (Dell PowerEdge R740) and the new 2019 Hyper-V cluster. We migrated the VMs with live compute+storage migration. Everything went without problems. After the migration, our customers started to report, that the VMs are very slow after the migration. Our initial analysis showed high disk response times (1000 - 2500ms) monitored from the VM guest. The host is not showing high response times on the CSV disks. All drivers, firmware, and windows updates are updated. We also raised the machine version level and integration services are up to date. The storage array is All-flash and connected through SAN. We did a disk performance test with diskspd, between the host, and one virtual machine. All machine virtual disks are located on the same CSV disk. Here are the results: diskspd -b64K -d30 -o4 -t8 -Su -r -w25 -L -Z1G -c20G Host: Results for timespan 1: ------------------------------------------------------------------------ actual test time: 30.00s thread count: 8 proc count: 48 ## CPU | Usage | User | Kernel | Idle 0| 24.11%| 0.83%| 23.28%| 75.89% 1| 22.60%| 0.63%| 21.98%| 77.40% 2| 22.50%| 0.68%| 21.82%| 77.50% 3| 22.71%| 0.89%| 21.82%| 77.29% 4| 22.40%| 0.68%| 21.72%| 77.60% 5| 24.43%| 0.78%| 23.65%| 75.57% 6| 24.17%| 1.20%| 22.97%| 75.83% 7| 22.19%| 0.68%| 21.51%| 77.81% 8| 0.83%| 0.21%| 0.63%| 99.17% 9| 0.26%| 0.21%| 0.05%| 99.74% 10| 0.31%| 0.05%| 0.26%| 99.69% 11| 0.31%| 0.00%| 0.31%| 99.69% 12| 0.16%| 0.10%| 0.05%| 99.84% 13| 0.26%| 0.16%| 0.10%| 99.74% 14| 0.73%| 0.10%| 0.63%| 99.27% 15| 0.68%| 0.42%| 0.26%| 99.32% 16| 1.25%| 0.05%| 1.20%| 98.75% 17| 0.05%| 0.00%| 0.05%| 99.95% 18| 0.99%| 0.16%| 0.83%| 99.01% 19| 0.10%| 0.00%| 0.10%| 99.90% 20| 1.30%| 0.21%| 1.09%| 98.70% 21| 0.83%| 0.00%| 0.83%| 99.17% 22| 0.78%| 0.16%| 0.63%| 99.22% 23| 0.99%| 0.21%| 0.78%| 99.01% 24| 31.04%| 0.73%| 30.31%| 68.96% 25| 0.47%| 0.21%| 0.26%| 99.53% 26| 7.60%| 0.68%| 6.93%| 92.40% 27| 1.09%| 0.26%| 0.83%| 98.91% 28| 4.11%| 0.52%| 3.59%| 95.89% 29| 0.42%| 0.26%| 0.16%| 99.58% 30| 4.06%| 0.52%| 3.54%| 95.94% 31| 0.42%| 0.16%| 0.26%| 99.58% 32| 1.35%| 0.31%| 1.04%| 98.65% 33| 0.73%| 0.16%| 0.57%| 99.27% 34| 3.39%| 0.57%| 2.81%| 96.61% 35| 2.81%| 0.21%| 2.60%| 97.19% 36| 1.46%| 0.57%| 0.89%| 98.54% 37| 0.73%| 0.05%| 0.68%| 99.27% 38| 0.89%| 0.57%| 0.31%| 99.11% 39| 1.20%| 0.78%| 0.42%| 98.80% 40| 3.80%| 0.16%| 3.65%| 96.20% 41| 1.72%| 0.73%| 0.99%| 98.28% 42| 6.51%| 0.42%| 6.09%| 93.49% 43| 0.26%| 0.05%| 0.21%| 99.74% 44| 20.73%| 0.16%| 20.57%| 79.27% 45| 2.81%| 0.31%| 2.50%| 97.19% 46| 7.92%| 2.03%| 5.89%| 92.08% ## 47| 1.20%| 0.26%| 0.94%| 98.80% avg.| 6.28%| 0.40%| 5.89%| 93.72% Total IO ## thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file 0 | 8321761280 | 126980 | 264.54 | 4232.63 | 0.944 | 0.650 | C:\ClusterStorage\Volume17\iotest.dat (20GiB) 1 | 8333426688 | 127158 | 264.91 | 4238.57 | 0.943 | 0.621 | C:\ClusterStorage\Volume17\iotest.dat (20GiB) 2 | 8330346496 | 127111 | 264.81 | 4237.00 | 0.943 | 0.626 | C:\ClusterStorage\Volume17\iotest.dat (20GiB) 3 | 8335917056 | 127196 | 264.99 | 4239.83 | 0.943 | 0.652 | C:\ClusterStorage\Volume17\iotest.dat (20GiB) 4 | 8300331008 | 126653 | 263.86 | 4221.73 | 0.947 | 0.690 | C:\ClusterStorage\Volume17\iotest.dat (20GiB) 5 | 8350793728 | 127423 | 265.46 | 4247.40 | 0.941 | 0.680 | C:\ClusterStorage\Volume17\iotest.dat (20GiB) 6 | 8323858432 | 127012 | 264.61 | 4233.70 | 0.944 | 0.647 | C:\ClusterStorage\Volume17\iotest.dat (20GiB) ## 7 | 8354398208 | 127478 | 265.58 | 4249.23 | 0.940 | 0.652 | C:\ClusterStorage\Volume17\iotest.dat (20GiB) total: 66650832896 | 1017011 | 2118.76 | 33900.10 | 0.943 | 0.653 Read IO ## thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file 0 | 6231621632 | 95087 | 198.10 | 3169.54 | 0.905 | 0.672 | C:\ClusterStorage\Volume17\iotest.dat (20GiB) 1 | 6258753536 | 95501 | 198.96 | 3183.34 | 0.905 | 0.644 | C:\ClusterStorage\Volume17\iotest.dat (20GiB) 2 | 6234439680 | 95130 | 198.19 | 3170.98 | 0.904 | 0.642 | C:\ClusterStorage\Volume17\iotest.dat (20GiB) 3 | 6275465216 | 95756 | 199.49 | 3191.84 | 0.905 | 0.682 | C:\ClusterStorage\Volume17\iotest.dat (20GiB) 4 | 6228082688 | 95033 | 197.98 | 3167.74 | 0.908 | 0.716 | C:\ClusterStorage\Volume17\iotest.dat (20GiB) 5 | 6266224640 | 95615 | 199.20 | 3187.14 | 0.903 | 0.718 | C:\ClusterStorage\Volume17\iotest.dat (20GiB) 6 | 6245777408 | 95303 | 198.55 | 3176.74 | 0.903 | 0.661 | C:\ClusterStorage\Volume17\iotest.dat (20GiB) ## 7 | 6278217728 | 95798 | 199.58 | 3193.24 | 0.902 | 0.649 | C:\ClusterStorage\Volume17\iotest.dat (20GiB) total: 50018582528 | 763223 | 1590.04 | 25440.57 | 0.904 | 0.673 Write IO ## thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file 0 | 2090139648 | 31893 | 66.44 | 1063.09 | 1.061 | 0.566 | C:\ClusterStorage\Volume17\iotest.dat (20GiB) 1 | 2074673152 | 31657 | 65.95 | 1055.23 | 1.056 | 0.531 | C:\ClusterStorage\Volume17\iotest.dat (20GiB) 2 | 2095906816 | 31981 | 66.63 | 1066.03 | 1.060 | 0.559 | C:\ClusterStorage\Volume17\iotest.dat (20GiB) 3 | 2060451840 | 31440 | 65.50 | 1047.99 | 1.058 | 0.536 | C:\ClusterStorage\Volume17\iotest.dat (20GiB) 4 | 2072248320 | 31620 | 65.87 | 1053.99 | 1.062 | 0.589 | C:\ClusterStorage\Volume17\iotest.dat (20GiB) 5 | 2084569088 | 31808 | 66.27 | 1060.26 | 1.056 | 0.536 | C:\ClusterStorage\Volume17\iotest.dat (20GiB) 6 | 2078081024 | 31709 | 66.06 | 1056.96 | 1.067 | 0.586 | C:\ClusterStorage\Volume17\iotest.dat (20GiB) ## 7 | 2076180480 | 31680 | 66.00 | 1055.99 | 1.056 | 0.651 | C:\ClusterStorage\Volume17\iotest.dat (20GiB) total: 16632250368 | 253788 | 528.72 | 8459.53 | 1.060 | 0.570 GuestVM (Windows server 2016): Results for timespan 1: ------------------------------------------------------------------------ actual test time: 30.00s thread count: 8 proc count: 16 ## CPU | Usage | User | Kernel | Idle 0| 1.61%| 0.63%| 0.99%| 98.39% 1| 2.03%| 1.77%| 0.26%| 97.97% 2| 1.72%| 1.30%| 0.42%| 98.28% 3| 2.19%| 0.83%| 1.35%| 97.81% 4| 0.57%| 0.47%| 0.10%| 99.43% 5| 0.31%| 0.26%| 0.05%| 99.69% 6| 0.47%| 0.26%| 0.21%| 99.53% 7| 0.47%| 0.36%| 0.10%| 99.53% 8| 2.29%| 1.46%| 0.83%| 97.71% 9| 0.21%| 0.10%| 0.10%| 99.79% 10| 1.98%| 0.78%| 1.20%| 98.02% 11| 1.72%| 1.67%| 0.05%| 98.28% 12| 1.20%| 0.52%| 0.68%| 98.80% 13| 1.35%| 1.30%| 0.05%| 98.65% 14| 3.91%| 2.50%| 1.41%| 96.09% ## 15| 0.16%| 0.00%| 0.16%| 99.84% avg.| 1.39%| 0.89%| 0.50%| 98.61% Total IO ## thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file 0 | 11206656 | 171 | 0.36 | 5.70 | 688.358 | 945.686 | E:\iotest.dat (20GiB) 1 | 9764864 | 149 | 0.31 | 4.97 | 783.187 | 952.255 | E:\iotest.dat (20GiB) 2 | 9895936 | 151 | 0.31 | 5.03 | 772.906 | 921.165 | E:\iotest.dat (20GiB) 3 | 8519680 | 130 | 0.27 | 4.33 | 898.271 | 939.896 | E:\iotest.dat (20GiB) 4 | 11075584 | 169 | 0.35 | 5.63 | 699.540 | 896.609 | E:\iotest.dat (20GiB) 5 | 9830400 | 150 | 0.31 | 5.00 | 782.480 | 933.612 | E:\iotest.dat (20GiB) 6 | 10289152 | 157 | 0.33 | 5.23 | 738.098 | 929.830 | E:\iotest.dat (20GiB) ## 7 | 11862016 | 181 | 0.38 | 6.03 | 630.648 | 852.503 | E:\iotest.dat (20GiB) total: 82444288 | 1258 | 2.62 | 41.93 | 742.060 | 923.016 Read IO ## thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file 0 | 7929856 | 121 | 0.25 | 4.03 | 389.060 | 723.862 | E:\iotest.dat (20GiB) 1 | 7208960 | 110 | 0.23 | 3.67 | 477.191 | 741.434 | E:\iotest.dat (20GiB) 2 | 7602176 | 116 | 0.24 | 3.87 | 579.250 | 849.748 | E:\iotest.dat (20GiB) 3 | 6225920 | 95 | 0.20 | 3.17 | 708.906 | 894.934 | E:\iotest.dat (20GiB) 4 | 8454144 | 129 | 0.27 | 4.30 | 486.887 | 771.755 | E:\iotest.dat (20GiB) 5 | 7340032 | 112 | 0.23 | 3.73 | 601.953 | 845.077 | E:\iotest.dat (20GiB) 6 | 7798784 | 119 | 0.25 | 3.97 | 600.108 | 840.095 | E:\iotest.dat (20GiB) ## 7 | 9306112 | 142 | 0.30 | 4.73 | 537.521 | 846.196 | E:\iotest.dat (20GiB) total: 61865984 | 944 | 1.97 | 31.47 | 542.452 | 819.194 Write IO ## thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | LatStdDev | file 0 | 3276800 | 50 | 0.10 | 1.67 | 1412.658 | 1024.290 | E:\iotest.dat (20GiB) 1 | 2555904 | 39 | 0.08 | 1.30 | 1646.254 | 951.272 | E:\iotest.dat (20GiB) 2 | 2293760 | 35 | 0.07 | 1.17 | 1414.739 | 855.258 | E:\iotest.dat (20GiB) 3 | 2293760 | 35 | 0.07 | 1.17 | 1412.261 | 863.605 | E:\iotest.dat (20GiB) 4 | 2621440 | 40 | 0.08 | 1.33 | 1385.345 | 927.101 | E:\iotest.dat (20GiB) 5 | 2490368 | 38 | 0.08 | 1.27 | 1314.560 | 978.064 | E:\iotest.dat (20GiB) 6 | 2490368 | 38 | 0.08 | 1.27 | 1170.222 | 1056.216 | E:\iotest.dat (20GiB) ## 7 | 2555904 | 39 | 0.08 | 1.30 | 969.727 | 786.905 | E:\iotest.dat (20GiB) total: 20578304 | 314 | 0.65 | 10.47 | 1342.155 | 957.000 We cannot find the problem. Every comment, help, and solution is welcome. BR, Ivan
Hello, @Ivan Efremovski
Please observe storage performance during off hours and use robocopy to measure disk performance.
--Thanks
pm.tha.7@Stuff .com