Problems with Hyper-V on Server 2019 (1809) after August 2020 patches

asked 2020-09-03T16:29:03.36+00:00
Cannon, Chris 121 Reputation points

We've been having problems after installing August 2020 Patches on our 2019 HyperV hosts. We have multiple hyper-v clusters across Dell VRTX and UCS blades w/ ISCSI backend SANS. Both environments have seen backup times double. Additionally loading a VMs settings in HyperV or Failover cluster manager is taking a very long time. We are not using a 3rd party AV but defender managed by SCCM. Usually when i see issues like this it feels like a storage performance issue but i'm seeing the issue across the board with iscsi as well as direct attached.

Patches applied
KB4566424
KB4565349
KB4569776

our change log indicates no other changes. Our hyperv hosts have no other roles.

I'm going to try roll back the august patching and do a quick A/B test to see if that remediates the issue, but thought i'd post to see if anyone else has seen this issue.

Chris

Windows Server 2019
Windows Server 2019
A Microsoft server operating system that supports enterprise-level management, data storage, applications, and communications.
2,178 questions
Hyper-V
Hyper-V
A Windows technology providing a hypervisor-based virtualization solution enabling customers to consolidate workloads onto a single server.
1,817 questions
{count} votes

33 answers

Sort by: Most helpful
  1. answered 2020-09-28T18:34:54.517+00:00
    Cannon, Chris 121 Reputation points

    I have an open ticket with Microsoft and i'm hoping to get some attention to the problem. Unfortunately, i seem to be the only person who has brought this up. Today, I did some testing beyond just removing and installing the patch. I believe the problem is tied to the VmDatastore.dll which was updated in August 2020.

    I did some Process Monitor research and found that duration for vmms.exe file reads of any virtual machines vmcx file is nearly double of what it was previously. If you look at the stack for those reads you can see VmDataStore.dll is prominent. While the duration of those reads are tiny fractions of a second, when you consider the loading of a single VM can lead to 40,000 file reads, it addes up. When scaled to a multi-node cluster with a lot of VMs, it becomes more noticeable.

    I found i can just roll back and then forward that single .dll and change the performance behavior. Powershell modules , Hyperv manager, Failover cluster manager and i suspect on host backups leverage vmms significantly. I've given MS all this info and hope it will get sent to someone who can do something about it. Maybe it will get some traction.

    Chris


  2. answered 2020-10-15T11:52:09.397+00:00
    Cannon, Chris 121 Reputation points

    I plan on updating once my ticket with MS is resolved/closed. Seems like they recognize the issue now and they're working through updating the file(s) in play. I've tested one set thusfar; I'm not sure if Alex has or not. The first set of files i received presented a different problem. There is a significant time zone difference between my tech and I so it can go a half day before i hear back.

    Chris


  3. answered 2020-09-08T13:37:03.403+00:00
    Cannon, Chris 121 Reputation points

    Quick update. After the removal of KB4565349 backup times returned to normal (With the patch backups went from 4-5 hours to 11-12 hours on the cluster i'm testing against) . I can also launch Vm settings within seconds now that the patch is gone. Are there known issues with this KB?

    https://borncity.com/win/2020/08/14/probleme-microsoft-august-2020-patchday-nachlese/ There is a comment that leads me to believe there are others that are having the issue as well. I don't see any new comments however.

    I'm seeing the issue across different hardware platforms on both datacenter and standard editions.

    Chris


  4. answered 2020-09-09T12:57:20.63+00:00
    chriZa 21 Reputation points

    Same problem here - since KB4565349 backup and replication tasks are very slow.
    Just installed KB4570333 on all hyper-v hosts but still backup times are doubled.
    Will this be fixed in an upcoming CU or do I have to uninstall the CUs KB4570333 and KB4565349 to get back to a normal backup behavior.

    Regards,
    chriZa

    No comments

  5. answered 2020-09-10T12:35:50.33+00:00
    Cannon, Chris 121 Reputation points

    I too have installed Septembers KB4570333 and have found the problems have returned; assuming the problem was rolled forward. Backups times are again more than double what they were previously. I am finding that when it comes to looking at the VM settings either through FCM or Hyper-v Manager, the slowness loading settings is more prevalent on my deeper clusters. Some of my larger clusters have 120-150 VMs.

    Our backups leverage on-host checkpoints so I suspect whatever problem is causing the settings and console connection slowness is also related to the backups. Smaller backups doubling in minutes isn't a huge big deal but when the big backups double, it causes a significant problem.

    Daniel, is there a way to file a bug report?

    Thanks

    Chris