Problems with Hyper-V on Server 2019 (1809) after August 2020 patches

Cannon, Chris 121 Reputation points
2020-09-03T16:29:03.36+00:00

We've been having problems after installing August 2020 Patches on our 2019 HyperV hosts. We have multiple hyper-v clusters across Dell VRTX and UCS blades w/ ISCSI backend SANS. Both environments have seen backup times double. Additionally loading a VMs settings in HyperV or Failover cluster manager is taking a very long time. We are not using a 3rd party AV but defender managed by SCCM. Usually when i see issues like this it feels like a storage performance issue but i'm seeing the issue across the board with iscsi as well as direct attached.

Patches applied
KB4566424
KB4565349
KB4569776

our change log indicates no other changes. Our hyperv hosts have no other roles.

I'm going to try roll back the august patching and do a quick A/B test to see if that remediates the issue, but thought i'd post to see if anyone else has seen this issue.

Chris

Windows Server 2019
Windows Server 2019
A Microsoft server operating system that supports enterprise-level management updated to data storage.
3,398 questions
Hyper-V
Hyper-V
A Windows technology providing a hypervisor-based virtualization solution enabling customers to consolidate workloads onto a single server.
2,506 questions
{count} votes

33 answers

Sort by: Most helpful
  1. Alessio P 66 Reputation points
    2020-10-07T12:01:17.553+00:00

    We've rolled back all nodes to June 16, 2020—KB4567513 (OS Build 17763.1294) and now backups and node management has come back to normal rate.
    We informed Microsoft about this and this is what they've replied :
    "The issue is under investigation by product group. We are having problems with our tools right now. I'll let you know when the Hotfix will be available, if we will have this information. I apologize for this inconvenience. We were not aware of it."
    Alex

    1 person found this answer helpful.

  2. Dawid 6 Reputation points
    2020-10-15T09:02:18.84+00:00

    Hi Guys,
    Same situation here. Backups was very slow. Failover cluster manager responsibility was very slow.
    In our case we have also performance issue with VMs on one of ours Hyper-V server.
    Only rollback helped us.

    Please let know in this thread when yours tickets was resolved.

    Dawid

    1 person found this answer helpful.
    0 comments No comments

  3. Jacob M - AIV 11 Reputation points
    2020-10-28T09:13:50.41+00:00

    hi Guys,

    We have been troubleshooting an Hyper-V server with the exact same issue as you guys described.
    an update (I believe the October or August update) caused the Hyper-V MMC to be extremely slow only showing 'connecting....' and stuck.

    Running a PowerShell command 'get-vm' took minuets to complete. the vmmworker services could not be restarted.
    Only killing the vmmworker.exe could stop it. we had 6 VM's in a saved-state and ware not able to recover from this scenario.

    finally we ended up reinstalling the Hyper-V server 2016 and manually imported the VM's again. This worked.

    this is the second time we had this issue with the same server. we reinstalled last week and now this update broke it again.

    Unfortunately a roll back of the update was not possible because a 'servicesing update' was installed and there was no uninstall option available any more...

    any news from you guys? this was the only post I was able to find with the same problem

    1 person found this answer helpful.

  4. Cannon, Chris 121 Reputation points
    2020-10-28T12:31:21.957+00:00

    I was given a hotfix last week which looked to be a build up from the July rollup. It did work but my understanding is that subsequent monthly rollups will overwrite it until it is released mainstream. I wasn't given an ETA for that. It did work in the cluster i tested it on. I didn't put it into our production environment just yet. i'll likely sit where i am at and update when its generally released in a monthly update.

    Yes my issue is with 2019. I'm not seeing the issue in 2016 but I only have 1 cluster left at 2016 and its fairly small.

    I wasn't getting anywhere until I started looking around myself; i have little patience for repeating tier1 actions over and over. I found process monitor and Process explorer useful tools in identifying the component that was operating slowly. One helps see what is in the stack and the other was helpful in seeing timing of certain actions and the files involved. I did the same actions and recorded it in the tools then reviewed them side by side. I ultimately found that vmms.exe's reads to config files were taking twice as long; it was a fraction of a second but when you have thousands of those reads, it adds up. It was also fairly easy to find b/c those reads made up the bulk of the registry monitor results. I found the dll using process explorer and noticed it was prominent in the stack for vmms.

    i've seen elsewhere comments similar to what Jacob was stating but i have not seen that specific problem.

    Chris

    1 person found this answer helpful.

  5. Tim Smals 6 Reputation points
    2020-11-05T09:23:00.787+00:00

    Could you maybe share the hotfix you received?

    We've got the same performance issues on our 8 node HCI-stack Server 2019 cluster. Lot of I/O troubles in Microsoft-Windows-Hyper-V-StorageVSP/Admin.
    Unfortunately I've installed Server 2019 with an ISO from august so we cannot uninstall the latest CU's (SW_DVD9_Win_Server_STD_CORE_2019_1809.6_64Bit_English_DC_STD_MLF_X22-34397) .
    Thanks!

    1 person found this answer helpful.