Problems with Hyper-V on Server 2019 (1809) after August 2020 patches

Cannon, Chris 121

We've been having problems after installing August 2020 Patches on our 2019 HyperV hosts. We have multiple hyper-v clusters across Dell VRTX and UCS blades w/ ISCSI backend SANS. Both environments have seen backup times double. Additionally loading a VMs settings in HyperV or Failover cluster manager is taking a very long time. We are not using a 3rd party AV but defender managed by SCCM. Usually when i see issues like this it feels like a storage performance issue but i'm seeing the issue across the board with iscsi as well as direct attached.

Patches applied
KB4566424
KB4565349
KB4569776

our change log indicates no other changes. Our hyperv hosts have no other roles.

I'm going to try roll back the august patching and do a quick A/B test to see if that remediates the issue, but thought i'd post to see if anyone else has seen this issue.

Chris

Daniel Zhou (Shanghai Wicresoft Co,.Ltd.) 256 Reputation points

2020-09-04T07:23:51.51+00:00

Hi，

Yes, you can roll back to the previous version to see if the question still exists. And you can update your result here.

And if anyone has the same issue, welcome to post your question here.

Best Regards,
Daniel
Cannon, Chris 121 Reputation points

2020-09-04T14:58:07.043+00:00

I removed only KB4565349 from a 5 node cluster (150 vms) on UCS blade chasssis and from a smaller 2 node cluster (dell vrtx) with around a dozen VMs. With the larger cluster, i was previously waiting on the window to open for a VM settings from FCM was taking between 20-40 seconds. I'm loading them now between 3 - 5. I do not yet have statistics on backups; I had to pause backups on the larger cluster b/c the job was running past the maintenance window. I'm re-enabling that back up now. With the smaller cluster, i made the change after backups had completed so i don't have a comparison.

With it being a longer holiday weekend i'll likely let things sit as they are over the weekend. If backups seem back to normal, i'm open to do some more testing with a couple other clusters (brand new UCS blades). To my knowledge, we're not seeing any VM performance issues at this point.

Thanks Chris
Daniel Zhou (Shanghai Wicresoft Co,.Ltd.) 256 Reputation points

2020-09-07T01:52:19.217+00:00

Sounds good.

Please let me know if you have any other questions.
Paul Webb 26 Reputation points

2020-11-12T06:30:17.36+00:00

We are also seeing delays on our 2019 Cluster. This manifests in slow "Get-VM" commands on the nodes and slowness opening settings for any of the VMs. It also seems to be causing delayed stats coming into Veeam One - sometimes these can be 15 minutes behind.

I don't like the idea of uninstalling updates to fix the issue - essentially if Microsoft is aware of the issue, they need to fix it! It could also affect anyone that has ISO compliance requiring fixes to be applied within a particular time window.

Does anyone have a link or reference to the Hotfix (Hotfix ID?) which solves the issue? I don't mind opening a case with Microsoft, if I need to go down that route to receive the fix, but it will be quicker if I can point them in the right direction.
Alessio P 66 Reputation points

2020-11-12T06:35:09.667+00:00

I don't think there is a public hotfix yet, only a private hotfix created ad hoc. You'll have to wait for a general release.
Alex
Cannon, Chris 121 Reputation points

2020-11-12T11:55:23.927+00:00

I'm not sure i feel comfortable sharing the hotfix MS provided me as i'm not sure it was anywhere close to a finished product. I should add that I am not comfortable putting this on any of our production 2019 Hyper V servers.; I'm waiting on the general release.
Paul Webb 26 Reputation points

2020-11-13T10:09:23.627+00:00

Hi Chris
I just looked at all the hotfixes and updates. Do you know which one you uninstalled on your test environment to get things back to speed?

Source Description HotFixID InstalledBy InstalledOn

NODE Update KB4580422 24/10/2020 00:00:00
NODE Security Update KB4470788 12/03/2019 00:00:00
NODE Update KB4494174 11/05/2020 00:00:00
NODE Security Update KB4549947 NT AUTHORITY\SYSTEM 11/05/2020 00:00:00
NODE Security Update KB4570332 NT AUTHORITY\SYSTEM 24/10/2020 00:00:00
NODE Security Update KB4577667 NT AUTHORITY\SYSTEM 24/10/2020 00:00:00
NODE Security Update KB4577668 NT AUTHORITY\SYSTEM 24/10/2020 00:00:00

Cheers
Paul Webb 26 Reputation points

2020-11-13T10:27:03.567+00:00

Notice there is a new CU released ....
2020-11 Cumulative Update for Windows Server 2019 (1809) for x64-based Systems (KB4586793) released Nov 11.
Tayschrenn 6 Reputation points

2020-11-23T15:11:51.463+00:00

This is certainly happening on clusters we run. We have not yet rolled back (more complicated than removing a single update) and inherently puts our environment in risk by running on an old rollup if we do.

Very frustrating to not have Microsoft already patched this, so we'll likely burn a support case to get a hotfix.

Can you provide any estimate on resolution or acknowledgment that Microsoft has even identified this?
Cannon, Chris 121 Reputation points

2020-11-23T15:41:20.103+00:00

Its been a month since i've heard from them. I pinged them a moment ago to see if there is a schedule for it to be included in the normal rollup. Believe me... I share your frustration. It took several calls/screen shares with multiple teams to get anyone to really acknowledge the issue. I'm still waiting on it to be rolled into the monthly cumulative. I'll respond if i hear back.

Chris
Cannon, Chris 121 Reputation points

2020-11-23T15:43:59.977+00:00

I just heard back. The update is scheduled for inclusion in January.
Tayschrenn 6 Reputation points

2020-11-23T16:19:23.44+00:00

I appreciate you taking the time to update the thread - hopefully it helps others as well.
Daniel Walker 1 Reputation point

2020-11-24T21:08:22.617+00:00

Any chance you can share the hotfix ID (even if you aren't sharing the hotfix) so we can request it via our cases?
Cannon, Chris 121 Reputation points

2020-11-24T21:25:35.513+00:00

The exe has KB900644.
Brendon Holt 16 Reputation points

2020-11-30T10:51:24.997+00:00

Add us to the mix, this weekend (Thanksgiving) we decided to update the Hardware Platform. Installation of Server 2019 went fine and updates were applied, up until November.

We experienced VERY SLOW Hyper-V Manager to Create or to Manage Settings on the Two New Super Powerful Computers just installed. Testing showed that during Hyper-V Manager could slow Guests down so much login to RDS servers was slow/failed. Was just taking forever to work in settings. Sunday we re-installed 2019 Server on one Machine and it was fine until updates were applied.

We decided to apply updates and deal with this issue because we provide this platform to end users, and the Legal/Financial Liability from Zero Net Logon outweighs the speed of IT Management. NOTE WE CANNOT WORK IN HYPER-V MANAGER IN PRODUCTION HOURS, 11PM to 2AM.

This BUG NEEDS TO BE FIXED and it is a SEVERE LIMITATION. I understand we all work in a complex and dangerous world, but hopefully we get a Fix SOON for this issue.
Jens Klarskov Jensen 1 Reputation point

2021-01-14T11:20:42.117+00:00
On a 6 Nodes Win2019 SCVMM managed FailOverCluster, that we use for Automated Testing , which heavily relies on CheckPoints we have these findings:

Win2019/SCVMM2019 has double duration, when operations are performed on Non SCV owner, whereas Win2016/SCVMM2016 has no measureable difference.

Win2019+KB4586793 doubles Hyper-V Restore on Non owner. SCVMM Restore is four times slowere, mostly affected by prolonged Refresh durations

Win2019+KB4598230 adds 30% to Hyper-V Restore. Doubles SCVMM Restore duration,

Eventually MigrateVm AWAY from SCV Owner ends up lasting 6 – 14 minutes

Measurements when Restoring a complex CheckPoint (Numbers are seconds):

So January Update was even worse!
Jens Klarskov Jensen 1 Reputation point

2021-01-14T11:28:34.283+00:00

BTW: Due to lots of SCVMM Error ID 2606 From the superslow Refresh(*), we have added 'Retry' SCVMM Command' for ERRORID:2606 to our Test Automation Execution , but are still severely down on Test Thrioughput.. We are still on KB4586793!

Unable to acquire a 'Delete' lock on object '52112b4e-3c4c-42b0-a2d9-0e708231d1c5' of type 'VirtualHardDisk' because it is locked by task 'ec55c407-f8dc-4900-9f3a-ce570d4a5d02' 'Refresh host cluster' with a 'Write' lock. (Error ID: 2606)
Jens Klarskov Jensen 1 Reputation point

2021-01-14T14:37:52.13+00:00

For completeness, we changed VmDatStore.dll back to the latest available in WinSxS before August 2020, an got these numbers - so no relief compared to August, but sime relief compared to the Jan 2021 results (We have only tried this on a Staging environment!)

.
Mikkel Lund Knudsen 106 Reputation points

2021-01-15T09:49:37.533+00:00

Hello Jens.
Have you been in talk with Microsoft regarding this issue? :)
DaxtechIT 6 Reputation points

2021-01-17T19:59:04.62+00:00

We opened a ticket with MS and referenced this post and the case# CannonChris-5226 provided. We were provided a private hotfix that is time limited which they stated they have had for a few months. It was time stamped in October 2020. Installing this hotfix requires using bcdedit to put and leave the server in test mode until the patch is removed and replaced with the public version. They also make accept a bunch of warnings that the hotfix is provided as-in and may cause other issues.

Given the impact on our environment and that there were no issues encountered in our test environment we have put this into production and it immediately resolved the IO performance issue. It did not solve the issue with VM settings taking a long time to load or the right click menu in cluster manager disappearing when you try to select something.
Jens Klarskov Jensen 1 Reputation point

2021-01-19T16:30:09.727+00:00

Yes we have a ticket and our 24/7 Team received HF KB900264 to test, but it is too old.

Here are Results with Nodes rolled back from Jan 2021 CU/Nov 2020 CU and with July 2020 CU:

To Compare with:
Jens Klarskov Jensen 1 Reputation point

2021-01-19T16:35:22.277+00:00

BTW, Numbers are in Seconds and average of three on an almost idle Two node Staging/Test Cluster with Lenovo SR650 + SSD SAN with duplicated Fibers.
Cannon, Chris 121 Reputation points

2021-02-26T21:18:42.927+00:00

Quick Followup. I'm hearing this fix is set to be released in March 2021. Why the delay?.. The fix was expanded to include additional issues and got pushed until the adjacent issues got looped in. I'm hopeful again

Chris
Smith, Trevor 1 Reputation point

2021-03-30T17:28:47.787+00:00

Does anyone know if the March 2021 updates resolved the slowness issue?
Smith, Trevor 1 Reputation point

2021-03-30T17:29:03.037+00:00

Does anyone know if the March 2021 updates resolved the slowness issue?
Brendon Holt 16 Reputation points

2021-05-26T02:56:22.203+00:00

I have read this and we decided to make the update to 2021-05 Cumulative Update. Our VEEAM backups are now completed without issue at normal speeds and after a lag the first time going into settings on Hyper-V we can now open all settings quickly. 40 Seconds to 1 Minute First Time, then 1 Second after that.

I believe they have fixed this issue, at least for my environment.

Thanks to all for your persistence on this issue.

33 answers

Paul Webb 26 Reputation points

2021-03-10T07:48:49.107+00:00

Looks like we are out of luck. I don't see anything but a Microcode update. Does anyone still have a support channel open with Microsoft to find out if they ever plan to fix?
Please sign in to rate this answer.

1 person found this answer helpful.
OneNerdyGuy 1 Reputation point

2021-03-10T13:50:07.383+00:00

What version of the DLL are you seeing on your updated hypervisors?

Manuel Cramer 1 Reputation point

2021-03-10T18:40:46.687+00:00

Just installed Hyper-V Core 2019 on a machine and updated to the latest patches including KB5000822.

Here I got version 10.0.17763.1697 of the VMDataStore.dll, signed at 2021-01-07.

OneNerdyGuy 1 Reputation point

2021-03-10T19:15:45.717+00:00

Do you see the performance impact like before, or is it resolved?

Cannon, Chris 121 Reputation points

2021-03-10T19:19:09.973+00:00

There was an update to it a month or so ago, but believe the performance impact is still present. MS told me a few hours ago that it is scheduled for release, 3rd week in March. I wasn't given an explanation.
Sign in to comment
Cannon, Chris 121 Reputation points

2020-11-12T15:31:43.047+00:00

kb4586793 file list includes both Vmdatastore.dll and Vmwp.exe. Going to check if the file revs match up to what was in the hotfix now.

Chris
Please sign in to rate this answer.
Cannon, Chris 121 Reputation points

2020-11-12T15:42:36.917+00:00

Nope. the file list must include the previous files b/c of the rollup status. HATE that they moved to only rollups...what a pita. The fix doesn't seem to be in Nov 2020.
Sign in to comment
Hyper-Z 1 Reputation point

2020-11-29T10:50:34.213+00:00

Similar to Cannon, We too are running Cisco UCS blades on iSCSI storage and experience much worse behavior than just slow performance.

If you are running Server 2019 1809 Hyper-V in a failover cluster and you apply any monthly update listed below from August till current, your Hyper-V servers may become slower or worse completely freak out and stop responding to simple management requests.

The RHS will fail randomly on all hosts. This will cause your VMs to reboot and attempt to run on another physical server as it will assume the server is unavailable.
"The cluster Resource Hosting Subsystem (RHS) process was terminated and will be restarted. This is typically associated with cluster health detection and recovery of a resource. Refer to the System event log to determine which resource and resource DLL is causing the issue."

August 2020 Update KB4565349
September 2020 Update KB4570333
October 2020 Update KB4577668
November 2020 Update KB4586793

Uninstall these updates as a workaround until Microsoft resolve this issue.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment
Jacques Magne 6 Reputation points

2020-11-29T16:58:47.497+00:00

Be careful, KB4565349 fix zero netlogon vulnerability (CVE-2020-1472)

Sorry for my poor English

Jacques
Please sign in to rate this answer.

0 comments No comments
Sign in to comment
Hans van Deursen 1 Reputation point

2021-01-12T10:53:50.9+00:00

Hi All,

Had the same issue with new installed 2019 servers and SAS storage. Then updated with KB4592440 and the problem was solved. Both Live Migrating and Quick migration works fine now.
Please sign in to rate this answer.
Josh Hargense 1 Reputation point

2021-01-12T20:51:53.74+00:00

Hi, the KB4592440 Update doesn't appear to fix the issue with performance (in relation to Vmdatastore.dll and Vmwp.exe). Running 'Get-VM -name *' on a host can still take up to 3 to 4 minutes.

Hanging out for the January CU to be released! This is causing a major heading in our datacentre.

Menk Wolfgang 1 Reputation point

2021-01-13T08:46:12.927+00:00

Hi, just got info from our MS support tech that the public hotfix is now scheduled for the 2021.02 C update. Not quite January either... Alas.

J. Vestergaard 1 Reputation point

2021-01-13T13:48:33.21+00:00

Hi - is there any way to get the hotfix before the public release?

Daniel Walker 1 Reputation point

2021-01-13T13:58:59.523+00:00

You don't want it unless they've changed it from one you need to disable SecureBoot for.

Alessio P 66 Reputation points

2021-01-13T14:05:03.703+00:00

You could replace Vmdatastore.dll taken from an unpatched W2019... as stated from CannonChris somewhere in this post...
"... CannonChris-5226 · Oct 02 at 3:02 PM
I have reviewed 2 days of backups after Installing Sept 2020 rollup AND rollback of only VmDataStore.dll. Backup times have gone back to normal. ..."

Cannon, Chris 121 Reputation points

2021-01-13T17:55:15.057+00:00

Total bummer that it wasn't included this month. That is really irritating. I haven't deployed in our testing env but hope to do so tonight. AFterwards i'll folllow up on my ticket and see what they tell me.

Chris

Jens Klarskov Jensen 1 Reputation point

2021-01-14T14:40:27.703+00:00

For completeness, we changed VmDatStore.dll back to the latest available in WinSxS before August 2020, an got these numbers - so no relief compared to August, but sime relief compared to the Jan 2021 results (We have only tried this on a Staging environment!)

.

Alessio P 66 Reputation points

2021-01-14T15:09:58.627+00:00

Please, double check you vmdatastore.dll that match this version:
10.0.17763.348

Alex

Jens Klarskov Jensen 1 Reputation point

2021-01-14T15:14:31.463+00:00

We reveted to the newest we could find in WinSxS, that was before August 2020, and it was the one you list :-)
So multiple isses.

Jens Klarskov Jensen 1 Reputation point

2021-01-14T15:19:46.787+00:00

As our setup is utterly company internal and we have problems meeting Feature Teams expectations for Test Automation, we are currently testing Rollback to Summer 2020.

Another Win2016 Cluster runs perfect, but does not support Win2019 Vms.
Sign in to comment