Virtual machines failed to start after installing Oct 2023 Update (KB5031364)

Martin Vogt 71

Hi all

After installing the Oct patch on Windows Server 2022 Standard 21H2, most of the virtual machines don't start anymore. There is a known issue described in the KB below, but it's not true only VMs which secure boot are affected.

Uninstalling the update fixes the problem and the VMs are starting up normally. Anyone have a fix for this?? I cannot beleve that MS is rolling out an update like this > This will hit a lot of admins quiet hard!

After installing this update on guest virtual machines (VMs) running Windows Server 2022 on some versions of VMware ESXi, Windows Server 2022 might not start up. Only Windows Server 2022 VMs with Secure Boot enabled are affected by this issue. Affected versions of VMware ESXi are versions vSphere ESXi 7.0.x and below.

https://support.microsoft.com/en-us/topic/october-10-2023-kb5031364-os-build-20348-2031-7f1d69e7-c468-4566-887a-1902af791bbc

The errors i receive when i want to power the machine (from the Hyper-V worker event log):

EventID 12140
'<vm-name>': Attachment '<path to disk>.vhdx' failed to open because of error: 'Incorrect function. ' (7864368). (Virtual machine ID xxxx)

EventID 12010
'<vm-name>' Synthetic SCSI Controller (Instance ID xxxx): Failed to Power on with Error 'Incorrect function. ' (0x80070001). (Virtual machine ID xxx)

EventID 12030
'<vm-name>' failed to start. (Virtual machine ID xxxx)

Paul Coleman 20 Reputation points

2023-10-13T19:37:55.64+00:00

I'm having the Same Exact issue on a HyperV 2019 Server, and I am seeing the same type of errors "incorrect function" when starting my VMs (they wont start)

Two updates were installed on my server KB5031361 and KB5031222

Here is the message SErver Manager is giving me.

RemoteException: Failed to start virtual machine TOOLS. Error: 'TOOLS' failed to start. Microsoft Emulated IDE Controller (Instance ID 83F8638B-8DCA-4152-9EDA-2CA8B33039B4): Failed to Power on with Error 'Incorrect function.'. Failed to open attachment 'D:\Hyper-V\Virtual Hard Disks\TOOLS_1C1580BD-0253-4888-88AA-259EE9D43246.avhdx'. Error: 'Incorrect function.'.
Martin Vogt 71 Reputation points

2023-10-17T13:59:57.37+00:00

My Ticket TrackingID#2310130010003816
stephc_msft 21 Reputation points

2023-10-18T18:17:50.9633333+00:00

I have a set of files (vhdx and .rct/.mrt) that show the issue, and am looking into it.

Also see the "Incorrect funtion" if try to locally mount the vhdx on a post 10B system.

See vhdmp event log event 24 (on WS2022) indicating its having some issue with the .rct file.

Workaround for now is to delete the .rct and .mrt file, to allow the VM to start

And of course that will mean the next host level backup of the VM will be a full backup, but that is not too unreasonable.
Karl Wester-Ebbinghaus 25 Reputation points

2023-10-19T08:09:50.1533333+00:00

Thank you for your confirmation, Stephen. Do you believe this CU will be drawn or KIR? This all, the performance fix for Veeam and the side-effect should be included in the update history, please. Could you promote this?
stephc_msft 21 Reputation points

2023-10-19T09:49:52.44+00:00

It's being looked into, but any decision is above my level of influence.
I doubt the CU will be withdraw or re-released with the RCT fix disabled by default
[KIR for others that aren't aware is a way that some individual fixes inside the CU can be enabled/disable by registry keys or group policy, but only for fixes that are designed that way (and this RCT fix isnt)]
Its 'interesting' and 'good' that only some customers using veeam or other host level backups, and with VM's using RCT files, seem to have issues.
Of course it is alarming and painful for those that do hit it unexpectedly, but luckily so far that has limited. But of course many Enterprises are probably still scheduling their October CU update and havent done it yet ...
Thanks for 'documenting' it here and in the other forums, so hopefully others can find the info in the short term whilst things get sorted out. And relevant support specialists in Microsoft have been alerted and informed, as so hopefully should be able to quickly help other customers with mitigation.
Karl Wester-Ebbinghaus 25 Reputation points

2023-10-19T10:15:21.2766667+00:00

Thank you for your time, confirmation, and appreciation.
Glad Martin brought this up, so early after CU release.
Sylvain RÉMY 1 Reputation point

2023-10-20T12:45:39.0033333+00:00

No issue here this morning (rreboot once a week)
HyperV Server 2019 + Win 2022 ((full english)
Karl Wester-Ebbinghaus 25 Reputation points

2023-11-20T17:49:14.5266667+00:00

The issue is fixed with the CU 11-2023. Not listed in Update History, though.
Have updated my posting on techcommunity.microsoft.com

Accepted answer

Chris Wroten 150 Reputation points

2023-10-14T20:56:06.2433333+00:00

For me, on an impacted VM, I was able to resolve this by deleting some extra files used when running the VHDX. Specifically, there was an .mrt (Modifiable Region Table) file and a .rct (Resilient Change Tracking) file for the VHD still present after shutdown/save/patch update and these were the cause behind the system being unwilling to start the VM. Deleted them, and it started right up. Probably lost information regarding the delta from the last backup time, as that seems to be what these files are used for, but it's better than rolling back an update or having an unusable VM.
Please sign in to rate this answer.

15 people found this answer helpful.
Meaghan Aldridge 0 Reputation points

2023-10-15T06:29:53.7+00:00

Thanks for the tips, removing 5031361 was the trick for me on Server 2019. After a reboot, servers set to always start started. Thank god this was noticed on a Sunday afternoon and not first thing Monday but none of my VM's started on this host.

Just to be safe I also removed 5031222

wusa /uninstall /kb:5031361

Martin Kren 0 Reputation points

2023-10-15T21:23:23.72+00:00

Thank you, the Tip with removing this two file types saved my Hyper-V Cluster i was updating last night, already thought I had serious issues with the Cluster Storage.

Karl Wester-Ebbinghaus 31 Reputation points MVP

2023-11-26T19:15:55.4633333+00:00

This workaround is obsolete.
Sign in to comment

14 additional answers

stephc_msft 21 Reputation points

2023-11-03T11:00:53.8833333+00:00

This issue with the October 2023 Windows Server update (where a fix that helps overcome RCT related VM IO performance issues has the unexpected side effect of not allowing some VM's to start) is due to be addressed in the November update.
Exact confirmation cannot be confirmed until closer to the date.
Please sign in to rate this answer.
Karthik B 0 Reputation points

2023-11-26T09:12:31.6766667+00:00

Hi All,

We too faced this issue based on HyperV 2019 Servers for the kb5031361 and kb5031364.

Is there any fix\ latest workaround is applicable for this issue from Microsoft.
Sign in to comment
Dennis 11 Reputation points

2023-11-15T12:56:14.4433333+00:00

Does the November 2023 Windows Server update include the fix? If it does I can't find it in the release notes.
Please sign in to rate this answer.
Koen DV 0 Reputation points

2023-11-19T10:40:09.1633333+00:00

I experienced the same problem after updating to the October update. I was running Veeam backup 9.4.

I've updated Veeam Backup & Replication to version 12.0.0.1420.

Today I installed the November update for Windows Server 2019 (1809) (KB5032196) without issues.

Karl Wester-Ebbinghaus 25 Reputation points

2023-11-20T17:47:38.89+00:00

Hi Dennis, it is fixed with the CU 11-2023. Not listed in Update History, though.

JOSE MEDINA 0 Reputation points

2023-12-08T14:30:26.5533333+00:00

Looking for that answer as well. Can't seem to find any information.

Karl Wester-Ebbinghaus 25 Reputation points

2023-12-11T09:31:01.8066667+00:00

Jose it is fixed with the 11-2023 CU, but not included in update history, very unfortunately.
Sign in to comment