Virtual machines failed to start after installing Oct 2023 Update (KB5031364)

Martin Vogt 71

Hi all

After installing the Oct patch on Windows Server 2022 Standard 21H2, most of the virtual machines don't start anymore. There is a known issue described in the KB below, but it's not true only VMs which secure boot are affected.

Uninstalling the update fixes the problem and the VMs are starting up normally. Anyone have a fix for this?? I cannot beleve that MS is rolling out an update like this > This will hit a lot of admins quiet hard!

After installing this update on guest virtual machines (VMs) running Windows Server 2022 on some versions of VMware ESXi, Windows Server 2022 might not start up. Only Windows Server 2022 VMs with Secure Boot enabled are affected by this issue. Affected versions of VMware ESXi are versions vSphere ESXi 7.0.x and below.

https://support.microsoft.com/en-us/topic/october-10-2023-kb5031364-os-build-20348-2031-7f1d69e7-c468-4566-887a-1902af791bbc

The errors i receive when i want to power the machine (from the Hyper-V worker event log):

EventID 12140
'<vm-name>': Attachment '<path to disk>.vhdx' failed to open because of error: 'Incorrect function. ' (7864368). (Virtual machine ID xxxx)

EventID 12010
'<vm-name>' Synthetic SCSI Controller (Instance ID xxxx): Failed to Power on with Error 'Incorrect function. ' (0x80070001). (Virtual machine ID xxx)

EventID 12030
'<vm-name>' failed to start. (Virtual machine ID xxxx)

Paul Coleman 20 Reputation points

2023-10-13T19:37:55.64+00:00

I'm having the Same Exact issue on a HyperV 2019 Server, and I am seeing the same type of errors "incorrect function" when starting my VMs (they wont start)

Two updates were installed on my server KB5031361 and KB5031222

Here is the message SErver Manager is giving me.

RemoteException: Failed to start virtual machine TOOLS. Error: 'TOOLS' failed to start. Microsoft Emulated IDE Controller (Instance ID 83F8638B-8DCA-4152-9EDA-2CA8B33039B4): Failed to Power on with Error 'Incorrect function.'. Failed to open attachment 'D:\Hyper-V\Virtual Hard Disks\TOOLS_1C1580BD-0253-4888-88AA-259EE9D43246.avhdx'. Error: 'Incorrect function.'.
Martin Vogt 71 Reputation points

2023-10-17T13:59:57.37+00:00

My Ticket TrackingID#2310130010003816
stephc_msft 21 Reputation points

2023-10-18T18:17:50.9633333+00:00

I have a set of files (vhdx and .rct/.mrt) that show the issue, and am looking into it.

Also see the "Incorrect funtion" if try to locally mount the vhdx on a post 10B system.

See vhdmp event log event 24 (on WS2022) indicating its having some issue with the .rct file.

Workaround for now is to delete the .rct and .mrt file, to allow the VM to start

And of course that will mean the next host level backup of the VM will be a full backup, but that is not too unreasonable.
Karl Wester-Ebbinghaus 25 Reputation points

2023-10-19T08:09:50.1533333+00:00

Thank you for your confirmation, Stephen. Do you believe this CU will be drawn or KIR? This all, the performance fix for Veeam and the side-effect should be included in the update history, please. Could you promote this?
stephc_msft 21 Reputation points

2023-10-19T09:49:52.44+00:00

It's being looked into, but any decision is above my level of influence.
I doubt the CU will be withdraw or re-released with the RCT fix disabled by default
[KIR for others that aren't aware is a way that some individual fixes inside the CU can be enabled/disable by registry keys or group policy, but only for fixes that are designed that way (and this RCT fix isnt)]
Its 'interesting' and 'good' that only some customers using veeam or other host level backups, and with VM's using RCT files, seem to have issues.
Of course it is alarming and painful for those that do hit it unexpectedly, but luckily so far that has limited. But of course many Enterprises are probably still scheduling their October CU update and havent done it yet ...
Thanks for 'documenting' it here and in the other forums, so hopefully others can find the info in the short term whilst things get sorted out. And relevant support specialists in Microsoft have been alerted and informed, as so hopefully should be able to quickly help other customers with mitigation.
Karl Wester-Ebbinghaus 25 Reputation points

2023-10-19T10:15:21.2766667+00:00

Thank you for your time, confirmation, and appreciation.
Glad Martin brought this up, so early after CU release.
Sylvain RÉMY 1 Reputation point

2023-10-20T12:45:39.0033333+00:00

No issue here this morning (rreboot once a week)
HyperV Server 2019 + Win 2022 ((full english)
Karl Wester-Ebbinghaus 25 Reputation points

2023-11-20T17:49:14.5266667+00:00

The issue is fixed with the CU 11-2023. Not listed in Update History, though.
Have updated my posting on techcommunity.microsoft.com

Accepted answer

Chris Wroten 150 Reputation points

2023-10-14T20:56:06.2433333+00:00

For me, on an impacted VM, I was able to resolve this by deleting some extra files used when running the VHDX. Specifically, there was an .mrt (Modifiable Region Table) file and a .rct (Resilient Change Tracking) file for the VHD still present after shutdown/save/patch update and these were the cause behind the system being unwilling to start the VM. Deleted them, and it started right up. Probably lost information regarding the delta from the last backup time, as that seems to be what these files are used for, but it's better than rolling back an update or having an unusable VM.
Please sign in to rate this answer.

15 people found this answer helpful.
Meaghan Aldridge 0 Reputation points

2023-10-15T06:29:53.7+00:00

Thanks for the tips, removing 5031361 was the trick for me on Server 2019. After a reboot, servers set to always start started. Thank god this was noticed on a Sunday afternoon and not first thing Monday but none of my VM's started on this host.

Just to be safe I also removed 5031222

wusa /uninstall /kb:5031361

Martin Kren 0 Reputation points

2023-10-15T21:23:23.72+00:00

Thank you, the Tip with removing this two file types saved my Hyper-V Cluster i was updating last night, already thought I had serious issues with the Cluster Storage.

Karl Wester-Ebbinghaus 41 Reputation points MVP

2023-11-26T19:15:55.4633333+00:00

This workaround is obsolete.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

14 additional answers

Paul Coleman 20 Reputation points

2023-10-13T22:10:53.22+00:00

Got my HyperV 2019 Server working again.

uninstalled KB5031222 which also removed 5031361. Rebooted, and They did not autostart, but I can now start them again.

wusa /uninstall /kb:5031222
Please sign in to rate this answer.

2 people found this answer helpful.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Martin Vogt 71 Reputation points

2023-10-14T05:54:56.6166667+00:00

I opened a support ticket at Microsoft. Can't beleve they released this update though the knew that it will cause problems on Hyper-V machines.

Interesting fact: Not all VM are affected, but i couldn't figure out the difference between the affected/not affected ones. For the affected VMs, i'm event unable to mount the .vhdx disks. Also Linux machines are affected regardless of secure boot turned on or not.
Please sign in to rate this answer.

2 people found this answer helpful.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Jetze Mellema 11 Reputation points

2023-10-14T19:47:37.6733333+00:00

Was able to resolve this on Server 2022 by uninstalling KB5031364.

wusa /uninstall /kb:5031364
Please sign in to rate this answer.

2 people found this answer helpful.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Jake Camp 11 Reputation points

2023-10-16T15:54:05.98+00:00

Removing the SATA controller from the VM did the trick for me
Please sign in to rate this answer.

2 people found this answer helpful.
Karl Wester-Ebbinghaus 25 Reputation points

2023-11-15T18:17:47.6233333+00:00

The issue (unrelated to this topic) on VMware ESXi WS 2022 VBS and other issues preventing startup has been addressed with the November CU according to the release health notes.

No information I have on the RCT Hyper-V Veeam. issue @stephc_msft any news from your end.

Sean H 0 Reputation points

2023-11-26T18:23:29.5066667+00:00

That worked for me. Thanks.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Virtual machines failed to start after installing Oct 2023 Update (KB5031364)

14 additional answers

Your answer