Virtual machines failed to start after installing Oct 2023 Update (KB5031364)

Question

Hi all

After installing the Oct patch on Windows Server 2022 Standard 21H2, most of the virtual machines don't start anymore. There is a known issue described in the KB below, but it's not true only VMs which secure boot are affected.

Uninstalling the update fixes the problem and the VMs are starting up normally. Anyone have a fix for this?? I cannot beleve that MS is rolling out an update like this > This will hit a lot of admins quiet hard!

After installing this update on guest virtual machines (VMs) running Windows Server 2022 on some versions of VMware ESXi, Windows Server 2022 might not start up. Only Windows Server 2022 VMs with Secure Boot enabled are affected by this issue. Affected versions of VMware ESXi are versions vSphere ESXi 7.0.x and below.

https://support.microsoft.com/en-us/topic/october-10-2023-kb5031364-os-build-20348-2031-7f1d69e7-c468-4566-887a-1902af791bbc

The errors i receive when i want to power the machine (from the Hyper-V worker event log):

EventID 12140
'<vm-name>': Attachment '<path to disk>.vhdx' failed to open because of error: 'Incorrect function. ' (7864368). (Virtual machine ID xxxx)

EventID 12010
'<vm-name>' Synthetic SCSI Controller (Instance ID xxxx): Failed to Power on with Error 'Incorrect function. ' (0x80070001). (Virtual machine ID xxx)

EventID 12030
'<vm-name>' failed to start. (Virtual machine ID xxxx)

Answer 1

For me, on an impacted VM, I was able to resolve this by deleting some extra files used when running the VHDX. Specifically, there was an .mrt (Modifiable Region Table) file and a .rct (Resilient Change Tracking) file for the VHD still present after shutdown/save/patch update and these were the cause behind the system being unwilling to start the VM. Deleted them, and it started right up. Probably lost information regarding the delta from the last backup time, as that seems to be what these files are used for, but it's better than rolling back an update or having an unusable VM.

Answer 2

The system I have that is affected by this has 3 Gen2 VMs with no SATA controllers. Secure boot is enabled on the two Windows VMs and disabled on the Linux VM. All 3 would fail to start until I removed KB5031364 from the host. Several other Windows Server 2022 system are fine.

Answer 3

We figured out the following:

1. only standalone VM's are affected not clustered VM's
2. affected VM's have .mrt and .rct File in addition to the .vhdx
   1. in your case they were created by DPM

Fast workaround: delete the .mrt and .rct file

negative Side Effect: error in DPM --> Recovery Point creation failed and backup needs consistency check.

Answer 4

As a remediation we have renamed the Veeam files and this made the VM working (starting) again.

Veeam RCT Backup tonight recreated the files.
VM remains working when stopped / started.

Conclusion: RCT is affected when RCT has been used AND CU 10-2023 has been installed and the VM will restart / stopped / started thereafter.

We can see, one does create a new RCT backup post patch and VMs are all's fine.

MSFT Support TrackingID#2310170050003786

Please feel encouraged to reference the SR given here so the escalation engineer understands the bigger picture instead starting from 0 for each SR.

As one affected customer has a lightning-fast internet line, we were tasked to upload an export of an affected VM. Thankfully, Server Core. Mean Microsoft can now try to repro and test with an affected asset.

Answer 5

Hi, as there are several questions. The 11-2023 contains the fix.