Virtual machines failed to start after installing Oct 2023 Update (KB5031364)

Question

Hi all

After installing the Oct patch on Windows Server 2022 Standard 21H2, most of the virtual machines don't start anymore. There is a known issue described in the KB below, but it's not true only VMs which secure boot are affected.

Uninstalling the update fixes the problem and the VMs are starting up normally. Anyone have a fix for this?? I cannot beleve that MS is rolling out an update like this > This will hit a lot of admins quiet hard!

After installing this update on guest virtual machines (VMs) running Windows Server 2022 on some versions of VMware ESXi, Windows Server 2022 might not start up. Only Windows Server 2022 VMs with Secure Boot enabled are affected by this issue. Affected versions of VMware ESXi are versions vSphere ESXi 7.0.x and below.

https://support.microsoft.com/en-us/topic/october-10-2023-kb5031364-os-build-20348-2031-7f1d69e7-c468-4566-887a-1902af791bbc

The errors i receive when i want to power the machine (from the Hyper-V worker event log):

EventID 12140
'<vm-name>': Attachment '<path to disk>.vhdx' failed to open because of error: 'Incorrect function. ' (7864368). (Virtual machine ID xxxx)

EventID 12010
'<vm-name>' Synthetic SCSI Controller (Instance ID xxxx): Failed to Power on with Error 'Incorrect function. ' (0x80070001). (Virtual machine ID xxx)

EventID 12030
'<vm-name>' failed to start. (Virtual machine ID xxxx)

Answer 1

For me, on an impacted VM, I was able to resolve this by deleting some extra files used when running the VHDX. Specifically, there was an .mrt (Modifiable Region Table) file and a .rct (Resilient Change Tracking) file for the VHD still present after shutdown/save/patch update and these were the cause behind the system being unwilling to start the VM. Deleted them, and it started right up. Probably lost information regarding the delta from the last backup time, as that seems to be what these files are used for, but it's better than rolling back an update or having an unusable VM.

Answer 2

"on some versions of VMware ESXi"

this one is an older issue, one need to patch the ESXi hosts
Disabling Secure Boot is not a permanent solution
https://kb.vmware.com/s/article/90947

It is important to use VBS and SecureBoot starting with WS2022, and at least Secure Boot with WS2019, it will make future transitions lot easier.

Can you confirm that you have an affected VM Gen1 running on Hyper-V? Are Gen2 VMs affected, too?

Answer 3

can confirm what has been said
also that the quickest remediation is to rename or delete the rct mct files
Thank you @MarkusGrudl

we have chosen to rename the files to make it reproducible for MSFT Azure Support

Affected VMs were unclustered Gen2 VMs (management and Domain Controller for / on Azure Stack HCI 22H2)

[Content]
'azDC02' failed to start.Synthetic SCSI Controller (Instance ID 6B5F6BE4-DEC2-4A11-8FD6-E77205FDF6EA): Failed to Power on with Error 'Incorrect function.'.Attachment 'C:\Hyper-V\azDC02\Virtual Hard Disks\azDC02.vhdx' failed to open because of error: 'Incorrect function.'.[Expanded Information]
'azDC02' failed to start. (Virtual machine ID B4A7FC43-676C-4DC2-9604-9ABC83DB7B2A)'azDC02' Synthetic SCSI Controller (Instance ID 6B5F6BE4-DEC2-4A11-8FD6-E77205FDF6EA): Failed to Power on with Error 'Incorrect function.' (0x80070001). (Virtual machine ID B4A7FC43-676C-4DC2-9604-9ABC83DB7B2A)'azDC02': Attachment 'C:\Hyper-V\azDC02\Virtual Hard Disks\azDC02.vhdx' failed to open because of error: 'Incorrect function.' (0x80070001). (Virtual machine ID B4A7FC43-676C-4DC2-9604-9ABC83DB7B2A)[^] Hide details [Close]

Answer 4

hopefully increasing the traction and finding the root cause and reproducibility I have picked this thread up and backlinked it here
https://techcommunity.microsoft.com/t5/azure-stack/starting-vms-may-fail-after-installation-of-10-2023-cu-when/m-p/3956844/highlight/true#M148

just avoid people consider me spamming this, it would be great of someone could also post it on Windows Server community on techcommunity, but as I cannot personally confirm it on Windows Server but only Azure Stack HCI, I decided to focus on what I have seen and remediated myself.

Thank you everyone for bringing this up and further details!

Answer 5

Can somone confirm that KB5031364 fix that performance problems with CSVFS?

When VM is on diffrent host compared to vhdx file.