Hey All,
I deployed November updates to 800+ servers as part of our monthly patching. the servers are scheduled to got the updates @ 8 pm and if they are pending reboots they get rebooted automatically @ 4 am the following morning.
post reboots 10 servers have failed to load OS (All Server 2012 R2) with 2 different symptoms:
- Servers completely fail to load OS and go straight to recovery mode
- Servers try to load, display a message collecting information then it reboot to recovery mode.
reboot in safe mode wont work.
to fix this:
- Attach Windows Server 2012 installation ISO to CD-ROM
- Boot from CD-ROM
- Go to Repair section and then select Command prompt
- From the VM menu
Go to Guest
Install/Upgrade VMWare Tools
Click Okay
- Go back to Command prompt
- Type this: drvload "D:\Program Files\VMware\VMware Tools\Drivers\pvscsi\Win8\amd64\pvscsi.inf" (hit enter)
- Diskpart (Enter)
List Vol
Exit
(Our aim is to find the Windows drive. Try each drive one by one to find which drive has the Windows folder)
- del X:\windows\winsxs\pending.xml (X: is representing the Windows Drive. replace it with the one you found)
- sfc /scannow /offbootdir=X:\ /offwindir=X:\windows (X: is representing the Windows Drive. replace it with the one you found)
all servers are VM's running on VMWare.
we have 2 Data Centers, all broken Servers are in the same Data Center. other Server 2012 R2 in the same Data Center got updated and rebooted without any issues.
Cannot find anything in any log or Event Viewer. I found one .dmp file on one of the servers:
Microsoft (R) Windows Debugger Version 10.0.20153.1000 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [C:\Users\xxxxx\Desktop\113020-41218-01.dmp]
Mini Kernel Dump File: Only registers and stack trace are available
************* Path validation summary **************
Response Time (ms) Location
Deferred srv*
Symbol search path is: srv*
Executable search path is:
Windows 8.1 Kernel Version 9600 MP (2 procs) Free x64
Product: Server, suite: TerminalServer SingleUserTS
Edition build lab: 9600.19846.amd64fre.winblue_ltsb_escrow.200923-1735
Machine Name:
Kernel base = 0xfffff801f0e7f000 PsLoadedModuleList = 0xfffff801
f11445d0
Debug session time: Mon Nov 30 05:17:11.633 2020 (UTC + 11:00)
System Uptime: 0 days 0:00:42.326
Loading Kernel Symbols
...............................................................
.......................................
Loading User Symbols
Loading unloaded module list
....
For analysis of this file, run !analyze -v
nt!KeBugCheckEx:
fffff801f0fbf4c0 48894c2408 mov qword ptr [rsp+8],rcx ss:0018:ffffd000
453e4800=000000000000004c
1: kd> !analyze -v
**
Bugcheck Analysis
**
WINLOGON_FATAL_ERROR (c000021a)
The Winlogon process terminated unexpectedly.
Arguments:
Arg1: ffffc00152ea4800, String that identifies the problem.
Arg2: ffffffffc0000428, Error Code.
Arg3: 0000000000000000
Arg4: 0000005f19d10768
Debugging Details:
------------------
ETW minidump data unavailable
fffff801f1111e58: Unable to get Flags value from nt!KdVersionBlock
GetUlongPtrFromAddress: unable to read from fffff801f11ce308
KEY_VALUES_STRING: 1
Key : Analysis.CPU.mSec
Value: 2436
Key : Analysis.DebugAnalysisProvider.CPP
Value: Create: 8007007e on AU69054
Key : Analysis.DebugData
Value: CreateObject
Key : Analysis.DebugModel
Value: CreateObject
Key : Analysis.Elapsed.mSec
Value: 2470
Key : Analysis.Memory.CommitPeak.Mb
Value: 63
Key : Analysis.System
Value: CreateObject
Key : WER.OS.Branch
Value: winblue_ltsb_escrow
Key : WER.OS.Timestamp
Value: 2020-09-23T17:35:00Z
Key : WER.OS.Version
Value: 8.1.9600.19846
ADDITIONAL_XML: 1
OS_BUILD_LAYERS: 1
VIRTUAL_MACHINE: VMware
ERROR_CODE: (NTSTATUS) 0xc000021a - {Fatal System Error} The %hs system process terminated unexpectedly with a status of 0x
EXCEPTION_CODE_STR: c000021a
EXCEPTION_PARAMETER1: ffffc00152ea4800
EXCEPTION_PARAMETER2: ffffffffc0000428
EXCEPTION_PARAMETER3: 0000000000000000
EXCEPTION_PARAMETER4: 5f19d10768
BUGCHECK_CODE: c000021a
BUGCHECK_P1: ffffc00152ea4800
BUGCHECK_P2: ffffffffc0000428
BUGCHECK_P3: 0
BUGCHECK_P4: 5f19d10768
PROCESS_NAME: smss.exe
ADDITIONAL_DEBUG_TEXT: initial session process or
IMAGE_NAME: ntkrnlmp.exe
MODULE_NAME: nt
CUSTOMER_CRASH_COUNT: 1
STACK_TEXT:
ffffd000453e47f8 fffff801
f11f78a9 : 000000000000004c 00000000
c000021a ffffd000463fd538 ffffe001
23fb7860 : nt!KeBugCheckEx
ffffd000453e4800 fffff801
f11ef194 : 0000000000000000 ffffd000
453e4919 0000000000000000 00000000
00000002 : nt!PopGracefulShutdown+0x2c9
ffffd000453e4840 fffff801
f0fcf3e3 : ffffe00123c7d880 00000000
00000000 00000000c0000004 ffffd000
453e4a00 : nt! ?? ::OKHAJAOM::string'+0x1d64 ffffd000
453e4980 fffff801f0fc34a0 : fffff801
f13f678d 0000000000000001 ffffd000
453e4b98 00000000c0000004 : nt!KiSystemServiceCopyEnd+0x13 ffffd000
453e4b18 fffff801f13f678d : 00000000
00000001 ffffd000453e4b98 00000000
c0000004 fffff801f1160180 : nt!KiServiceLinkage ffffd000
453e4b20 fffff801f1339d83 : 00000000
00000000 0000000000000000 fffff801
f1160180 ffffe00123c7d9c0 : nt! ?? ::NNGAKEGL::
string'+0x6341d
ffffd000453e4be0 fffff801
f0f28722 : fffff801f0f28668 00000000
00000000 0000000000000002 00000000
00000000 : nt!PopPolicyWorkerAction+0x63
ffffd000453e4c50 fffff801
f0eec5bf : fffff80100000002 ffffe001
23c7d880 fffff801f112b040 fffff801
f1124e60 : nt!PopPolicyWorkerThread+0xba
ffffd000453e4c90 fffff801
f0f4293e : 351bdc594dba662b fffff801
f1160180 0000000000000080 ffffe001
238b1040 : nt!ExpWorkerThread+0x69f
ffffd000453e4d40 fffff801
f0fc6f66 : fffff801f1160180 ffffe001
23c7d880 ffffe00123898880 00000002
2d5f0f64 : nt!PspSystemThreadStartup+0x18a
ffffd000453e4da0 00000000
00000000 : ffffd000453e5000 ffffd000
453df000 0000000000000000 00000000
00000000 : nt!KiStartSystemThread+0x16
SYMBOL_NAME: nt! ?? ::OKHAJAOM::`string'+1d64
IMAGE_VERSION: 6.3.9600.19846
STACK_COMMAND: .thread ; .cxr ; kb
BUCKET_ID_FUNC_OFFSET: 1d64
FAILURE_BUCKET_ID: 0xc000021a_SmpDestroyControlBlock_smss.exe_Terminated_c0000428_nt!??::OKHAJAOM::string
OS_VERSION: 8.1.9600.19846
BUILDLAB_STR: winblue_ltsb_escrow
OSPLATFORM_TYPE: x64
OSNAME: Windows 8.1
FAILURE_ID_HASH: {341f3cd7-cdf2-aefe-6083-b8fc309194a4}
Followup: MachineOwner
Has anyone else had issues with last month updates?
Much appriciated.