Event ID 41 , Source: Microsoft-Windows-Kernel-Power

Question

Hi All;

Im running into critical Error causing my VM server keep restarting.

Appreciate if anybody can help me in the dump file analysis and pinpoint me to the root cause.

VM Win 2019 .
Event ID details:
Log Name: System
Source: Microsoft-Windows-Kernel-Power
Date: 06-May-22 2:50:24 PM
Event ID: 41
Task Category: (63)
Level: Critical
Keywords: (70368744177664),(2)
User: SYSTEM
Computer:
Description:
The system has rebooted without cleanly shutting down first. This error could be caused if the system stopped responding, crashed, or lost power unexpectedly.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Microsoft-Windows-Kernel-Power" Guid="{331c3b3a-2005-44c2-ac5e-77220c37d6b4}" />
<EventID>41</EventID>
<Version>6</Version>
<Level>1</Level>
<Task>63</Task>
<Opcode>0</Opcode>
<Keywords>0x8000400000000002</Keywords>
<TimeCreated SystemTime="2022-05-06T11:50:24.047293800Z" />
<EventRecordID>469822</EventRecordID>
<Correlation />
<Execution ProcessID="4" ThreadID="8" />
<Channel>System</Channel>
<Computer> </Computer>
<Security UserID="S-1-5-18" />
</System>
<EventData>
<Data Name="BugcheckCode">193</Data>
<Data Name="BugcheckParameter1">0xffffbf85b3208ec0</Data>
<Data Name="BugcheckParameter2">0xffffbf85b3208585</Data>
<Data Name="BugcheckParameter3">0x174140</Data>
<Data Name="BugcheckParameter4">0x23</Data>
<Data Name="SleepInProgress">0</Data>
<Data Name="PowerButtonTimestamp">0</Data>
<Data Name="BootAppStatus">0</Data>
<Data Name="Checkpoint">0</Data>
<Data Name="ConnectedStandbyInProgress">false</Data>
<Data Name="SystemSleepTransitionsToOn">0</Data>
<Data Name="CsEntryScenarioInstanceId">0</Data>
<Data Name="BugcheckInfoFromEFI">false</Data>
<Data Name="CheckpointStatus">0</Data>
</EventData>
</Event>

• Last memory Dump details (extracted by WinDbg tool) :

Microsoft (R) Windows Debugger Version 10.0.22549.1000 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [C:\Temp\MiniDump-new\050622-15531-01.dmp]
Mini Kernel Dump File: Only registers and stack trace are available

************* Path validation summary **************
Response Time (ms) Location
Deferred srv*
Symbol search path is: srv*
Executable search path is:
Windows 10 Kernel Version 17763 MP (8 procs) Free x64
Product: Server, suite: TerminalServer SingleUserTS
Edition build lab: 17763.1.amd64fre.rs5_release.180914-1434
Machine Name:
Kernel base = 0xfffff8035da11000 PsLoadedModuleList = 0xfffff8035de28470
Debug session time: Fri May 6 18:00:55.638 2022 (UTC + 3:00)
System Uptime: 0 days 0:29:25.452
Loading Kernel Symbols
..

Press ctrl-c (cdb, kd, ntsd) or ctrl-break (windbg) to abort symbol loads that take too long.
Run !sym noisy before .reload to track down problems loading symbols.

.............................................................
................................................................
...................................
Loading User Symbols
Loading unloaded module list
......
For analysis of this file, run !analyze -v
nt!KeBugCheckEx:
fffff8035dbc9340 48894c2408 mov qword ptr [rsp+8],rcx ss:0018:fffff80360334430=00000000000000c1
0: kd> !analyze -v

---

• *
• Bugcheck Analysis *
• *

  ---

SPECIAL_POOL_DETECTED_MEMORY_CORRUPTION (c1)
Special pool has detected memory corruption. Typically the current thread's
stack backtrace will reveal the guilty party.
Arguments:
Arg1: ffffa18891a40d80, address trying to free
Arg2: ffffa18891a40c05, address where bits are corrupted
Arg3: 0000000000c74278, (reserved)
Arg4: 0000000000000023, caller is freeing an address where nearby bytes within the same page have been corrupted

Debugging Details:

---

*** WARNING: Unable to verify timestamp for win32k.sys

KEY_VALUES_STRING: 1

Key  : Analysis.CPU.mSec
Value: 6546

Key  : Analysis.DebugAnalysisManager
Value: Create

Key  : Analysis.Elapsed.mSec
Value: 62057

Key  : Analysis.Init.CPU.mSec
Value: 2139

Key  : Analysis.Init.Elapsed.mSec
Value: 14176

Key  : Analysis.Memory.CommitPeak.Mb
Value: 84

Key  : WER.OS.Branch
Value: rs5_release

Key  : WER.OS.Timestamp
Value: 2018-09-14T14:34:00Z

Key  : WER.OS.Version
Value: 10.0.17763.1

FILE_IN_CAB: 050622-15531-01.dmp

BUGCHECK_CODE: c1

BUGCHECK_P1: ffffa18891a40d80

BUGCHECK_P2: ffffa18891a40c05

BUGCHECK_P3: c74278

BUGCHECK_P4: 23

SPECIAL_POOL_CORRUPTION_TYPE: 23

BLACKBOXBSD: 1 (!blackboxbsd)

BLACKBOXPNP: 1 (!blackboxpnp)

CUSTOMER_CRASH_COUNT: 1

PROCESS_NAME: System

STACK_TEXT:
fffff80360334428 fffff8035dd2b19e : 00000000000000c1 ffffa18891a40d80 ffffa18891a40c05 0000000000c74278 : nt!KeBugCheckEx
fffff80360334430 fffff8035dd58de1 : fffff8035da11000 fffff80360334820 ffffa18891a40d80 0000000000000000 : nt!ExpFreeHeapSpecialPool+0x18a
fffff80360334480 fffff807ca408bf1 : ffffa18891a40d80 0000000000000000 0000000000000002 fffff807ca36f13d : nt!ExFreePoolWithTag+0x181
fffff803603345b0 fffff8035e334827 : ffffa18891a40d80 fffff80360334820 ffffa18833fd6520 ffffa18833fd6520 : VerifierExt!ExFreePoolWithTag_wrapper+0x11
fffff803603345e0 fffff8035e33479c : ffffa1883ac06f80 fffff8035da2a202 0000000000000280 fffff807ca36ef06 : nt!VerifierExFreePoolWithTag+0x57
fffff80360334610 fffff8035e3347b9 : ffffa18891a40db0 ffffa18891a40de8 0000000000000002 ffffa18833e47e38 : nt!VerifierExFreePool+0x1c
fffff80360334640 fffff8036048c479 : fffff80360334798 ffffa18882cfcf88 0000000000000280 ffffa18891a40de8 : nt!VerifierExFreePoolEx+0x9
fffff80360334670 fffff8036048378f : ffffa18891a40d80 0000000000000000 ffffa18833e47e28 0000000000000000 : tcpip!PplGenericFreeFunction+0x59
fffff803603346a0 fffff80360488ba1 : ffffa1883b24ad78 fffff80360334820 ffffa18891a40f88 0000000000000000 : tcpip!WfpAleFreeRemoteEndpoint+0x29f
fffff80360334720 fffff8035da4136c : fffff8035b124f80 fffff80360601520 fffff8035b122180 0000000000000000 : tcpip!LruCleanupDpcRoutine+0x521
fffff80360334910 fffff8035da409ae : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : nt!KiExecuteAllDpcs+0x2ec
fffff80360334a50 fffff8035dbccd6a : 0000000000000000 fffff8035b122180 0000000000000000 fffff8035df6d840 : nt!KiRetireDpcList+0x1ae
fffff80360334c60 0000000000000000 : fffff80360335000 fffff8036032f000 0000000000000000 0000000000000000 : nt!KiIdleLoop+0x5a

SYMBOL_NAME: tcpip!PplGenericFreeFunction+59

MODULE_NAME: tcpip

IMAGE_NAME: tcpip.sys

IMAGE_VERSION: 10.0.17763.2746

STACK_COMMAND: .cxr; .ecxr ; kb

BUCKET_ID_FUNC_OFFSET: 59

FAILURE_BUCKET_ID: 0xC1_23_VRF_tcpip!PplGenericFreeFunction

OS_VERSION: 10.0.17763.1

BUILDLAB_STR: rs5_release

OSPLATFORM_TYPE: x64

OSNAME: Windows 10

FAILURE_ID_HASH: {540e5758-7e83-17ba-47f3-c40a97625a84}

Followup: MachineOwner

---

0: kd> lmvm tcpip
Browse full module list
start end module name
fffff80360400000 fffff803606d9000 tcpip (pdb symbols) C:\ProgramData\Dbg\sym\tcpip.pdb\6CD3025FC195A29D5A02589D8196C86A1\tcpip.pdb
Loaded symbol image file: tcpip.sys
Mapped memory image file: C:\ProgramData\Dbg\sym\tcpip.sys\6FE275492d9000\tcpip.sys
Image path: tcpip.sys
Image name: tcpip.sys
Browse all global symbols functions data
Image was built with /Brepro flag.
Timestamp: 6FE27549 (This is a reproducible build file hash, not a timestamp)
CheckSum: 002D7760
ImageSize: 002D9000
File version: 10.0.17763.2746
Product version: 10.0.17763.2746
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 3.6 Driver
File date: 00000000.00000000
Translations: 0409.04b0
Information from resource tables:
CompanyName: Microsoft Corporation
ProductName: Microsoft® Windows® Operating System
InternalName: tcpip.sys
OriginalFilename: tcpip.sys
ProductVersion: 10.0.17763.2746
FileVersion: 10.0.17763.2746 (WinBuild.160101.0800)
FileDescription: TCP/IP Driver
LegalCopyright: © Microsoft Corporation. All rights reserved.

Answer 1

Please run the DM log collector and post a share link into this thread using one drive, drop box, or google drive.

If the server can run the V2 log collector it will collect more files useful in the troubleshooting.

https://www.tenforums.com/bsod-crashes-debugging/2198-bsod-posting-instructions.html

https://www.elevenforum.com/t/bsod-posting-instructions.103/

Run:
https://www.tenforums.com/attachments/bsod-crashes-debugging/346094d1631611972-post-problem-reports-here-batch-files-use-bsod-debugging-collect_logs_from_event_viewer-2021-09-14.bat

Post a share link using one drive, drop box, or google drive.

.
.
.
.
.

Please remember to vote and to mark the replies as answers if they help.

On the bottom of each post there is:

Propose as answer = answered the question

On the left side of each post there is /\ with a number: click = a helpful post
.
.
.
.
.

Answer 2

Thanks @Docs for your reply.

DM and V2 output files are ready via the following OneDrive link:

https://1drv.ms/u/s!AjHi2Mpr9TXbgSvt7F6HlRzXPL2j?e=NC2Y2q

Appreciated.

Raed.

Answer 3

Windows error reporting was cleaned and mini dump files were missing.

Please make sure that Windows clean and third party software are not used to clean files during the troubleshooting.

1) Search for:

C:\Windows\LiveKernelReports\PoW32kWatchdog-20220506-1443.dmp
C:\Windows\MEMORY.DMP

Save each to the downloads folder > ZIP > post separate share links for each zipped file

2) Make a new restore point.

3) Restart Windows Driver Verifier (WDV) with these customized settings:

[ ] 0x00000200 Force pending I/O requests.
[ ] 0x00000400 IRP logging.
[ ] 0x00002000 Invariant MDL checking for stack.
[ ] 0x00004000 Invariant MDL checking for driver.
[ ] 0x00008000 Power framework delay fuzzing.
[ ] 0x00010000 Port/miniport interface checking.
[ ] 0x00040000 Systematic low resources simulation.
[ ] 0x00080000 DDI compliance checking (additional).
[ ] 0x00200000 NDIS/WIFI verification.
[ ] 0x00800000 Kernel synchronization delay fuzzing.
[ ] 0x01000000 VM switch verification.
[ ] 0x02000000 Code integrity checks.

For any BSOD > run V2 > post a new share link into the newest post

If there is no immediate BSOD then open administrative command prompt and type or copy and paste:

verifier /querysettings

Post a share link into this thread.

.
.
.
.
.

Please remember to vote and to mark the replies as answers if they help.

On the bottom of each post there is:

Propose as answer = answered the question

On the left side of each post there is /\ with a number: click = a helpful post
.
.
.
.
.

Answer 4

The images / results displayed the use of the prior customized tests.
Please use the customized tests displayed in the latest post.

Please turn off WDV.

Then restart WDV with the customized test settings in the prior post (step #3).

[ ] 0x00000200 Force pending I/O requests.
[ ] 0x00000400 IRP logging.
[ ] 0x00002000 Invariant MDL checking for stack.
[ ] 0x00004000 Invariant MDL checking for driver.
[ ] 0x00008000 Power framework delay fuzzing.
[ ] 0x00010000 Port/miniport interface checking.
[ ] 0x00040000 Systematic low resources simulation.
[ ] 0x00080000 DDI compliance checking (additional).
[ ] 0x00200000 NDIS/WIFI verification.
[ ] 0x00800000 Kernel synchronization delay fuzzing.
[ ] 0x01000000 VM switch verification.
[ ] 0x02000000 Code integrity checks.

If there is no immediate BSOD then run the command: verifier /querysettings
and post a share link into this thread displaying the command with results.

For any BSOD > run the V2 log collector > post a share link into the newest post

.
.
.
.
.

Please remember to vote and to mark the replies as answers if they help.

On the bottom of each post there is:

Propose as answer = answered the question

On the left side of each post there is /\ with a number: click = a helpful post
.
.
.
.
.

Answer 5

Please see the instructions in the prior post:

For any BSOD > run the V2 log collector > post a share link into the newest post