Hi All,
Since around December my Org have been seeing an increase of BSOD reports across our Dell estate.
It appears to be linked to updates of some description, either Windows or Office.
Steps taken so far are:
- Patch Windows to latest patch (Jan 24)
- Update all Dell drivers using utility tool
- Check for malware/virus using CrowdStrike
Any help would be greatly appreciated as we'e at a loss as to the cause. Please see below minidump extract (via WinDbg)
All crashes relate to the below:
DRIVER_IRQL_NOT_LESS_OR_EQUAL
************* Preparing the environment for Debugger Extensions Gallery repositories **************
ExtensionRepository : Implicit
UseExperimentalFeatureForNugetShare : true
AllowNugetExeUpdate : true
AllowNugetMSCredentialProviderInstall : true
AllowParallelInitializationOfLocalRepositories : true
-- Configuring repositories
----> Repository : LocalInstalled, Enabled: true
----> Repository : UserExtensions, Enabled: true
>>>>>>>>>>>>> Preparing the environment for Debugger Extensions Gallery repositories completed, duration 0.016 seconds
************* Waiting for Debugger Extensions Gallery to Initialize **************
>>>>>>>>>>>>> Waiting for Debugger Extensions Gallery to Initialize completed, duration 0.031 seconds
----> Repository : UserExtensions, Enabled: true, Packages count: 0
----> Repository : LocalInstalled, Enabled: true, Packages count: 36
Microsoft (R) Windows Debugger Version 10.0.25921.1001 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [C:\Users\ash.horne\OneDrive - Northern Trains Ltd\Desktop\012924-48921-01.dmp]
Mini Kernel Dump File: Only registers and stack trace are available
Symbol search path is: srv*
Executable search path is:
Windows 10 Kernel Version 22621 MP (8 procs) Free x64
Product: WinNt, suite: TerminalServer SingleUserTS
Edition build lab: 22621.1.amd64fre.ni_release.220506-1250
Kernel base = 0xfffff80113a00000 PsLoadedModuleList = 0xfffff80114613530
Debug session time: Mon Jan 29 14:02:43.814 2024 (UTC + 0:00)
System Uptime: 0 days 0:10:37.834
Loading Kernel Symbols
...............................................................
................................................................
................................................................
.............................................................
Loading User Symbols
Loading unloaded module list
...........................
For analysis of this file, run !analyze -v
nt!KeBugCheckEx:
fffff80113e16bc0 48894c2408 mov qword ptr [rsp+8],rcx ss:0018:ffff960628b34150=000000000000000a
6: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.
If kernel debugger is available get stack backtrace.
Arguments:
Arg1: 0000000000000028, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000001, value 0 = read operation, 1 = write operation
Arg4: fffff801168855ff, address which referenced memory
Debugging Details:
*** WARNING: Unable to verify timestamp for Netwtw10.sys
KEY_VALUES_STRING: 1
Key : Analysis.CPU.mSec
Value: 3233
Key : Analysis.Elapsed.mSec
Value: 28265
Key : Analysis.IO.Other.Mb
Value: 6
Key : Analysis.IO.Read.Mb
Value: 18
Key : Analysis.IO.Write.Mb
Value: 53
Key : Analysis.Init.CPU.mSec
Value: 468
Key : Analysis.Init.Elapsed.mSec
Value: 18383
Key : Analysis.Memory.CommitPeak.Mb
Value: 115
Key : Bugcheck.Code.LegacyAPI
Value: 0xd1
Key : Failure.Bucket
Value: AV\_fwpkclnt!FwppInjectComplete
Key : Failure.Hash
Value: {0cd1ec7c-9b34-fb98-d3bd-b9ce089ba9de}
Key : Hypervisor.Enlightenments.ValueHex
Value: 1417cf94
Key : Hypervisor.Flags.AnyHypervisorPresent
Value: 1
Key : Hypervisor.Flags.ApicEnlightened
Value: 1
Key : Hypervisor.Flags.ApicVirtualizationAvailable
Value: 0
Key : Hypervisor.Flags.AsyncMemoryHint
Value: 0
Key : Hypervisor.Flags.CoreSchedulerRequested
Value: 0
Key : Hypervisor.Flags.CpuManager
Value: 1
Key : Hypervisor.Flags.DeprecateAutoEoi
Value: 0
Key : Hypervisor.Flags.DynamicCpuDisabled
Value: 1
Key : Hypervisor.Flags.Epf
Value: 0
Key : Hypervisor.Flags.ExtendedProcessorMasks
Value: 1
Key : Hypervisor.Flags.HardwareMbecAvailable
Value: 1
Key : Hypervisor.Flags.MaxBankNumber
Value: 0
Key : Hypervisor.Flags.MemoryZeroingControl
Value: 0
Key : Hypervisor.Flags.NoExtendedRangeFlush
Value: 0
Key : Hypervisor.Flags.NoNonArchCoreSharing
Value: 1
Key : Hypervisor.Flags.Phase0InitDone
Value: 1
Key : Hypervisor.Flags.PowerSchedulerQos
Value: 0
Key : Hypervisor.Flags.RootScheduler
Value: 0
Key : Hypervisor.Flags.SynicAvailable
Value: 1
Key : Hypervisor.Flags.UseQpcBias
Value: 0
Key : Hypervisor.Flags.Value
Value: 4853999
Key : Hypervisor.Flags.ValueHex
Value: 4a10ef
Key : Hypervisor.Flags.VpAssistPage
Value: 1
Key : Hypervisor.Flags.VsmAvailable
Value: 1
Key : Hypervisor.RootFlags.AccessStats
Value: 1
Key : Hypervisor.RootFlags.CrashdumpEnlightened
Value: 1
Key : Hypervisor.RootFlags.CreateVirtualProcessor
Value: 1
Key : Hypervisor.RootFlags.DisableHyperthreading
Value: 0
Key : Hypervisor.RootFlags.HostTimelineSync
Value: 1
Key : Hypervisor.RootFlags.HypervisorDebuggingEnabled
Value: 0
Key : Hypervisor.RootFlags.IsHyperV
Value: 1
Key : Hypervisor.RootFlags.LivedumpEnlightened
Value: 1
Key : Hypervisor.RootFlags.MapDeviceInterrupt
Value: 1
Key : Hypervisor.RootFlags.MceEnlightened
Value: 1
Key : Hypervisor.RootFlags.Nested
Value: 0
Key : Hypervisor.RootFlags.StartLogicalProcessor
Value: 1
Key : Hypervisor.RootFlags.Value
Value: 1015
Key : Hypervisor.RootFlags.ValueHex
Value: 3f7
Key : WER.OS.Branch
Value: ni\_release
Key : WER.OS.Version
Value: 10.0.22621.1
BUGCHECK_CODE: d1
BUGCHECK_P1: 28
BUGCHECK_P2: 2
BUGCHECK_P3: 1
BUGCHECK_P4: fffff801168855ff
FILE_IN_CAB: 012924-48921-01.dmp
WRITE_ADDRESS: fffff8011471d470: Unable to get MiVisibleState
Unable to get NonPagedPoolStart
Unable to get NonPagedPoolEnd
Unable to get PagedPoolStart
Unable to get PagedPoolEnd
unable to get nt!MmSpecialPagesInUse
0000000000000028
BLACKBOXBSD: 1 (!blackboxbsd)
BLACKBOXNTFS: 1 (!blackboxntfs)
BLACKBOXPNP: 1 (!blackboxpnp)
BLACKBOXWINLOGON: 1
CUSTOMER_CRASH_COUNT: 1
PROCESS_NAME: System
TRAP_FRAME: ffff960628b34290 -- (.trap 0xffff960628b34290)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=ffffcf8818f9eb40 rbx=0000000000000000 rcx=ffffcf8818f9eb40
rdx=ffffcf8818f9eb40 rsi=0000000000000000 rdi=0000000000000000
rip=fffff801168855ff rsp=ffff960628b34420 rbp=0000000000000000
r8=ffffcf8818f9eb40 r9=0000000000000065 r10=ffffcf88071030c0
r11=ffffcf8818f9eb40 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei ng nz na pe nc
fwpkclnt!FwppInjectComplete+0xaf:
fffff801168855ff f0ff4d28 lock dec dword ptr [rbp+28h] ss:0018:0000000000000028=????????
Resetting default scope
STACK_TEXT:
ffff960628b34148 fffff80113e2c4e9 : 000000000000000a 0000000000000028 0000000000000002 0000000000000001 : nt!KeBugCheckEx
ffff960628b34150 fffff80113e27a34 : 00000000ebf0e9b8 ffffcf8700024b00 ffff960600024b00 ffff960628b34600 : nt!KiBugCheckDispatch+0x69
ffff960628b34290 fffff801168855ff : 0000000000000000 ffffcf8818f9eb40 ffff960628b34540 fffff80116a2d355 : nt!KiPageFault+0x474
ffff960628b34420 fffff80115ef2967 : ffffcf8818f9eb40 0000000000000000 ffffcf881a598c80 00001e0000001e00 : fwpkclnt!FwppInjectComplete+0xaf
ffff960628b34460 fffff80115f1f726 : ffffcf881a598c80 0000000000000000 ffffcf8818f9eb40 ffff960628b345e0 : NETIO!NetioDereferenceNetBufferList+0x187
ffff960628b344a0 fffff80116987e10 : 0000000000000000 ffffcf8806ff6080 0000000000000000 ffff960628b34510 : NETIO!StreamRequestInjectCallback+0x66
ffff960628b344e0 fffff80116983e19 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : tcpip!TcpTcbReceive+0x6c0
ffff960628b34680 fffff80116982e0f : ffffcf87fb1ec008 000000017c2dcc53 ffffcf8801897138 0000000000000000 : tcpip!TcpMatchReceive+0x519
ffff960628b34850 fffff80116982a97 : ffffcf87f85f49d0 ffffcf880189901f ffffcf8800000000 ffff960628b374ea : tcpip!TcpReceive+0x35f
ffff960628b34960 fffff801169fb3c8 : ffffcf8806ff6002 0000000000000000 000000000000000c ffffcf8805dacb20 : tcpip!TcpNlClientReceivePreValidatedDatagrams+0x17
ffff960628b34990 fffff801169c6a1b : ffffcf8806ff6080 0000000000000000 fffff8011470e206 fffff80113e59c00 : tcpip!IpFlcReceivePreValidatedPackets+0xec8
ffff960628b34b20 fffff80113ceb8aa : 0000000000000006 fffff801169c68d0 ffffcf87f875e0f4 0000000000000002 : tcpip!FlReceiveNetBufferListChainCalloutRoutine+0x14b
ffff960628b34c70 fffff80113ceb81d : fffff801169c68d0 ffff960628b34db8 ffffcf87f875cbc0 0000000000000000 : nt!KeExpandKernelStackAndCalloutInternal+0x7a
ffff960628b34ce0 fffff801169c7d30 : ffffcf87fb0e6e40 fffff80115ef549e 0000000000000000 0000000000000000 : nt!KeExpandKernelStackAndCalloutEx+0x1d
ffff960628b34d20 fffff80115d726e1 : 0000000000000001 0000000000000001 0000000000000002 fffff80113c7a0c1 : tcpip!FlReceiveNetBufferListChain+0x530
ffff960628b35000 fffff80115d720fa : ffffcf8807089ae0 ffffcf8803d80801 ffffcf8800000000 fffff80100000002 : NDIS!ndisMIndicateNetBufferListsToOpen+0x141
ffff960628b350e0 fffff80115d96225 : ffffcf87fe4a51a0 0000000000000000 ffffcf8800000000 ffffcf87fe4a51a0 : NDIS!ndisMTopReceiveNetBufferLists+0x24a
ffff960628b351c0 fffff80115d95c93 : 0000000000000000 ffff960628b352b0 fffff80115d71eb0 0000000000000000 : NDIS!ndisCallReceiveHandler+0xb9
ffff960628b35210 fffff80115d9604b : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : NDIS!ndisCallNextDatapathHandler<2,void * __ptr64 & __ptr64,void (__cdecl*& __ptr64)(void * __ptr64,_NET_BUFFER_LIST * __ptr64,unsigned long,unsigned long,unsigned long),void * __ptr64 & __ptr64,_NET_BUFFER_LIST * __ptr64 & __ptr64,unsigned long & __ptr64,unsigned long & __ptr64,unsigned long & __ptr64>+0x3f
ffff960628b35260 fffff80115d95d82 : ffffcf87fe4a51a0 fffff80115d71eb0 ffffcf87fe4a51a0 0000000000000001 : NDIS!ndisIterativeDPInvokeHandlerOnTracker<2,void __cdecl(void * __ptr64,_NET_BUFFER_LIST * __ptr64,unsigned long,unsigned long,unsigned long)>+0x8b
ffff960628b352d0 fffff80115d962ea : ffffcf8803d87b50 0000000000000000 0000000000000000 ffffcf8807178ad0 : NDIS!ndisInvokeIterativeDatapath<2,void __cdecl(void * __ptr64,_NET_BUFFER_LIST * __ptr64,unsigned long,unsigned long,unsigned long)>+0xe2
ffff960628b35340 fffff80115d73766 : 0000000000009f75 0000000000000000 0000000000000000 0000000000000000 : NDIS!ndisInvokeNextReceiveHandler+0xa6
ffff960628b353a0 fffff80138fdc94d : ffffcf87fe641000 ffffcf87fe641010 ffff960628b35530 0000000000000002 : NDIS!NdisMIndicateReceiveNetBufferLists+0x116
ffff960628b35430 fffff80138fdbf9e : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : wdiwifi!CPort::IndicateFrames+0xad
ffff960628b35570 fffff80138fdb417 : fffff80113a14980 ffff960628b35708 fffff801390bf040 0000000000000000 : wdiwifi!CRxMgr::RxProcessAndIndicateNblChain+0x41e
ffff960628b356a0 fffff80138fdb268 : ffffcf8807178ad0 ffffcf8700000003 0000000000000002 000000000000002c : wdiwifi!CRxMgr::RxInOrderDataInd+0x127
ffff960628b35750 fffff80138aac4be : ffffcf87fe9152c0 ffffcf87fef45590 0000000000000001 ffffcf87fb0e0080 : wdiwifi!AdapterRxInorderDataInd+0x88
ffff960628b357a0 ffffcf87fe9152c0 : ffffcf87fef45590 0000000000000001 ffffcf87fb0e0080 ffff960628b357e8 : Netwtw10+0x4c4be
ffff960628b357a8 ffffcf87fef45590 : 0000000000000001 ffffcf87fb0e0080 ffff960628b357e8 ffff960628b357e0 : 0xffffcf87fe9152c0 ffff960628b357b0 0000000000000001 : ffffcf87fb0e0080 ffff960628b357e8 ffff960628b357e0 0000000000000002 : 0xffffcf87fef45590
ffff960628b357b8 ffffcf87fb0e0080 : ffff960628b357e8 ffff960628b357e0 0000000000000002 ffff960628b357e8 : 0x1
ffff960628b357c0 ffff960628b357e8 : ffff960628b357e0 0000000000000002 ffff960628b357e8 000001a800000000 : 0xffffcf87fb0e0080 ffff960628b357c8 ffff960628b357e0 : 0000000000000002 ffff960628b357e8 000001a800000000 ffff9606ffffffff : 0xffff960628b357e8
ffff960628b357d0 0000000000000002 : ffff960628b357e8 000001a800000000 ffff9606ffffffff ffffd58458672290 : 0xffff960628b357e0 ffff960628b357d8 ffff960628b357e8 : 000001a800000000 ffff9606ffffffff ffffd58458672290 ffff960628b357e4 : 0x2 ffff960628b357e0 000001a800000000 : ffff9606ffffffff ffffd58458672290 ffff960628b357e4 0000000000000004 : 0xffff960628b357e8
ffff960628b357e8 ffff9606ffffffff : ffffd58458672290 ffff960628b357e4 0000000000000004 fffff80138aac63e : 0x000001a800000000 ffff960628b357f0 ffffd58458672290 : ffff960628b357e4 0000000000000004 fffff80138aac63e ffffcf87fb0e0080 : 0xffff9606ffffffff
ffff960628b357f8 ffff960628b357e4 : 0000000000000004 fffff80138aac63e ffffcf87fb0e0080 fffff80113c884e5 : 0xffffd58458672290 ffff960628b35800 0000000000000004 : fffff80138aac63e ffffcf87fb0e0080 fffff80113c884e5 ffffcf87fb0e0080 : 0xffff960628b357e4
ffff960628b35808 fffff80138aac63e : ffffcf87fb0e0080 fffff80113c884e5 ffffcf87fb0e0080 0000000000000000 : 0x4
ffff960628b35810 ffffcf87fb0e0080 : fffff80113c884e5 ffffcf87fb0e0080 0000000000000000 0000000000000000 : Netwtw10+0x4c63e
ffff960628b35818 fffff80113c884e5 : ffffcf87fb0e0080 0000000000000000 0000000000000000 ffffcf87f875e080 : 0xffffcf87fb0e0080 ffff960628b35820 fffff80113c2f860 : 0000000000000000 0000000000000000 ffffcf87fe953c10 ffffcf87fb0a4d80 : nt!PsImpersonateContainerOfThread+0x185 ffff960628b35890 fffff80113c34f85 : ffffcf87f74e5c50 ffffcf87f875e080 ffff960628b35a00 0000000000000000 : nt!IopProcessWorkItem+0x100 ffff960628b35900 fffff80113d07317 : ffffcf87f875e080 00000000000000da ffffcf87f875e080 fffff80113c34e30 : nt!ExpWorkerThread+0x155 ffff960628b35af0 fffff80113e1bc54 : fffff8010f116180 ffffcf87f875e080 fffff80113d072c0 0000000000000000 : nt!PspSystemThreadStartup+0x57 ffff960628b35b40 0000000000000000 : ffff960628b36000 ffff960628b2f000 0000000000000000 00000000`00000000 : nt!KiStartSystemThread+0x34
SYMBOL_NAME: fwpkclnt!FwppInjectComplete+af
MODULE_NAME: fwpkclnt
IMAGE_NAME: fwpkclnt.sys
IMAGE_VERSION: 10.0.22621.3061
STACK_COMMAND: .cxr; .ecxr ; kb
BUCKET_ID_FUNC_OFFSET: af
FAILURE_BUCKET_ID: AV_fwpkclnt!FwppInjectComplete
OS_VERSION: 10.0.22621.1
BUILDLAB_STR: ni_release
OSPLATFORM_TYPE: x64
OSNAME: Windows 10
FAILURE_ID_HASH: {0cd1ec7c-9b34-fb98-d3bd-b9ce089ba9de}
Followup: MachineOwner
Thanks.
Ash