BSOD Windows Server 2016 Remote Desktop Session Host VM
Hello guys. I'm trying to figure out an issue with my remote desktop session host. This has about 30 people on it at the moment. Just a brief background about the machines hardware. Its a dell PowerEdge R630. CPU is dual Intel Xeon CPU E5-2670 v3 2.30GHz. Dell Toshiba 800 GB drives. H310 Raid controller. I have an identical machine to this one set up in a HA cluster via the Hypervisor known as Verge.io. I have changed which node the machine runs on and regardless it crashes. I feel like because of that we can rule out hardware being the issue but maybe not. We went a full 120 days without a single crash on this machine then March updates roll out. It seems like ever since then this machine is just not stable, so what did I do? I built a new RDSH since we are using FSLogix. The new RDSH also crashes but less frequent.
Every Crash dump is slightly different. Different blue screen error. They have cannon and Lexmark printers with 2 oki printer in the office. All these drivers are most recent. They have an IBM A+ that they use and a lot of printing goes through a transform that is located on a 2012R2 domain controller before it goes to the printer. I have done test prints on all of the machines and nothing crashes the server when I print. I feel like this is almost totally random.
The last 2 crashes were on the 29th and the 7th. Both crashes happened almost at the same time about 4 minutes apart. I don't know if that's just a coincidence or not so I checked to see if anything was running in task scheduler around that time. Nope.
The applications on the machine are as follows.
IBM I Access emulator 64bit
Adobe Acrobat Reader DC
Canon Genric Plus PCL6 Printer driver
Foxit Reader
Chrome
Java 8 Update 281 64bit
Kaseya Agent
Edge
Microsoft FSLogix
Microsoft Teams
Office standard 2016
Mozilla Firefox
Okie Network Extension
Also here is the latest 2 crash dump. I am so baffled by what is causing this. The customer is very unhappy and I completely understand. Event logs as far as I can tell aren't showing anything to point to the crashing.
Any help is much appreciated.
SYSTEM_SERVICE_EXCEPTION (3b)
An exception happened while executing a system service routine.
Arguments:
Arg1: 00000000c0000005, Exception code that caused the bugcheck
Arg2: ffff81ced21c7aac, Address of the instruction which caused the bugcheck
Arg3: ffffbd0153ece100, Address of the context record for the exception that caused the bugcheck
Arg4: 0000000000000000, zero.
Debugging Details:
KEY_VALUES_STRING: 1
Key : Analysis.CPU.Sec
Value: 2
Key : Analysis.DebugAnalysisProvider.CPP
Value: Create: 8007007e on ANDREW
Key : Analysis.DebugData
Value: CreateObject
Key : Analysis.DebugModel
Value: CreateObject
Key : Analysis.Elapsed.Sec
Value: 4
Key : Analysis.Memory.CommitPeak.Mb
Value: 84
Key : Analysis.System
Value: CreateObject
BUGCHECK_CODE: 3b
BUGCHECK_P1: c0000005
BUGCHECK_P2: ffff81ced21c7aac
BUGCHECK_P3: ffffbd0153ece100
BUGCHECK_P4: 0
CONTEXT: ffffbd0153ece100 -- (.cxr 0xffffbd0153ece100)
rax=0000000000000000 rbx=ffff8198c0053000 rcx=0000000000000000
rdx=ffffbd0153ece501 rsi=ffff8198c3b2d010 rdi=ffff8198c0053458
rip=ffff81ced21c7aac rsp=ffffbd0153eceaf0 rbp=ffffbd0153ecec40
r8=0000000000000000 r9=0000000000000000 r10=ffffbd0153ecf5ec
r11=ffffbd0153ecf5e0 r12=ffff8198c0053020 r13=ffff8198c4a208f0
r14=0000000000000008 r15=0000000000000000
iopl=0 nv up ei ng nz na pe nc
cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00010282
win32kbase!FreeThreadBufferWithTag+0x1c:
ffff81ced21c7aac 48395908 cmp qword ptr [rcx+8],rbx ds:002b:00000000
00000008=????????????????
Resetting default scope
CUSTOMER_CRASH_COUNT: 1
PROCESS_NAME: EXCEL.EXE
STACK_TEXT:
ffffbd0153eceaf0 ffff81ce
d1e77232 : 0000000000000438 ffff8198
c3b2d010 000000000000005a 00000000
00000002 : win32kbase!FreeThreadBufferWithTag+0x1c
ffffbd0153eceb20 ffff81ce
d1e752a2 : 0000000000000000 00000000
00000000 0000000000000008 ffffbd01
00000000 : win32kfull!EngTextOut+0x642
ffffbd0153ecf040 ffff81ce
d1e7e6d4 : 0000000000000000 00000000
00000008 0000000000001000 00000000
00000000 : win32kfull!GreExtTextOutWLocked+0x1a92
ffffbd0153ecf7e0 ffff81ce
d1e7e552 : 0000000000000000 ffff8198
c0003000 ffff8198c00030f0 00000000
00000001 : win32kfull!GreExtTextOutWInternal+0xec
ffffbd0153ecf8b0 fffff801
a3b84e03 : 00000000d404106e fffff801
00000001 0000000000000000 00000000
00000000 : win32kfull!NtGdiExtTextOutW+0x2e2
ffffbd0153ecfa90 00007ffe
66e217a4 : 0000000000000000 00000000
00000000 0000000000000000 00000000
00000000 : nt!KiSystemServiceCopyEnd+0x13
000000305f99ac28 00000000
00000000 : 0000000000000000 00000000
00000000 0000000000000000 00000000
00000000 : 0x00007ffe`66e217a4
SYMBOL_NAME: win32kbase!FreeThreadBufferWithTag+1c
MODULE_NAME: win32kbase
IMAGE_NAME: win32kbase.sys
IMAGE_VERSION: 10.0.14393.4288
STACK_COMMAND: .cxr 0xffffbd0153ece100 ; kb
BUCKET_ID_FUNC_OFFSET: 1c
FAILURE_BUCKET_ID: 0x3B_c0000005_win32kbase!FreeThreadBufferWithTag
OS_VERSION: 10.0.14393.4288
BUILDLAB_STR: rs1_release_inmarket
OSPLATFORM_TYPE: x64
OSNAME: Windows 10
FAILURE_ID_HASH: {eb2e4fac-aec2-6951-6fd4-7bef5564d9af}
Followup: MachineOwner
SYSTEM_SERVICE_EXCEPTION (3b)
An exception happened while executing a system service routine.
Arguments:
Arg1: 00000000c0000005, Exception code that caused the bugcheck
Arg2: fffff803584a0bfc, Address of the instruction which caused the bugcheck
Arg3: ffffc8820df43570, Address of the context record for the exception that caused the bugcheck
Arg4: 0000000000000000, zero.
Debugging Details:
KEY_VALUES_STRING: 1
Key : Analysis.CPU.Sec
Value: 3
Key : Analysis.DebugAnalysisProvider.CPP
Value: Create: 8007007e on ANDREW
Key : Analysis.DebugData
Value: CreateObject
Key : Analysis.DebugModel
Value: CreateObject
Key : Analysis.Elapsed.Sec
Value: 3
Key : Analysis.Memory.CommitPeak.Mb
Value: 75
Key : Analysis.System
Value: CreateObject
BUGCHECK_CODE: 3b
BUGCHECK_P1: c0000005
BUGCHECK_P2: fffff803584a0bfc
BUGCHECK_P3: ffffc8820df43570
BUGCHECK_P4: 0
CONTEXT: ffffc8820df43570 -- (.cxr 0xffffc8820df43570)
rax=ffffc8820df440c8 rbx=ffff868337f60c10 rcx=0000000000000000
rdx=0000000000000001 rsi=0000000000000000 rdi=0000000000000000
rip=fffff803584a0bfc rsp=ffffc8820df43f60 rbp=ffffb78188420e40
r8=fffff801019296e8 r9=0000000000000001 r10=7ffff801019296e8
r11=7ffffffffffffffc r12=fffff801018700c4 r13=ffffb781aa0e2cd8
r14=fffff801018609a8 r15=fffff8010185d5b0
iopl=0 nv up ei pl nz ac pe cy
cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00010213
msrpc!NdrMesTypeDecode2+0x28c:
fffff803584a0bfc 488b39 mov rdi,qword ptr [rcx] ds:002b:00000000
00000000=????????????????
Resetting default scope
CUSTOMER_CRASH_COUNT: 1
PROCESS_NAME: Teams.exe
LOCK_ADDRESS: fffff80101921900 -- (!locks fffff80101921900)
Cannot get _ERESOURCE type
Resource @ nt!PiEngineLock (0xfffff80101921900) Available
1 total locks
PNP_TRIAGE_DATA:
Lock address : 0xfffff80101921900
Thread Count : 0
Thread address: 0x0000000000000000
Thread wait : 0x0
STACK_TEXT:
ffffc8820df43f60 fffff803
584a0d62 : 0000000000000000 fffff801
0185d510 0000000000000000 00000000
00000000 : msrpc!NdrMesTypeDecode2+0x28c
ffffc8820df44320 fffff801
01ac9e2d : ffffb781aa0e2cc0 ffffb781
88420e40 ffffb781aa0e2cd0 00000000
00000000 : msrpc!NdrMesTypeDecode3+0x112
ffffc8820df446f0 fffff801
01ac9b2a : 0000000000000000 ffffb781
aa0e2cc0 ffffb781ae8534b0 00000000
00000000 : nt!PiDqIrpQueryCreate+0x115
ffffc8820df447b0 fffff801
01ac9a08 : fffff8010185d3e0 ffffb781
88420e40 0000000000000001 ffffb781
94e01800 : nt!PiDqDispatch+0x9a
ffffc8820df447f0 fffff801
01a130c0 : ffffb781ae493e30 00000000
00000002 0000000000000001 ffffb781
00000000 : nt!PiDaDispatch+0x40
ffffc8820df44820 fffff801
01a1245c : ffffb78100000000 ffffb781
ae493e04 fffff780000002dc ffffc882
0df44b80 : nt!IopSynchronousServiceTail+0x1a0
ffffc8820df448e0 fffff801
01a116b6 : 0000000000000000 00000000
00000000 0000000000000000 00000000
00000000 : nt!IopXxxControlFile+0xd9c
ffffc8820df44a20 fffff801
01771e03 : 0000000000000694 fffff801
01a6713b 0000000000000000 fffff801
00000000 : nt!NtDeviceIoControlFile+0x56
ffffc8820df44a90 00007ffd
5b055ca4 : 0000000000000000 00000000
00000000 0000000000000000 00000000
00000000 : nt!KiSystemServiceCopyEnd+0x13
000000f1889fe978 00000000
00000000 : 0000000000000000 00000000
00000000 0000000000000000 00000000
00000000 : 0x00007ffd`5b055ca4
SYMBOL_NAME: msrpc!NdrMesTypeDecode2+28c
MODULE_NAME: msrpc
IMAGE_NAME: msrpc.sys
IMAGE_VERSION: 10.0.14393.4169
STACK_COMMAND: .cxr 0xffffc8820df43570 ; kb
BUCKET_ID_FUNC_OFFSET: 28c
FAILURE_BUCKET_ID: 0x3B_c0000005_msrpc!NdrMesTypeDecode2
OS_VERSION: 10.0.14393.4288
BUILDLAB_STR: rs1_release_inmarket
OSPLATFORM_TYPE: x64
OSNAME: Windows 10
FAILURE_ID_HASH: {f0652a30-c06f-372c-300c-5d6d93167908}
Followup: MachineOwner