Our java application becomes unresponsive and the log file freezes , there is no details captured in log files when the issue occurs . We restart the windows services to bring the application back. We havent found a pattern but the issue occurs once a month or once in two months . We have the application hosted on tomcat8.5 in two servers and the issue occurs in both servers. I have memory dump but not sure how to read/understand it . can you please help me?
0:000> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
KEY_VALUES_STRING: 1
Key : Analysis.CPU.mSec
Value: 8328
Key : Analysis.DebugAnalysisManager
Value: Create
Key : Analysis.Elapsed.mSec
Value: 10212
Key : Analysis.Init.CPU.mSec
Value: 32030
Key : Analysis.Init.Elapsed.mSec
Value: 904104
Key : Analysis.Memory.CommitPeak.Mb
Value: 95
Key : Timeline.OS.Boot.DeltaSec
Value: 1778847
Key : Timeline.Process.Start.DeltaSec
Value: 1778834
Key : WER.OS.Branch
Value: rs1\_release
Key : WER.OS.Timestamp
Value: 2022-04-29T17:32:00Z
Key : WER.OS.Version
Value: 10.0.14393.5125
Key : WER.Process.Version
Value: 1.1.0.0
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
EXCEPTION_RECORD: (.exr -1)
ExceptionAddress: 0000000000000000
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 0
FAULTING_THREAD: 00000c10
PROCESS_NAME: tomcat8.exe
ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION} Breakpoint A breakpoint has been reached.
EXCEPTION_CODE_STR: 80000003
STACK_TEXT:
0000000000aff3d8 00007ffe36cc6d1f : 0000000000aff4c0 0000000000000000 0000000000000000 0000000000aff4e9 : ntdll!NtWaitForSingleObject+0x14
0000000000aff3e0 00007ffe39b409c8 : 00000000ffffffff 00007ffe39b6e462 0000000000000000 000000000000019c : KERNELBASE!WaitForSingleObjectEx+0x8f
0000000000aff480 00007ffe39b403b0 : 0000000000000000 00000000004f40fc 0000000000000000 00000000000001b8 : sechost!ScSendResponseReceiveControls+0x138
0000000000aff5c0 00007ffe39b3f344 : 0000000000aff718 0000000000000000 0000000000000000 0000000000000001 : sechost!ScDispatcherLoop+0x138
0000000000aff6d0 00007ff6a28df059 : 0000000000000000 0000000000b770d0 00007ffe3993f060 00007ffe3993f060 : sechost!StartServiceCtrlDispatcherW+0x54
0000000000aff710 00007ff6a28df49b : 0000000000d409f8 00007ff6a28e33a0 00000000000007e7 0000000000000006 : tomcat8+0xf059
0000000000aff780 00007ff6a28df989 : 0000000000000000 0000000000000000 0000000000000000 00007ff6a28e05d8 : tomcat8+0xf49b
0000000000aff7f0 00007ffe395784d4 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : tomcat8+0xf989
0000000000aff830 00007ffe39be1791 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : kernel32!BaseThreadInitThunk+0x14
0000000000aff860 0000000000000000 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : ntdll!RtlUserThreadStart+0x21
STACK_COMMAND: ~0s; .ecxr ; kb
SYMBOL_NAME: kernelbase!WaitForSingleObjectEx+8f
MODULE_NAME: KERNELBASE
IMAGE_NAME: KERNELBASE.dll
FAILURE_BUCKET_ID: BREAKPOINT_80000003_KERNELBASE.dll!WaitForSingleObjectEx
OS_VERSION: 10.0.14393.5125
BUILDLAB_STR: rs1_release
OSPLATFORM_TYPE: x64
OSNAME: Windows 10
IMAGE_VERSION: 10.0.14393.5850
FAILURE_ID_HASH: {6e08debd-ee31-ce56-32ef-878b2e67a5f2}
Followup: MachineOwner
0:000> !peb
PEB at 0000000000972000
InheritedAddressSpace: No
ReadImageFileExecOptions: No
BeingDebugged: No
ImageBaseAddress: 00007ff6a28d0000
NtGlobalFlag: 0
NtGlobalFlag2: 0
Ldr 00007ffe39ce03c0
Ldr.Initialized: Yes
Ldr.InInitializationOrderModuleList: 0000000000b62350 . 000000001d9fef90
Ldr.InLoadOrderModuleList: 0000000000b624c0 . 000000001d9fef70
Ldr.InMemoryOrderModuleList: 0000000000b624d0 . 000000001d9fef80
Base TimeStamp Module
7ff6a28d0000 5a022cf5 Nov 07 22:00:21 2017 E:\Tomcat\TomCat-8.5.42\bin\tomcat8.exe
7ffe39b90000 633688f6 Sep 30 07:13:10 2022 C:\Windows\SYSTEM32\ntdll.dll
7ffe39570000 640993b0 Mar 09 08:07:12 2023 C:\Windows\System32\KERNEL32.DLL
7ffe36ca0000 64253b4c Mar 30 08:33:32 2023 C:\Windows\System32\KERNELBASE.dll
52b30000 629aca3f Jun 04 03:58:07 2022 C:\ProgramData\Symantec\Symantec Endpoint Protection\14.3.8268.5000.105\Data\Sysfer\x64\sysfer.dll
7ffe39130000 64253b93 Mar 30 08:34:43 2023 C:\Windows\System32\ADVAPI32.dll
7ffe398b0000 5b7e2c5b Aug 23 04:39:07 2018 C:\Windows\System32\msvcrt.dll
7ffe39b30000 64253bf2 Mar 30 08:36:18 2023 C:\Windows\System32\sechost.dll
7ffe39000000 645467ab May 05 03:19:23 2023 C:\Windows\System32\RPCRT4.dll
7ffe37340000 645469d1 May 05 03:28:33 2023 C:\Windows\System32\SHELL32.dll
7ffe36ee0000 57899b3c Jul 16 03:26:04 2016 C:\Windows\System32\cfgmgr32.dll
7ffe365c0000 6454697c May 05 03:27:08 2023 C:\Windows\System32\windows.storage.dll
7ffe392a0000 64253b58 Mar 30 08:33:44 2023 C:\Windows\System32\combase.dll
7ffe361f0000 5e9140a1 Apr 11 04:59:29 2020 C:\Windows\System32\ucrtbase.dll
7ffe36350000 61808931 Nov 02 00:41:21 2021 C:\Windows\System32\bcryptPrimitives.dll
7ffe35fd0000 57899a1f Jul 16 03:21:19 2016 C:\Windows\System32\powrprof.dll
7ffe38850000 6336894b Sep 30 07:14:35 2022 C:\Windows\System32\shlwapi.dll
7ffe39730000 5ff78dea Jan 07 22:40:42 2021 C:\Windows\System32\GDI32.dll
7ffe36060000 64253c8f Mar 30 08:38:55 2023 C:\Windows\System32\gdi32full.dll
7ffe38e80000 5ff78d73 Jan 07 22:38:43 2021 C:\Windows\System32\USER32.dll
7ffe36ec0000 57a05800 Aug 02 09:21:20 2016 C:\Windows\System32\win32u.dll
7ffe36050000 5b1a18a2 Jun 08 06:48:18 2018 C:\Windows\System32\kernel.appcore.dll
7ffe37080000 6247cac1 Apr 02 05:02:09 2022 C:\Windows\System32\shcore.dll
7ffe36030000 57899b70 Jul 16 03:26:56 2016 C:\Windows\System32\profapi.dll
52040000 63510e58 Oct 20 10:01:12 2022 C:\\_sst\resources\Java\Java8\jre\bin\server\jvm.dll
7ffe39120000 57899a03 Jul 16 03:20:51 2016 C:\Windows\System32\PSAPI.DLL
7ffe2aee0000 57899b57 Jul 16 03:26:31 2016 C:\Windows\SYSTEM32\WSOCK32.dll
7ffe396c0000 5d786d1a Sep 11 04:42:18 2019 C:\Windows\System32\WS2\_32.dll
7ffe34440000 57899ae3 Jul 16 03:24:35 2016 C:\Windows\SYSTEM32\WINMM.dll
7ffe2bf90000 57899b7f Jul 16 03:27:11 2016 C:\Windows\SYSTEM32\VERSION.dll
7ffe2cd50000 577e0cc7 Jul 07 09:03:19 2016 C:\Windows\SYSTEM32\MSVCR120.dll
7ffe34310000 57899ade Jul 16 03:24:30 2016 C:\Windows\SYSTEM32\WINMMBASE.dll
7ffe29210000 63511022 Oct 20 10:08:50 2022 C:\\_sst\resources\Java\Java8\jre\bin\verify.dll
7ffe24dc0000 63511024 Oct 20 10:08:52 2022 C:\\_sst\resources\Java\Java8\jre\bin\java.dll
7ffe25720000 63511025 Oct 20 10:08:53 2022 C:\\_sst\resources\Java\Java8\jre\bin\zip.dll
7ffe22a50000 63511026 Oct 20 10:08:54 2022 C:\\_sst\resources\Java\Java8\jre\bin\management.dll
7ffe22030000 63511025 Oct 20 10:08:53 2022 C:\\_sst\resources\Java\Java8\jre\bin\net.dll
7ffe358f0000 5e91421e Apr 11 05:05:50 2020 C:\Windows\system32\mswsock.dll
7ffe22000000 63511027 Oct 20 10:08:55 2022 C:\\_sst\resources\Java\Java8\jre\bin\nio.dll
180000000 5c4b4c4d Jan 25 17:50:05 2019 E:\Tomcat\TomCat-8.5.42\bin\tcnative-1.dll
7ffe363c0000 6336893a Sep 30 07:14:18 2022 C:\Windows\System32\CRYPT32.dll
7ffe36020000 63647bb2 Nov 04 02:40:50 2022 C:\Windows\System32\MSASN1.dll
7ffe35f10000 63b793a8 Jan 06 03:21:12 2023 C:\Windows\SYSTEM32\bcrypt.dll
7ffe35aa0000 5ccd166d May 04 05:34:53 2019 C:\Windows\SYSTEM32\CRYPTSP.dll
7ffe354b0000 60baff23 Jun 05 05:35:47 2021 C:\Windows\system32\rsaenh.dll
7ffe35ac0000 57899bc7 Jul 16 03:28:23 2016 C:\Windows\SYSTEM32\CRYPTBASE.dll
7ffe21b30000 63511023 Oct 20 10:08:51 2022 C:\\_sst\resources\Java\Java8\jre\bin\sunec.dll
7ffe21b20000 63511023 Oct 20 10:08:51 2022 C:\\_sst\resources\Java\Java8\jre\bin\sunmscapi.dll
7ffe35b80000 6409915d Mar 09 07:57:17 2023 C:\Windows\SYSTEM32\ncrypt.dll
7ffe35b40000 57899b58 Jul 16 03:26:32 2016 C:\Windows\SYSTEM32\NTASN1.dll
7ffe35720000 6104d618 Jul 31 05:48:24 2021 C:\Windows\SYSTEM32\USERENV.dll
7ffe35620000 5b1f16bd Jun 12 01:41:33 2018 C:\Windows\SYSTEM32\IPHLPAPI.DLL
7ffe39770000 5d93bb44 Oct 01 21:47:00 2019 C:\Windows\System32\NSI.dll
7ffe33470000 63368ad2 Sep 30 07:21:06 2022 C:\Windows\SYSTEM32\dhcpcsvc6.DLL
7ffe33340000 63368a4c Sep 30 07:18:52 2022 C:\Windows\SYSTEM32\dhcpcsvc.DLL
7ffe2dd40000 57899b54 Jul 16 03:26:28 2016 C:\Windows\system32\napinsp.dll
7ffe35670000 606eb110 Apr 08 08:30:24 2021 C:\Windows\SYSTEM32\DNSAPI.dll
7ffe2de90000 57899b79 Jul 16 03:27:05 2016 C:\Windows\System32\winrnr.dll
7ffe32590000 628071c7 May 15 04:21:43 2022 C:\Windows\system32\NLAapi.dll
7ffe30800000 57899bae Jul 16 03:27:58 2016 C:\Windows\System32\rasadhlp.dll
7ffe335c0000 57899809 Jul 16 03:12:25 2016 C:\Windows\System32\fwpuclnt.dll
7ffe288e0000 63511025 Oct 20 10:08:53 2022 C:\\_sst\resources\Java\Java8\jre\bin\awt.dll
7ffe391e0000 6087963c Apr 27 05:42:36 2021 C:\Windows\System32\OLEAUT32.dll
7ffe36f30000 5ce36bd0 May 21 04:09:04 2019 C:\Windows\System32\msvcp\_win.dll
7ffe1f500000 577e0cd4 Jul 07 09:03:32 2016 C:\Windows\SYSTEM32\MSVCP120.dll
7ffe34340000 606eb2c8 Apr 08 08:37:44 2021 C:\Windows\SYSTEM32\apphelp.dll
7ffe334a0000 5ff78da3 Jan 07 22:39:31 2021 C:\Windows\system32\DWMAPI.DLL
7ffe28830000 63510d7e Oct 20 09:57:34 2022 C:\\_sst\resources\Java\Java8\jre\bin\freetype.dll
7ffe28750000 63511028 Oct 20 10:08:56 2022 C:\\_sst\resources\Java\Java8\jre\bin\fontmanager.dll
7ffe2db50000 63511027 Oct 20 10:08:55 2022 C:\\_sst\resources\Java\Java8\jre\bin\lcms.dll
7ffe0c400000 57899b28 Jul 16 03:25:44 2016 C:\Windows\system32\opengl32.dll
7ffe2d7d0000 57899b88 Jul 16 03:27:20 2016 C:\Windows\SYSTEM32\GLU32.dll
7ffe0a780000 58ba5e25 Mar 04 06:26:45 2017 C:\Windows\SYSTEM32\DDRAW.dll
7ffe2eae0000 57899c03 Jul 16 03:29:23 2016 C:\Windows\SYSTEM32\DCIMAN32.dll
7ffe16540000 5ff78d21 Jan 07 22:37:21 2021 C:\Windows\WinSxS\amd64\_microsoft.windows.common-controls\_6595b64144ccf1df\_6.0.14393.4169\_none\_7de0bbf28341b1f2\COMCTL32.dll
7ffe371a0000 64546b62 May 05 03:35:14 2023 C:\Windows\System32\ole32.dll
SubSystemData: 0000000000000000
ProcessHeap: 0000000000b60000
ProcessParameters: 0000000000b61b00
CurrentDirectory: 'E:\Tomcat\TomCat-8.5.42\'
WindowTitle: 'E:\Tomcat\TomCat-8.5.42\bin\tomcat8.exe'
ImageFile: 'E:\Tomcat\TomCat-8.5.42\bin\tomcat8.exe'
CommandLine: 'E:\Tomcat\TomCat-8.5.42\bin\tomcat8.exe //RS//Tomcat8'
DllPath: '< Name not readable >'
Environment: 0000000000b60e30
ALLUSERSPROFILE=C:\ProgramData
APPDATA=C:\Users\SVC\_SSOnline\_Live\AppData\Roaming
CATALINA\_BASE=E:\Tomcat\TomCat-8.5.42
CATALINA\_HOME=E:\Tomcat\TomCat-8.5.42
CommonProgramFiles=C:\Program Files\Common Files
CommonProgramFiles(x86)=C:\Program Files (x86)\Common Files
CommonProgramW6432=C:\Program Files\Common Files
ComSpec=C:\Windows\system32\cmd.exe
JAVA\_HOME=C:\\_sst\resources\Java\Java8
NUMBER\_OF\_PROCESSORS=4
OS=Windows\_NT
Path=C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Users\SVC\_SSOnline\_Live\AppData\Local\Microsoft\WindowsApps
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PROCESSOR\_ARCHITECTURE=AMD64
PROCESSOR\_IDENTIFIER=Intel64 Family 6 Model 63 Stepping 0, GenuineIntel
PROCESSOR\_LEVEL=6
PROCESSOR\_REVISION=3f00
ProgramData=C:\ProgramData
ProgramFiles=C:\Program Files
ProgramFiles(x86)=C:\Program Files (x86)
ProgramW6432=C:\Program Files
PSModulePath=%ProgramFiles%\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules;E:\Program Files\Microsoft Monitoring Agent\Agent\PowerShell\;C:\Program Files\Microsoft Monitoring Agent\Agent\PowerShell\
PUBLIC=C:\Users\Public
snow\_agent=C:\Program Files\Snow Software\Inventory\Agent
SystemDrive=C:
SystemRoot=C:\Windows
TEMP=C:\Users\SVC\_SS~1\AppData\Local\Temp
TMP=C:\Users\SVC\_SS~1\AppData\Local\Temp
UATDATA=C:\Windows\CCM\UATData\D9F8C395-CAB8-491d-B8AC-179A1FE1BE77
windir=C:\Windows
0:000> !heap -stat
_HEAP 0000000000b60000
Segments 0000000b
Reserved bytes 0000000006ed5000
Committed bytes 0000000005d58000
VirtAllocBlocks 0000000f
VirtAlloc bytes 000000013fecb110
_HEAP 00000000010c0000
Segments 00000002
Reserved bytes 000000000010e000
Committed bytes 0000000000041000
VirtAllocBlocks 00000000
VirtAlloc bytes 0000000000000000
_HEAP 0000000000d40000
Segments 00000001
Reserved bytes 000000000001f000
Committed bytes 0000000000010000
VirtAllocBlocks 00000000
VirtAlloc bytes 0000000000000000
_HEAP 0000000000790000
Segments 00000001
Reserved bytes 0000000000010000
Committed bytes 0000000000001000
VirtAllocBlocks 00000000
VirtAlloc bytes 0000000000000000
0:000> !heap -stat -s
_HEAP 0000000000b60000
Segments 0000000b
Reserved bytes 0000000006ed5000
Committed bytes 0000000005d58000
VirtAllocBlocks 0000000f
VirtAlloc bytes 000000013fecb110
_HEAP 00000000010c0000
Segments 00000002
Reserved bytes 000000000010e000
Committed bytes 0000000000041000
VirtAllocBlocks 00000000
VirtAlloc bytes 0000000000000000
_HEAP 0000000000d40000
Segments 00000001
Reserved bytes 000000000001f000
Committed bytes 0000000000010000
VirtAllocBlocks 00000000
VirtAlloc bytes 0000000000000000
_HEAP 0000000000790000
Segments 00000001
Reserved bytes 0000000000010000
Committed bytes 0000000000001000
VirtAllocBlocks 00000000
VirtAlloc bytes 0000000000000000
0:000> !heap -stat s
_HEAP 0000000000b60000
Segments 0000000b
Reserved bytes 0000000006ed5000
Committed bytes 0000000005d58000
VirtAllocBlocks 0000000f
VirtAlloc bytes 000000013fecb110
_HEAP 00000000010c0000
Segments 00000002
Reserved bytes 000000000010e000
Committed bytes 0000000000041000
VirtAllocBlocks 00000000
VirtAlloc bytes 0000000000000000
_HEAP 0000000000d40000
Segments 00000001
Reserved bytes 000000000001f000
Committed bytes 0000000000010000
VirtAllocBlocks 00000000
VirtAlloc bytes 0000000000000000
_HEAP 0000000000790000
Segments 00000001
Reserved bytes 0000000000010000
Committed bytes 0000000000001000
VirtAllocBlocks 00000000
VirtAlloc bytes 0000000000000000
0:000> ~
. 0 Id: c0c.c10 Suspend: 0 Teb: 00000000`00973000 Unfrozen
1 Id: c0c.e54 Suspend: 0 Teb: 00000000`0097b000 Unfrozen
2 Id: c0c.e58 Suspend: 0 Teb: 00000000`0097d000 Unfrozen
3 Id: c0c.eb8 Suspend: 0 Teb: 00000000`0097f000 Unfrozen
4 Id: c0c.ee8 Suspend: 0 Teb: 00000000`00981000 Unfrozen
5 Id: c0c.eec Suspend: 0 Teb: 00000000`00983000 Unfrozen
6 Id: c0c.ef0 Suspend: 0 Teb: 00000000`00985000 Unfrozen
7 Id: c0c.ef4 Suspend: 0 Teb: 00000000`00987000 Unfrozen
8 Id: c0c.f7c Suspend: 0 Teb: 00000000`00989000 Unfrozen
9 Id: c0c.fa8 Suspend: 0 Teb: 00000000`0098b000 Unfrozen
10 Id: c0c.fac Suspend: 0 Teb: 00000000`0098d000 Unfrozen
11 Id: c0c.fb8 Suspend: 0 Teb: 00000000`0098f000 Unfrozen
12 Id: c0c.fbc Suspend: 0 Teb: 00000000`00991000 Unfrozen
13 Id: c0c.fc0 Suspend: 0 Teb: 00000000`00993000 Unfrozen
14 Id: c0c.fc4 Suspend: 0 Teb: 00000000`00995000 Unfrozen
15 Id: c0c.fc8 Suspend: 0 Teb: 00000000`00997000 Unfrozen
16 Id: c0c.fcc Suspend: 0 Teb: 00000000`00999000 Unfrozen
17 Id: c0c.fd0 Suspend: 0 Teb: 00000000`0099b000 Unfrozen
18 Id: c0c.fec Suspend: 0 Teb: 00000000`0099d000 Unfrozen
19 Id: c0c.f04 Suspend: 0 Teb: 00000000`0099f000 Unfrozen
20 Id: c0c.101c Suspend: 0 Teb: 00000000`009a1000 Unfrozen
21 Id: c0c.123c Suspend: 0 Teb: 00000000`009a7000 Unfrozen
22 Id: c0c.1248 Suspend: 0 Teb: 00000000`009ab000 Unfrozen
23 Id: c0c.1268 Suspend: 0 Teb: 00000000`009ad000 Unfrozen
24 Id: c0c.1360 Suspend: 0 Teb: 00000000`009af000 Unfrozen
25 Id: c0c.1280 Suspend: 0 Teb: 00000000`009b3000 Unfrozen
26 Id: c0c.12b0 Suspend: 0 Teb: 00000000`009b5000 Unfrozen
27 Id: c0c.12bc Suspend: 0 Teb: 00000000`009b7000 Unfrozen
28 Id: c0c.12c0 Suspend: 0 Teb: 00000000`009b9000 Unfrozen
29 Id: c0c.1264 Suspend: 0 Teb: 00000000`009bb000 Unfrozen
30 Id: c0c.1260 Suspend: 0 Teb: 00000000`009bd000 Unfrozen
31 Id: c0c.12f4 Suspend: 0 Teb: 00000000`009bf000 Unfrozen
32 Id: c0c.12e4 Suspend: 0 Teb: 00000000`009c1000 Unfrozen
33 Id: c0c.12f0 Suspend: 0 Teb: 00000000`009c3000 Unfrozen
34 Id: c0c.12d4 Suspend: 0 Teb: 00000000`009c5000 Unfrozen
35 Id: c0c.12d0 Suspend: 0 Teb: 00000000`009c7000 Unfrozen
36 Id: c0c.12e0 Suspend: 0 Teb: 00000000`009c9000 Unfrozen
37 Id: c0c.12c8 Suspend: 0 Teb: 00000000`009cb000 Unfrozen
38 Id: c0c.12dc Suspend: 0 Teb: 00000000`009cd000 Unfrozen
39 Id: c0c.12ec Suspend: 0 Teb: 00000000`009cf000 Unfrozen
40 Id: c0c.23e4 Suspend: 0 Teb: 00000000`009d7000 Unfrozen
41 Id: c0c.2984 Suspend: 0 Teb: 00000000`009e5000 Unfrozen