iis crash dump analysis (Related Event 5011)

Question

Dear all
i'm dying. my iis system occurred crash dump with event 5011.
I attached the result of analyzing the dump file with Windbg.
Please anybody help me what is root cause of the problems.

------------------------------------------------------------------------------------------------------

Opened log file 'd:\logs.txt

Symbol Path validation summary
Response Time (ms) Location
Deferred SRVc:\symbolshttp://msdl.microsoft.com/download/symbols
0:280> .reload
................................................................
................................................................
................................................................
...........................................................
Loading unloaded module list
................................................................
0:280> !analyze -v
**

                 Exception Analysis                                   *  

ERROR: Symbol file could not be found. Defaulted to export symbols for crxf_pdf.dll -
ERROR: Symbol file could not be found. Defaulted to export symbols for crpe32.dll -
ERROR: Symbol file could not be found. Defaulted to export symbols for cslibu-3-0.dll -
ERROR: Symbol file could not be found. Defaulted to export symbols for u312frko.dll -
ERROR: Symbol file could not be found. Defaulted to export symbols for etc-1-0-12-6.dll -
ERROR: Symbol file could not be found. Defaulted to export symbols for cxlibw-5-0.dll -
ERROR: Symbol file could not be found. Defaulted to export symbols for secSSO.dll -
ERROR: Symbol file could not be found. Defaulted to export symbols for dtsagent.dll -
ERROR: Symbol file could not be found. Defaulted to export symbols for sacommlayer.dll -
ERROR: Symbol file could not be found. Defaulted to export symbols for rptcontrollers.dll -
ERROR: Symbol file could not be found. Defaulted to export symbols for clientdoc.dll -
ERROR: Symbol file could not be found. Defaulted to export symbols for cxlib-5-0.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for rptdefmodel.dll -
!ip2md 6223e460

Failed to request MethodData, not in JIT code range
GetUrlPageData2 (WinHttp) failed: 12002.

DUMP_CLASS: 2

DUMP_QUALIFIER: 400

CONTEXT: (.ecxr)
.ecxr
rax=0000000000000040 rbx=0000000000000412 rcx=0000000007c2ccd0
rdx=fffffffff83d3330 rsi=0000000000000000 rdi=0000000000000001
rip=000000006223e460 rsp=0000000c9afafa38 rbp=0000000041dd52b0
r8=fffffff3699d2100 r9=07ffffff9b4ce908 r10=0000000c9e25ab80
r11=0000000c9e25abd0 r12=0000000041dd52b0 r13=0000000000000001
r14=0000000c9e25abd0 r15=0000000007c2ccd0
iopl=0 nv up ei pl nz na pe nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010200
msvcr80!memcpy+0x2c0:
000000006223e460 4c8b4c0af8 mov r9,qword ptr [rdx+rcx-8] ds:fffffffffffffff8=????????????????
.cxr
Resetting default scope

FAULTING_IP:
msvcr80!memcpy+2c0 [F:\dd\vctools\crt_bld\SELF_64_AMD64\crt\src\AMD64\memcpy.asm @ 376]
00000000`6223e460 4c8b4c0af8 mov r9,qword ptr [rdx+rcx-8]

EXCEPTION_RECORD: (.exr -1)
.exr -1
ExceptionAddress: 000000006223e460 (msvcr80!memcpy+0x00000000000002c0)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000000
Parameter[1]: fffffffffffffff8
Attempt to read from address fffffffffffffff8

DEFAULT_BUCKET_ID: INVALID_POINTER_READ

PROCESS_NAME: w3wp.exe

ERROR_CODE: (NTSTATUS) 0xc0000005 - <Unable to get error code text>

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - <Unable to get error code text>

EXCEPTION_CODE_STR: c0000005

EXCEPTION_PARAMETER1: 0000000000000000

EXCEPTION_PARAMETER2: fffffffffffffff8

FOLLOWUP_IP:
msvcr80!memcpy+2c0 [F:\dd\vctools\crt_bld\SELF_64_AMD64\crt\src\AMD64\memcpy.asm @ 376]
00000000`6223e460 4c8b4c0af8 mov r9,qword ptr [rdx+rcx-8]

READ_ADDRESS: fffffffffffffff8

WATSON_BKT_PROCSTAMP: 5215df96

WATSON_BKT_PROCVER: 8.5.9600.16384

PROCESS_VER_PRODUCT: Internet Information Services

WATSON_BKT_MODULE: msvcr80.dll

WATSON_BKT_MODSTAMP: 520b0ac2

WATSON_BKT_MODOFFSET: 1e460

WATSON_BKT_MODVER: 8.0.50727.8428

BUILD_VERSION_STRING: 6.3.9600.19994 (winblue_ltsb_escrow.210331-1613)

MODLIST_WITH_TSCHKSUM_HASH: 022ba75515edd5b880b2478ee3e1bfa5d86a1f87

MODLIST_SHA1_HASH: 8bb64d6e8508dc802ea0c15f4c6da5636d78c168

NTGLOBALFLAG: 0

PROCESS_BAM_CURRENT_THROTTLED: 0

PROCESS_BAM_PREVIOUS_THROTTLED: 0

APPLICATION_VERIFIER_FLAGS: 0

PRODUCT_TYPE: 3

SUITE_MASK: 272

DUMP_FLAGS: 8000c07

DUMP_TYPE: 3

MISSING_CLR_SYMBOL: 0

ANALYSIS_SESSION_HOST: DESKTOP-V3P2R65

ANALYSIS_SESSION_TIME: 06-14-2021 10:52:53.0147

ANALYSIS_VERSION: 10.0.15063.468 amd64fre

MANAGED_CODE: 1

MANAGED_ENGINE_MODULE: clr

MANAGED_ANALYSIS_PROVIDER: SOS

THREAD_ATTRIBUTES:
OS_LOCALE: KOR

PROBLEM_CLASSES:

ID: [0n292]
Type: [@access _VIOLATION]
Class: Addendum
Scope: BUCKET_ID
Name: Omit
Data: Omit
PID: [Unspecified]
TID: [0x1710]
Frame: [0] : msvcr80!memcpy
ID: [0n264]
Type: [INVALID_POINTER_READ]
Class: Primary
Scope: DEFAULT_BUCKET_ID (Failure Bucket ID prefix)
BUCKET_ID
Name: Add
Data: Omit
PID: [Unspecified]
TID: [0x1710]
Frame: [0] : msvcr80!memcpy

BUGCHECK_STR: APPLICATION_FAULT_INVALID_POINTER_READ

PRIMARY_PROBLEM_CLASS: APPLICATION_FAULT

LAST_CONTROL_TRANSFER: from 00000000622a6a70 to 000000006223e460

STACK_TEXT:
0000000c9afafa38 00000000622a6a70 : 0000000000000004 0000000062226e97 0000000c915358b0 0000000c00000002 : msvcr80!memcpy+0x2c0
0000000c9afafa40 0000000041d6a6d4 : 0000000000000000 0000000000000001 00000000000001c2 0000000000000002 : msvcr80!memmove_s+0x80
0000000c9afafa80 0000000041d6bbdc : 0000000c9d20ebd0 0000000c9afafb20 0000000c9e25abd0 0000000000000000 : crxf_pdf!UXFInitializeW+0x1834
0000000c9afafb00 0000000037b18a60 : 0000000c9d20ebd0 0000000c00000412 0000000c9e1ca560 000000000042beae : crxf_pdf!UXFGetExportFormatsExW+0x10c
0000000c9afafb50 00000000004b6e95 : 0000000c9d20ebd0 0000000000000000 00000000004b37c0 0000000000000000 : crpe32!PEGetNthTableQualifier+0x16c2d0
0000000c9afafba0 00000000004b3898 : 0000000c903f63d0 0000000000000537 0000000000000537 00007ffe00000002 : cslibu_3_0!CSLib300::CSThreadSafeDLL::UninitForThread+0x965
0000000c9afafbd0 00007ffef2e424fd : 0000000000000000 0000000000000000 0000000000000537 00000000010a0152 : cslibu_3_0!CSLib300::CSDLLServerThreadWnd::ServerWindowProc+0xd8
0000000c9afafc00 00007ffef2e42357 : 000000025166ace0 0000000c741c8b78 00000000010a0152 000000005c9e1a37 : user32!UserCallWinProcCheckWow+0x149
0000000c9afafcd0 000000005c9e1922 : 0000000000000000 0000000ca2e7c990 0000000000000001 0000000038313040 : user32!DispatchMessageWorker+0x1a7
0000000c9afafd50 000000005c9e22a4 : 0000000ca2e7c990 0000000000000000 0000000000000000 0000000000000001 : mfc80u!AfxInternalPumpMessage+0x52
0000000c9afafd80 000000005c9e17fd : 0000000000000994 0000000000000000 0000000c9adabe70 0000000ca2e7c990 : mfc80u!CWinThread::Run+0x70
0000000c9afafdc0 00000000622237d7 : 0000000c9adabe70 0000000ca2e7c990 0000000ca2d90e20 0000000ca2d90e20 : mfc80u!_AfxThreadEntry+0x131
0000000c9afafec0 0000000062223894 : 00000000622d95c0 0000000ca2d90e20 0000000000000000 0000000000000000 : msvcr80!_callthreadstartex+0x17
0000000c9afafef0 00007ffef33613f2 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : msvcr80!_threadstartex+0x84
0000000c9afaff20 00007ffef4ec54f4 : 00007ffef33613d0 0000000000000000 0000000000000000 0000000000000000 : kernel32!BaseThreadInitThunk+0x22
0000000c9afaff50 0000000000000000 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : ntdll!RtlUserThreadStart+0x34

THREAD_SHA1_HASH_MOD_FUNC: 8682d5f8aeda4082adb9fdd5eb4274a4f4e1fbc3

THREAD_SHA1_HASH_MOD_FUNC_OFFSET: 2e19ac7c6f430330c5ac3eccc3eb37cc080def85

THREAD_SHA1_HASH_MOD: 0535aa0d03f77259e7bd031ea8c92d0abf210428

FAULT_INSTR_CODE: a4c8b4c

FAULTING_SOURCE_LINE: F:\dd\vctools\crt_bld\SELF_64_AMD64\crt\src\AMD64\memcpy.asm

FAULTING_SOURCE_FILE: F:\dd\vctools\crt_bld\SELF_64_AMD64\crt\src\AMD64\memcpy.asm

FAULTING_SOURCE_LINE_NUMBER: 376

SYMBOL_STACK_INDEX: 0

SYMBOL_NAME: msvcr80!memcpy+2c0

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: msvcr80

IMAGE_NAME: msvcr80.dll

DEBUG_FLR_IMAGE_TIMESTAMP: 520b0ac2

STACK_COMMAND: .ecxr ; kb

FAILURE_BUCKET_ID: INVALID_POINTER_READ_c0000005_msvcr80.dll!memcpy

BUCKET_ID: APPLICATION_FAULT_INVALID_POINTER_READ_msvcr80!memcpy+2c0

FAILURE_EXCEPTION_CODE: c0000005

FAILURE_IMAGE_NAME: msvcr80.dll

BUCKET_ID_IMAGE_STR: msvcr80.dll

FAILURE_MODULE_NAME: msvcr80

BUCKET_ID_MODULE_STR: msvcr80

FAILURE_FUNCTION_NAME: memcpy

BUCKET_ID_FUNCTION_STR: memcpy

BUCKET_ID_OFFSET: 2c0

BUCKET_ID_MODTIMEDATESTAMP: 520b0ac2

BUCKET_ID_MODCHECKSUM: cd8fa

BUCKET_ID_MODVER_STR: 8.0.50727.8428

BUCKET_ID_PREFIX_STR: APPLICATION_FAULT_INVALID_POINTER_READ_

FAILURE_PROBLEM_CLASS: APPLICATION_FAULT

FAILURE_SYMBOL_NAME: msvcr80.dll!memcpy

WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/w3wp.exe/8.5.9600.16384/5215df96/msvcr80.dll/8.0.50727.8428/520b0ac2/c0000005/0001e460.htm?Retriage=1

TARGET_TIME: 2021-05-23T02:20:26.000Z

OSBUILD: 9600

OSSERVICEPACK: 19994

SERVICEPACK_NUMBER: 0

OS_REVISION: 0

OSPLATFORM_TYPE: x64

OSNAME: Windows 8.1

OSEDITION: Windows 8.1 Server TerminalServer SingleUserTS

USER_LCID: 0

OSBUILD_TIMESTAMP: 2021-04-01 12:23:02

BUILDDATESTAMP_STR: 210331-1613

BUILDLAB_STR: winblue_ltsb_escrow

BUILDOSVER_STR: 6.3.9600.19994

ANALYSIS_SESSION_ELAPSED_TIME: 8ee8

ANALYSIS_SOURCE: UM

FAILURE_ID_HASH_STRING: um:invalid_pointer_read_c0000005_msvcr80.dll!memcpy

FAILURE_ID_HASH: {e117c5a4-11fe-47f8-fd1e-fc9c06ae830a}

Followup: MachineOwner

0:280> lmvm msvcr80
Browse full module list
start end module name
0000000062220000 00000000622e9000 msvcr80 (private pdb symbols) c:\symbols\msvcr80.AMD64.pdb\526D792BD9BA4E5CBB9104FE0BF437781\msvcr80.AMD64.pdb
Loaded symbol image file: msvcr80.dll
Image path: C:\Windows\WinSxS\amd64_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.8428_none_88dcdb0b2fb19957\msvcr80.dll
Image name: msvcr80.dll
Browse all global symbols functions data
Timestamp: Wed Aug 14 13:42:42 2013 (520B0AC2)
CheckSum: 000CD8FA
ImageSize: 000C9000
File version: 8.0.50727.8428
Product version: 8.0.50727.8428
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Microsoft Corporation
ProductName: Microsoft® Visual Studio® 200 InternalName: MSVCR80.DLL
OriginalFilename: MSVCR80.DLL
ProductVersion: 8.00.50727.8428
FileVersion: 8.00.50727.8428
FileDescription: Microsoft® C Runtime Library LegalCopyright: ? Microsoft Corporation. All rights reserved.
0:280> lmvm crxf_pdf
0000000041d30000 0000000041de3000 crxf_pdf (export symbols) crxf_pdf.dll
Loaded symbol image file: crxf_pdf.dll
Image path: C:\Program Files (x86)\SAP BusinessObjects\Crystal Reports for .NET Framework 4.0\Common\SAP BusinessObjects Enterprise XI 4.0\win64_x64\crxf_pdf.dll
Image name: crxf_pdf.dll
Browse all global symbols functions data
Timestamp: Tue Apr 18 18:59:41 2017 (58F5E38D)
CheckSum: 000BDCC7
ImageSize: 000B3000
File version: 13.0.20.2399
Product version: 13.0.20.2399
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: SAP SE
ProductName: SBOP Portable Document Format DLL for Crystal Reports
InternalName: crxf_pdf.dll
OriginalFilename: crxf_pdf.dll
ProductVersion: 13.0.20.2399
FileVersion: 13.0.20.2399
FileDescription: Portable Document Format DLL for Crystal Reports
LegalCopyright: Copyright 2015 SAP SE. All rights reserved.
LegalTrademarks: Crystal Reports is a trademark of Business Objects or one of its subsidiaries

Answer 1

I don't know how much help we will be to you. In the docs I see it crashing in Crystal Reports.

Thread 280

msvcr80!memcpy+0x2c0
msvcr80!memmove_s+0x80
crxf_pdf!UXFInitializeW+0x1834
crxf_pdf!UXFGetExportFormatsExW+0x10c
crpe32!PEGetNthTableQualifier+0x16c2d0
cslibu_3_0!CSLib300::CSThreadSafeDLL::UninitForThread+0x965
cslibu_3_0!CSLib300::CSDLLServerThreadWnd::ServerWindowProc+0xd8
user32!UserCallWinProcCheckWow+0x149
user32!DispatchMessageWorker+0x1a7
mfc80u!AfxInternalPumpMessage+0x52
mfc80u!CWinThread::Run+0x70
mfc80u!_AfxThreadEntry+0x131
msvcr80!_callthreadstartex+0x17
msvcr80!_threadstartex+0x84
kernel32!BaseThreadInitThunk+0x22
ntdll!RtlUserThreadStart+0x34

crxf_pdf - Crystal Reports export to pdf
crpe32 - Crystal Reports print engine
cslibu_3_0 - Crystal Reports

I would think that the problem is most likely data related. That is, you have a user running some giant query and Crystal Reports cannot process it or has some bug where memory gets overwritten when it tries to print to PDF.

How often does this happen? One time? Every 2 hours?

Check the logs in C:\Windows\System32\LogFiles\HTTPERR. I would expect find a "Connection_Abandoned" error that occurs at the time of the crash. Get the source IP and then query your IIS logs to find the user. Ask them what they were doing.

Based on some of the file names, it would appear that you are running something called Apriso Portal (Flexnet?) . You might have more luck troubleshooting this problem if you contact the vendors support team and ask them for help. The information that you provided looks more like an application problem and not a common Windows\IIS bug.

Answer 2

@hotmail.com bangu00 It is impossible to get useful information from your dump, you need to use Debug Diagnostic Tool.

Answer 3

Hi~SamWu-MSFT
Thank a lot.

I have already tried the analysis using the debug diag tool.(please refer to attached a file)

Answer 4

Dear MotoX80
Thank you very very much.
I'm so happy, you are my savior. :---))