https://support.microsoft.com/kb/2580960 has all the info.

Well, in such scenarios, the best way to find out the root cause is to take the debugging approach. Using Debugdiag 1.1 we got some memory dumps  and ran the analysis against them.

When looking at the report, it seems that all ASP worker threads are waiting on kernel for a file IO operations. We could see worker threads that look like below:

   1:  02eceb68 7c827339 ntdll!KiFastSystemCallRet

   2:  02eceb6c 77e6b974 ntdll!ZwOpenFile+0xc

   3:  02ecee60 77e6bb5f kernel32!FindFirstFileExW+0x1c3

   4:  02ecee80 736f20df kernel32!FindFirstFileW+0x16

   5:  02ecf0ec 736f20a1 scrrun!GetComponent+0x32

   6:  02ecf10c 736f2acd scrrun!MakePathCanonical+0xb8

   7:  02ecf45c 736f2a1f scrrun!PathObject::InternalGetName+0xab

   8:  02ecf468 77d05186 scrrun!FileObject::get_Name+0x13

   9:  02ecf484 77d14acf oleaut32!tPushValJmpTab+0x102

  10:  02ecf514 736f1cf7 oleaut32!CTypeInfo2::Invoke+0x234

  11:  02ecf540 73463ef2 scrrun!FileObject::Invoke+0x2e

  12:  02ecf57c 73463e62 vbscript!IDispatchInvoke2+0xb2

  13:  02ecf5b8 73463f23 vbscript!IDispatchInvoke+0x59

  14:  02ecf6cc 73463c80 vbscript!InvokeDispatch+0x13a

  15:  02ecf6f0 73463e04 vbscript!InvokeByName+0x42

  16:  02ecf7fc 73465576 vbscript!CScriptRuntime::RunNoEH+0x10b4

  17:  02ecf844 734654fa vbscript!CScriptRuntime::Run+0x62

  18:  02ecf93c 73465988 vbscript!CScriptEntryPoint::Call+0x51

  19:  02ecf998 73465beb vbscript!CSession::Execute+0xc8

  20:  02ecf9e8 73464d1b vbscript!COleScript::ExecutePendingScripts+0x144

  21:  02ecfa04 709e2bfa vbscript!COleScript::SetScriptState+0x14d

  22:  02ecfa30 709e2bbc asp!CActiveScriptEngine::TryCall+0x19

  23:  02ecfa6c 709e2af2 asp!CActiveScriptEngine::Call+0x31

  24:  02ecfa88 709e29f6 asp!CallScriptFunctionOfEngine+0x5b

  25:  02ecfadc 709e2921 asp!ExecuteRequest+0x17e

  26:  02ecfb44 709e24e0 asp!Execute+0x24c

...

...

   1:  ChildEBP RetAddr  

   2:  02e0ee3c 7c826d49 ntdll!KiFastSystemCallRet

   3:  02e0ee40 77e6c003 ntdll!ZwClose+0xc

   4:  02e0ee84 736f2102 kernel32!FindClose+0x5c

   5:  02e0f0ec 736f20a1 scrrun!GetComponent+0x57

   6:  02e0f10c 736f2acd scrrun!MakePathCanonical+0xb8

   7:  02e0f45c 736f2a1f scrrun!PathObject::InternalGetName+0xab

   8:  02e0f468 77d05186 scrrun!FileObject::get_Name+0x13

   9:  02e0f484 77d14acf oleaut32!tPushValJmpTab+0x102

  10:  02e0f514 736f1cf7 oleaut32!CTypeInfo2::Invoke+0x234

  11:  02e0f540 73463ef2 scrrun!FileObject::Invoke+0x2e

  12:  02e0f57c 73463e62 vbscript!IDispatchInvoke2+0xb2

  13:  02e0f5b8 73463f23 vbscript!IDispatchInvoke+0x59

  14:  02e0f6cc 73463c80 vbscript!InvokeDispatch+0x13a

  15:  02e0f6f0 73463e04 vbscript!InvokeByName+0x42

  16:  02e0f7fc 73465576 vbscript!CScriptRuntime::RunNoEH+0x10b4

  17:  02e0f844 734654fa vbscript!CScriptRuntime::Run+0x62

  18:  02e0f93c 73465988 vbscript!CScriptEntryPoint::Call+0x51

  19:  02e0f998 73465beb vbscript!CSession::Execute+0xc8

  20:  02e0f9e8 73464d1b vbscript!COleScript::ExecutePendingScripts+0x144

  21:  02e0fa04 709e2bfa vbscript!COleScript::SetScriptState+0x14d

  22:  02e0fa30 709e2bbc asp!CActiveScriptEngine::TryCall+0x19

  23:  02e0fa6c 709e2af2 asp!CActiveScriptEngine::Call+0x31

  24:  02e0fa88 709e29f6 asp!CallScriptFunctionOfEngine+0x5b

  25:  02e0fadc 709e2921 asp!ExecuteRequest+0x17e

  26:  02e0fb44 709e24e0 asp!Execute+0x24c

In some situation, the debugger host (dbghost.exe) grows uncontrollably in memory consumption causing it to shutdown which will bring the target process the debugger was attached to to terminate as well. 

In other situations, the debugger host (dbghost.exe) along with the target process seem to just be consuming all CPU cycles rendering the whole server unresponsive.

I looked into these issues to try to determine the root cause of these problems. And what I have noticed was something very interesting.

The first issue in which dbghost.exe seems to be leaking memory was actually an issue with .net 2.0 debugger extension (sos.dll).  Here is a description of the problem and how to resolve it.

As you may have already known, Debugdiag relies mainly on sos.dll for .net debugging. So when you create a crash rule to catch a .net exception and run actions, dbghost.exe loads sos.dll and runs the "!dumpobj" command to determine the type of exception that was throws every time a .net exception is thrown. Below how the script for a crash rule determines the type of .net exception thrown.

   1: Function GetCLRExceptionType(ByVal ExceptionObjHexAddr, ByVal bInnerException)

   2:     Dim Output, Lines, i

   3:  

   4:     If Debugger.IsClrExtensionMissing Then

   5:         WriteToLog "Unable to determine CLR exception type - extension dll could not be loaded."

   6:     Else

   7:         Output = Debugger.Execute("!DumpObj " & ExceptionObjHexAddr)

   8:         Lines = Split(Output, Chr(10))

   9:         For i = 0 To UBound(Lines)        

  10:             If bInnerException Then

  11:                 If InStr(Lines(i), "_innerException") <> 0 Then

  12:                     Tokens = Split(Lines(i), " ")

  13:                     For j = 0 To UBound(Tokens)

  14:                         If Len(Tokens(j)) = 8 Then                                

  15:                             GetCLRExceptionType = GetCLRExceptionType(Tokens(j), False)

  16:                             Exit For

  17:                         End If

  18:                     Next

  19:                 End If

  20:             ElseIf Len(Lines(i)) >= 7 Then

  21:                 If InStr(Lines(i), "Name: ") = 1 Then

  22:                     GetCLRExceptionType = Mid(Lines(i), 7)

  23:                     Exit For

  24:                 End If

  25:             End If

  26:         Next

  27:     End If

  28: End Function

Line 7 is where the "dumpobj" command is ran against the exception object to determine the type.

There is a known issue with the "dumpobj" command that it leaks 4k every time it is ran. This leads to a situation where dbghost.exe runs out of memory and terminates.

There is an updated sos.dll that fixes a different issue but contains a fix  for this problem. (https://support.microsoft.com/kb/967812). So all what needs to be done is to replace the existing sos.dll, and reactivate the crash rule.

The second issue has to do basically with the high rate of handled exceptions thrown by the target process (1st chance exceptions. It does not matter of the exceptions are managed or native.

For instance, if you look at the below log file generated for a crash rule, you see that there is a high number of exceptions thrown per sec.

   1: [1/27/2010 5:48:08 AM] First chance exception - 0xe0434f4d caused by thread with system id 2200

   2: [1/27/2010 5:48:08 AM] CLR Exception Type - 'System.Threading.ThreadAbortException'

   3: [1/27/2010 5:48:08 AM] First chance exception - 0xe0434f4d caused by thread with system id 2200

   4: [1/27/2010 5:48:08 AM] CLR Exception Type - 'System.Web.HttpContext'

   5: [1/27/2010 5:48:08 AM] First chance exception - 0xe0434f4d caused by thread with system id 6780

   6: [1/27/2010 5:48:08 AM] CLR Exception Type - 'System.InvalidOperationException'

   7: [1/27/2010 5:48:08 AM] First chance exception - 0xe0434f4d caused by thread with system id 6780

   8: [1/27/2010 5:48:08 AM] CLR Exception Type - 'System.InvalidOperationException'

   9: [1/27/2010 5:48:08 AM] First chance exception - 0xe0434f4d caused by thread with system id 6780

  10: [1/27/2010 5:48:08 AM] CLR Exception Type - 'System.InvalidOperationException'

  11: [1/27/2010 5:48:08 AM] First chance exception - 0xe0434f4d caused by thread with system id 6780

  12: [1/27/2010 5:48:08 AM] CLR Exception Type - 'System.InvalidOperationException'

  13: [1/27/2010 5:48:08 AM] First chance exception - 0xe0434f4d caused by thread with system id 6780

  14: [1/27/2010 5:48:08 AM] CLR Exception Type - 'System.InvalidOperationException'

  15: [1/27/2010 5:48:08 AM] First chance exception - 0xe0434f4d caused by thread with system id 6780

  16: [1/27/2010 5:48:08 AM] CLR Exception Type - 'System.InvalidOperationException'

  17: [1/27/2010 5:48:08 AM] First chance exception - 0xe0434f4d caused by thread with system id 6780

  18: [1/27/2010 5:48:08 AM] CLR Exception Type - 'System.InvalidOperationException'

  19: [1/27/2010 5:48:08 AM] First chance exception - 0xe0434f4d caused by thread with system id 6780

  20: [1/27/2010 5:48:08 AM] CLR Exception Type - 'System.InvalidOperationException'

  21: [1/27/2010 5:48:08 AM] First chance exception - 0xe0434f4d caused by thread with system id 6780

  22: [1/27/2010 5:48:08 AM] CLR Exception Type - 'System.InvalidOperationException'

  23: [1/27/2010 5:48:08 AM] First chance exception - 0xe0434f4d caused by thread with system id 6780

  24: [1/27/2010 5:48:08 AM] CLR Exception Type - 'System.InvalidOperationException'

  25: [1/27/2010 5:48:08 AM] First chance exception - 0xe0434f4d caused by thread with system id 6780

  26: [1/27/2010 5:48:08 AM] CLR Exception Type - 'System.InvalidOperationException'

  27: [1/27/2010 5:48:08 AM] First chance exception - 0xe0434f4d caused by thread with system id 6780

  28: [1/27/2010 5:48:08 AM] CLR Exception Type - 'System.InvalidOperationException'

  29: [1/27/2010 5:48:08 AM] First chance exception - 0xe0434f4d caused by thread with system id 6780

  30: [1/27/2010 5:48:08 AM] CLR Exception Type - 'System.InvalidOperationException'

  31: [1/27/2010 5:48:08 AM] First chance exception - 0xe0434f4d caused by thread with system id 6780

  32: [1/27/2010 5:48:08 AM] CLR Exception Type - 'System.InvalidOperationException'

  33: [1/27/2010 5:48:08 AM] First chance exception - 0xe0434f4d caused by thread with system id 6780

  34: [1/27/2010 5:48:08 AM] CLR Exception Type - 'System.InvalidOperationException'

  35: [1/27/2010 5:48:08 AM] First chance exception - 0xe0434f4d caused by thread with system id 6780

  36: [1/27/2010 5:48:08 AM] CLR Exception Type - 'System.InvalidOperationException'

  37: [1/27/2010 5:48:08 AM] First chance exception - 0xe0434f4d caused by thread with system id 6780

  38: [1/27/2010 5:48:08 AM] CLR Exception Type - 'System.InvalidOperationException'

  39: [1/27/2010 5:48:08 AM] First chance exception - 0xe0434f4d caused by thread with system id 2656

  40: [1/27/2010 5:48:08 AM] CLR Exception Type - 'System.InvalidOperationException'

  41: [1/27/2010 5:48:08 AM] First chance exception - 0xe0434f4d caused by thread with system id 6780

  42: [1/27/2010 5:48:08 AM] CLR Exception Type - 'System.InvalidOperationException'

An average of around 22 exceptions/sec.  Let look closely at what happens when an exception is raised: 

From https://support.microsoft.com/default.aspx?scid=kb;EN-US;105675...

.... if the application is being debugged, the debugger sees all exceptions before the program does. This is the distinction between the first and second chance exception: the debugger gets the first chance to see the exception (hence the name). If the debugger allows the program execution to continue and does not handle the exception, the program will see the exception as usual. If the program does not handle the exception, the debugger gets a second chance to see the exception. In this latter case, the program normally would crash if the debugger were not present.

...

In other words, when an exception is raised, control is passed to the debugger, the debuggers acts on that exception and passes control back to the program to handle the exception. This is an expensive operation and involve a lot of context switching. 

So, in this particular example, things are worse because for every CLR exception raised, debugdiag will run the command (!dumpObj ExceptionAddress ) to determine the type  "GetCLRExceptionType(GetCLRExceptionPtr(CausingThread), False)".  If sos.dll being used is the one with the leak issue, you will notice a high CPU utilization in both the debugger (dbghost.exe) and the target process  along with a quick increase in memory utilization in dbghost.exe. This will eventually lead to an unexpected termination of the debugger and the target process.

In the case of native exceptions, the issue will manifest itself as a high CPU that may lead to an unresponsive server as a whole! To be able to debug such environment, you would need to know what are the exceptions that are thrown in a high rate and avoid them. ]]> https://docs.microsoft.com/archive/blogs/lagdas/how-to-analyze-multiple-userdumps-and-generate-one-report-per-userdump-using-debugdiag-1-1 Mon, 18 Jan 2010 14:18:02 GMT https://blogs.msdn.microsoft.com/lagdas/2010/01/18/how-to-analyze-multiple-userdumps-and-generate-one-report-per-userdump-using-debugdiag-1-1/

In many situations, you may find it useful to analyze multiple userdumps at once but generate an analysis report for each userdump analyzed.

The Debugdiag UI can still do that, but there is no option now (1.1) to make it generate one analysis report per userdump. If -let's say- you would like to analyze 10 userdumps, then the generated analysis report would be just too big for IE to open and if it does navigating through the report would be painful.

The below VBScript code, allows you to specify a folder where the userdumps are located and run one type of analysis at a time against all the userdumps contained in the folder.

   1: 'Purpose

   2: '=======

   3: ' Analyze memory dumps using debugdiag's debugger host "dbghost.exe" without 

   4: 'requiring the user interface

   5:  

   6:  

   7:  

   8: 'Requirements for the script

   9: ' Save the script to the \Scripts folder where debugdiag is installed i.e 

  10: 'C:\Program Files\DebugDiag\Scripts

  11: ' Use the command prompt and run the command "cscript.exe AnalyzeAllDumps.vbs 

  12: '<Analysis Type>"

  13: ' <Analysis Type> could be Crash, Hang or Memory

  14:  

  15: ' For each run, the script creates a log file under C:\Program 

  16: 'Files\DebugDiag\Reports\Auto-RTW-1-1 containing run 

  17: 'information 

  18: ' The analysis reports are created under C:\Program 

  19: 'Files\DebugDiag\Reports\Auto-RTW-1-1\. For each memory dump 

  20: 'analyzed, there will be a report analysis file generated 

  21:  

  22:  

  23:  

  24:  

  25: 'Change to True if you want a script failure to trigger the JIT debugger

  26: Const DEBUG_ON_ERROR = FALSE 

  27: Const SYMBOL_PATH = "srv*c:\symcache*https://msdl.microsoft.com/download/symbols"

  28: Const SCRIPTS_VERSION = "Auto-RTW-1-1"

  29: Const DUMP_LOCATION = \\serverName\folderName

  30:  

  31: Const REPORT_LOCATION = "C:\Program Files\DebugDiag"

  32:  

  33: 'Location where the analysis reports are to be placed

  34: ANALYSIS_REPORT_PATH = REPORT_LOCATION & "\Reports\" & SCRIPTS_VERSION & "\" 

  35: 'Location where the run log file is to be created

  36: RUN_REPORT_PATH = REPORT_LOCATION & "\Logs\" & SCRIPTS_VERSION & "\" 

  37: Set PROGRESS_SINK = Nothing

  38:  

  39:  

  40: Set g_DumpFileList = CreateObject("Scripting.Dictionary")

  41:  

  42: Set WshArgs = WScript.Arguments

  43:  

  44: If WshArgs.Length = 1 then

  45: Select Case UCase(WshArgs(0))

  46: Case "HANG", "CRASH", "MEMORY"

  47: Case Else

  48: PrintHelp "INVALID ARGUMENT"

  49: WriteLine("Invalid argument passed. Run cancelled")

  50: WScript.Quit

  51: End Select

  52: Set oFSO = CreateObject("Scripting.FileSystemObject")

  53: startDate = Now()

  54: startDate = Replace(startDate, " ", "_")

  55: startDate = Replace(startDate, ":", "_")

  56: startDate = Replace(startDate, "/", "_")

  57:  

  58:  

  59: if ReportFolderStatus (RUN_REPORT_PATH)= false Then 

  60: oFSO.CreateFolder(RUN_REPORT_PATH)

  61: End if

  62:  

  63: if ReportFolderStatus (ANALYSIS_REPORT_PATH)= false Then 

  64: oFSO.CreateFolder(ANALYSIS_REPORT_PATH) 

  65: End if

  66:  

  67:  

  68: Set logFile = oFSO.CreateTextFile(RUN_REPORT_PATH & UCase(WshArgs(0)) & 

  69: "_Script_Run_" & startDate & ".log")

  70:  

  71: g_DumpFileList.RemoveAll()

  72:  

  73: ' Recurse through the memory dumps location and add all dump files to the analysis 

  74: 'list

  75: If Instr(UCase(WshArgs(0)), "CRASH") > 0 Then

  76: AddFilesToAL oFSO.GetFolder(DUMP_LOCATION)

  77: SCRIPT_TO_RUN = "CrashHangAnalysis.asp"

  78: End If

  79:  

  80: If Instr(UCase(WshArgs(0)), "HANG") > 0 Then

  81: AddFilesToAL oFSO.GetFolder(DUMP_LOCATION)

  82: SCRIPT_TO_RUN = "CrashHangAnalysis.asp"

  83: End If

  84:  

  85: If UCase(WshArgs(0)) = "MEMORY" Then

  86: AddFilesToAL oFSO.GetFolder(DUMP_LOCATION)

  87: SCRIPT_TO_RUN = "MemoryAnalysis.asp"

  88: End If

  89: WScript.Echo ""

  90: WScript.Echo "Starting run using analysis script " & SCRIPT_TO_RUN

  91:  

  92: WriteLine("Beginning " & UCase(WshArgs(0)) & " analysis run using " & 

  93: g_DumpFileList.Count & " dump files. Run 

  94: started at " & Now())

  95: WriteLine("")

  96:  

  97: For Each oFile in g_DumpFileList 

  98: bTest = False

  99: Progress = 0

 100: Set Controller = CreateObject("DbgHost.DbgControl")

 101: Set Analyzer = Controller.Analyzer

 102:  

 103: Analyzer.DataFiles.Add oFile

 104:  

 105: ' Set the analysis script we're gonna run these dump files against

 106: Analyzer.AddScriptToRunList SCRIPT_TO_RUN

 107:  

 108: WriteLine("Now analyzing dump file " & oFile)

 109: WScript.Echo ""

 110: WScript.Echo "Now analyzing dump file " & oFile

 111: tStart = Timer

 112: Analyzer.RunScripts PROGRESS_SINK, SYMBOL_PATH, ANALYSIS_REPORT_PATH, 

 113: DEBUG_ON_ERROR

 114:  

 115: 'Wait until the current analysis run is complete before starting the next 

 116: one

 117: Do Until Analyzer.ReportReady

 118: WScript.Sleep 500

 119: Loop

 120:  

 121: WriteLine("Analysis complete for " & oFile)

 122:  

 123: WScript.Echo ""

 124: WScript.Echo "Analysis complete for " & oFile

 125:  

 126: WScript.Echo ""

 127: WScript.Echo "Analysis Report generated at: " & ANALYSIS_REPORT_PATH & 

 128: "\" & Analyzer.ReportFileName

 129:  

 130: Set Controller = Nothing 

 131: Set Analyzer = Nothing

 132: Next

 133:  

 134: WScript.Echo ""

 135: WScript.Echo "Analysis batch job completed."

 136:  

 137: WriteLine("")

 138: WriteLine("Run completed at " & Now())

 139:  

 140: logFile.Close

 141: Else

 142: PrintHelp "Use the below syntax to perform Crash, Hang or Leak analysis against 

 143: memory dumps"

 144: End If

 145:  

 146:  

 147: ' This sub adds all files with .dmp extension to the list of files to analyze

 148: Sub AddFilesToAL(oFolder)

 149: Dim oSubFolder, oFile

 150:  

 151: For Each oFile in oFolder.Files

 152: If LCase(Right(oFile.Name, 4)) = ".dmp" Then

 153: g_DumpFileList.Add oFile, oFile.Name

 154: End If

 155: Next

 156:  

 157: ' Don't forget subfolders... recursivly

 158: For Each oSubFolder in oFolder.SubFolders

 159: AddFilesToAL oSubFolder

 160: Next

 161: End Sub

 162:  

 163: ' Help Usage

 164: Sub PrintHelp(ByVal sFirstLine)

 165: WScript.Echo ""

 166: WScript.Echo sFirstLine

 167: WScript.Echo ""

 168: WScript.Echo "Usage:"

 169: WScript.Echo WScript.ScriptName &" <Memory | Crash | Hang>"

 170: WScript.Echo "Example:"

 171: WScript.Echo WScript.ScriptName &" Crash"

 172: End Sub

 173:  

 174: ' Write to the log file

 175: Sub WriteLine(s)

 176: logFile.WriteLine Now & " -- " & s

 177: End Sub

 178:  

 179: ' If a folder exists

 180: Function ReportFolderStatus(fldr)

 181: Dim fso, bMsg

 182: Set fso = CreateObject("Scripting.FileSystemObject")

 183: If (fso.FolderExists(fldr)) Then

 184: bMsg = TRUE

 185: Else

 186: bMsg = FALSE

 187: End If

 188: ReportFolderStatus = bMsg

 189: End Function

To run the script, copy the code into a VBS file "AnalyzeAllDumps.vbs" and place it into the scripts folder in the Debugdiag installation folder (ie: C:\program files\Debugdiag>.

In the command prompt, run the below command:

   1: C:\Program Files\DebugDiag\Scripts>cscript.exe AnalyzeAllDumps.vbs <Memory | Crash | Hang>

The whitepaper gives a detailed description of the tool. It explains the main components of the tool and their usage. It also provides Walkthroughs on how to configure the tool to get the right data and how to analyze it. Debug Diagnostic Tool v1.1 is currently available from this Microsoft web site.

Post-mortem analysis with the windows core debuggers (windbg.exe or cdb.exe) is a time consuming process and requires a lot of debugging skills.

Automated post-mortem analysis of usedumps is basically one of the main goals of debugdiag. It is delivered via the analysis module of the tool and promises to give accurate solution by:

· Separating the raw data extraction from analysis algorithms.

· Providing a script based solution for building analysis algorithms, thus reducing the debugging skills necessary for implementing such analysis scripts.

· Providing an extensible object model solution to meet the demands of future unidentified requirements.

· Providing a built-in HTML based report generation and formatting solution similar to ASP pages.

Debugdiag is shipped with 2 main analysis scripts CrashHangAnalysis.asp and MemoryAnalysis.asp. The former is for analyzing crash and hang userdumps while the latter is for memory and handle leak analysis.

Note: Debugdiag 1.0 was shipped with "IISAnalysis.asp" and "MemoryAnalysis.asp" instead.

Before you start the analysis, make sure that you have lined up the debug symbols properly. Debugdiag accesses by default the Microsoft public symbol server, but you would need Internet connection for that. To add or modify the symbol path, go to Tool -->Options and Settings… --> Symbol Search Path for Analysis.

To analyze a userdump, open the debugdiag UI and click the Advanced Analysis view. The following window is displayed:

Add the userdumps to be analyzed by clicking Add data Files

You could select multiple userdumps for analysis, once the selection is made, click Open.

Choose the analysis category (Crash/Hang or Memory Pressure) or you could even choose both analysis categories to run against the userdump(s).

Once the selection is made, click Start Analysis, debugdiag will show the analysis progress as follows.

Another way to start the analysis is from the rules view when the rule has already generated dumps. Right-click the rule and select Analyze data.

You could also start the analysis of a user dump by just going to the userdump in windows explorer and right click the file and choose the type of analysis required.

Once the analysis is complete, debugdiag will automatically save and open the analysis report. The report will be saved in the DebugDiag\Reports folder and will open automatically in Internet Explorer.

Every Analysis report is composed of 3 main sections:

Analysis Summary

The analysis summary is an event viewer type of message that records errors, warnings and information relevant to the userdump analysis along with their descriptions and recommendations to resolve the problem they show.

Analysis Details

The Analysis Details section starts with a table of contents which lists all the memory dumps that are analyzed. For each memory dump, there is a listing of report titles indicating the type of analysis that was performed.

Script Summary

In this section, the analysis will report the status of the script that was ran to analyze the userdump. If there was any error(s) encountered while running the script, this section will show the Error Code, Source, Description, and the line(s) that caused the error(s) ]]> https://docs.microsoft.com/archive/blogs/lagdas/debugging-heap-corruption-with-application-verifier-and-debugdiag Tue, 24 Jun 2008 10:15:00 GMT https://blogs.msdn.microsoft.com/lagdas/2008/06/24/debugging-heap-corruption-with-application-verifier-and-debugdiag/ When dynamic allocation/deallocation of memory is not handled properly by user code, this might lead to memory blocks in the heap being corrupted. There are many causes of heap corruption. Some of the common causes are: Buffer overrun (Writing beyond the allocated memory), Double free (Freeing a pointer twice) and Old pointer reuse(Reusing a pointer after being freed).  The difficulty with troubleshooting heap corruption is because when a thread for instance corrupts the heap, the process does not terminate or throws an error! As long as the corrupted heap is not used, the process will not crash, but once a thread tries to use that corrupted block of memory in the heap, the process crashes! If a crash rule is active and the process crashes because of heap corruption,  what we would see as a “culprit” thread that caused the crash is actually nothing more than a victim thread!

So to get to the root of the problem and find out the cause of the corruption, that is the thread that corrupted the heap, Pagegeap should be enabled. Pageheap could be enabled directly in debugdiag via a crash rule and would provide the desired results, but if you would want to get more granular information about the corruption to simplify the code fix, Application verifier could be used in conjunction with debugdiag to get such information.

To turn on pageheap for the worker process w3wp.exe and attach the debugdiag debugger host to it, here is how to do that:

First, download and install both tools:

- Download and Install Debugdiag 1.2

- Download and Install Application Verifier v3.4

- Start Application Verifier (Start --> Programs --> Application Verifier --> Application Verifier).

- Click File --> Add Application and browse to C:\Windows\System32\Inetsrv\w3wp.exe

- In the Tests Panel, expand Basics checkbox and uncheck all except Heaps

- In the Tests Panel again, select Heaps checkbox and click Edit --> Verifier Stop Options

This basically shows the stop codes that application verifier generates. The defaults actions are for all stop codes. The most important action here is  the "Breakpoint" in the Error Reporting section which means that Application Verifier will call into the breakpoint exception when it detects that the heap is being corrupted.

- Start Debugdiag (Start --> Programs --> Debug Diagnostic Tool 1.2 --> Debugdiag 1.2 

- Add a crash rule against a specific process.

- Type in "w3wp.exe" in the "Select Target" window and make sure the "This process instance only" check box is unchecked!

- In the "Advanced Configuration (Optional)" window, click Exceptions... and add 80000003 exception with an action type of Full Userdump.

- Finish the wizard and Activate the rule. 

- Restart IIS so the new w3wp.exe loads both pageheap layer and application verifier dlls.

Note: Since pageheap is enabled per process, every instance of w3wp.exe running on the system will have pageheap on along with application verifier. There is also a performance impact associated with pageheap that would cause the processing to slow down due to heap verification.

So basically, the above configuration will make application verifier calls into the breakpoint exception when it detects that a heap operation is corrupting the heap. When the breakpoint exception is called, debugdiag will generate a full userdump. Post-mortem  analysis of the userdump will give details about the corruption such as the call stack, the type of corruption, the heap address being corrupted... etc.

Here is a simple example on how application verifier calls into the breakpoint exception after detecting a buffer overrun.

0:009> kb
ChildEBP RetAddr  Args to Child             
0685f71c 004c3933 139f8126 02206ff8 02206ff0 ntdll!DbgBreakPoint
0685f920 004c7487 004cb5d8 00000013 0a501000 vrfcore!VerifierStopMessageEx+0x4bd
0685f944 009030f9 00000013 008f33a8 0a501000 vrfcore!VfCoreRedirectedStopMessage+0x81
0685f974 008f97aa 00000013 008f33a8 0a501000 vfbasics!VfBasicsStopMessage+0x1c9
0685f9d8 008f8ed8 0685fa00 0685fa00 0685fa10 vfbasics!AVrfpCheckFirstChanceException+0x13a
0685f9e8 7c84f937 0685fa00 0685faac 0685faac vfbasics!AVrfpVectoredExceptionHandler+0x18
0685fa10 7c813fb5 00000000 02206ff0 7c888f68 ntdll!RtlpCallVectoredHandlers+0x57
0685fa24 7c814055 0685faac 0685fac8 77bd8930 ntdll!RtlCallVectoredExceptionHandlers+0x15
0685fa94 7c82ecc6 0685faac 0685fac8 0685faac ntdll!RtlDispatchException+0x19
0685fa94 09531614 0685faac 0685fac8 0685faac ntdll!KiUserExceptionDispatcher+0xe
0685fda4 095313ef 0686de18 00000001 0686de18 badEXT!doHC1+0x24
0685fdc4 5a322991 0686de18 0686cb60 0686d7a8 badEXT!HttpExtensionProc+0x108
0685fde4 5a3968ff 0686dd90 095312e7 0685fe10 w3isapi!ProcessIsapiRequest+0x214
0685fe18 5a3967e0 00000000 00000000 0686cb60 w3core!W3_ISAPI_HANDLER::IsapiDoWork+0x3fd
0685fe38 5a396764 0685fea8 0686cb60 00000000 w3core!W3_ISAPI_HANDLER::DoWork+0xb0
0685fe58 5a3966f4 0686cb60 00000000 0685fe84 w3core!W3_HANDLER::MainDoWork+0x16e
0685fe68 5a3966ae 0686cb68 0686cb60 00000001 w3core!W3_CONTEXT::ExecuteCurrentHandler+0x53
0685fe84 5a396648 00000001 0685fea8 07e84ff8 w3core!W3_CONTEXT::ExecuteHandler+0x51
0685feac 5a392264 00000000 00000000 00000000 w3core!W3_STATE_HANDLE_REQUEST::DoWork+0x9a
0685fed0 5a3965ea 00000000 00000000 00000000 w3core!W3_MAIN_CONTEXT::DoWork+0xa6

....

The exception code here that Application Verifier is raising is 00000013  which means a Buffer Overrun. 

The code that I used to test this is:

   1: {

   2:     char *ptr, *tmp;

   3:     int i;

   4:  

   5:     ptr = (char*)GlobalAlloc(GMEM_FIXED, 16);

   6:     tmp = ptr;

   7:  

   8:  

   9:     for (i = 0; i < 32; ++i)

  10:         *(tmp++) = 'a';

  11:  

  12:     GlobalFree(ptr);

  13:  

  14: }