Production Debugging for .NET Framework Applications
.gif "patterns and practices home")
Debugging Unexpected Process Termination
November 2002
Summary: This chapter describes how to approach debugging when the ASP.NET process terminates unexpectedly and the error and what caused it are unknown. A native exception might have been thrown, or the process might have been instructed to terminate. Troubleshooting steps for these scenarios are similar, regardless of what caused the problem. This chapter focuses on the latter scenario, and the walkthrough uses a COM Interop example to demonstrate unexpected terminations. However, keep in mind that these crashes can be caused by a variety of problems and are not a specific COM Interop issue. In the sample application, ASP.NET calls a COM component that deliberately causes Aspnet_wp.exe process to terminate. "Handling COM Threading in ASP.NET" gives some background information about how ASP.NET uses COM Interop and how the ASPCompat page directive works.
Although reproducing this problem on your machine may not give you the exact same results because of differing operating systems and runtime versions, the output should be similar, and the debugging concepts are the same. Also, because all Microsoft® .NET Framework applications use the same memory architecture, you can apply what you learn about COM Interop and debugging fatal exceptions in an ASP.NET context to other .NET environments, such as console applications, Microsoft Windows® operating system applications, and Windows services.
Contents
Handling COM Threading in ASP.NET
Scenario: Unexpected Process Termination
Conclusion
Handling COM Threading in ASP.NET
Any new technology has to be able to work with existing technologies because rewriting existing code can be cost prohibitive, and there is always the possibility of introducing new bugs when rewriting. Before you integrate older COM components with newer ASP.NET code, you should familiarize yourself with COM+ applications. The old COM rules still apply, and the ASP.NET architecture has adapted to accommodate these rules.
Note
For an in-depth discussion of COM threading models, see "COM Threading and Application Architecture in COM+ Applications" available on the MSDN Web site at https://msdn.microsoft.com/library/default.asp?url=/library/en-us/dncomser/html/comthread.asp.
ASP.NET and COM Interop
Typically, ASP.NET pages call managed .NET objects, so for the most part, programming with ASP.NET does not require intimate knowledge of COM, COM threading, or all of the COM rules. However, when migrating existing Web applications to the .NET Framework, it is sometimes necessary to call native COM components to perform some business functionality.
There are two ways to call native COM components from .NET: early binding and late binding. When using early binding, the information about the native COM component is known at design time. Early binding provides faster access to COM objects than late binding, and it allows you to include a reference to an imported COM component when you compile your assembly. When you use early binding and import a native COM component for use with the .NET Framework, you need to create an Interop assembly. A .NET assembly must contain type metadata, so you must create an Interop assembly that causes required metadata to be generated from the COM type library. In Microsoft Visual Studio® .NET, you would use Add References or the Tlbimp.exe utility in the .NET Framework SDK. Once you have an Interop assembly, you can create an instance of the native COM component using the "new" keyword in the same way you would create any managed object.
With late binding, information about the native COM component is not known until runtime. To create a late-bound server instance of a COM component, you can use the HttpServerUtility.CreateObject() method by passing the component's programmatic identifier (ProgID) to Server.CreateObject().
COM Components and ASP.NET
It is important to consider how COM and ASP.NET instantiate the components that you request. In Windows 2000 and XP, where the COM component is instantiated depends largely on how the threading model of the native COM component is marked in the registry, as either Free, Apartment, Neutral, or Both. This chapter uses an example that focuses on Free and Apartment.
Components Marked Free
When your ASP.NET code calls a COM component that is marked Free, the component is instantiated on the same thread pool thread that the ASP.NET page started running on. The thread pool threads that process ASP.NET pages, such as input/output (I/O) completion port threads and worker threads, are initialized as multithreaded apartment (MTA) threads. Because the component is marked Free and the ASP.NET thread is initialized as MTA, no thread switch is necessary and the performance penalty is marginal.
Components Marked Apartment
Traditionally, business COM components that are called from either ASP or MTS/COM+ have been marked Apartment. The single-threaded apartment (STA) threading model is not compatible with the default threading model for the ASP.NET thread pool, which is MTA. As a result, calling a native COM component marked Apartment from an ASP.NET page results in a thread switch and COM cross-apartment marshalling.
If the COM component marked Apartment is not configured in COM+, the component will be instantiated on the Host STA thread of the Aspnet_wp.exe process. If the COM component marked Apartment is configured in COM+, a thread switch occurs to a COM+ STA worker thread from the MTA thread that the ASP.NET started running on. In the unconfigured case, if the component marked Apartment makes a long running blocking call, no new unconfigured components marked Apartment can be instantiated or perform work on that same thread. Under stress, this presents a severe bottleneck. To work around this issue, a new directive called ASPCompat was introduced to the System.Web.UI.Page object.
How ASPCompat Works
The ASPCompat attribute minimizes thread switching due to incompatible COM threading models. More specifically, if a COM component is marked Apartment, the ASPCompat = "true" directive on an ASP.NET page runs the component marked Apartment on one of the COM+ STA worker threads.
Assume that you are requesting a page called UnexpectedCompat.aspx that contains the directive ASPCompat ="true". When the page is compiled, the page compiler checks to see if the page requires ASPCompat mode. Because this value is present, the page compiler modifies the generated page class to implement the IHttpAsyncHandler interface, adds methods to implement this interface, and modifies the page class constructor to reflect that ASPCompatMode will be used.
The two methods that are required to implement the IHttpAsyncHandler interface are BeginProcessRequest and EndProcessRequest. The implementation of these methods contains calls to this.ASPCompatBeginProcessRequest and this.ASPCompatEndProcessRequest, respectively.
You can view the code that the page compiler creates by setting Debug="true" in the <compilation> section of the web.config or machine.config files. In our example the file location of the code is C:\Windows\Microsoft.NET\Framework\v1.0.3705\Temporary ASP.NET Files\debugging\34374a37\79f52287\0n9-hgqi.cs.
The following is the source code for the class definition, the class constructor, and the implemented methods:
public class UnexpectedCompat_aspx : Debugging.Unexpected,
System.Web.SessionState.IRequiresSessionState,
System.Web.IHttpAsyncHandler {
...
public UnexpectedCompat_aspx() {
...
this.AspCompatMode = true;
}
...
public virtual System.IAsyncResult BeginProcessRequest(System.Web.HttpContext
context, System.AsyncCallback cb, object data)
{
return this.AspCompatBeginProcessRequest(context, cb, data);
}
public virtual void EndProcessRequest(System.IAsyncResult ar)
{
this.AspCompatEndProcessRequest(ar);
}
The Page.ASPCompatBeginProcessRequest() method determines if the page is already running on a COM+ STA worker thread. If it is, the call can continue to execute synchronously.
A more common scenario is a page running on a .NET MTA thread-pool thread. ASPCompatBeginProcessRequest() makes an asynchronous call to the native function ASPCompatProcessRequest() within Aspnet_isapi.dll. The following describes what happens when invoking COM+ in the latter scenario:
- The native ASPCompatProcessRequest() function constructs an ASPCompatAsyncCall class that contains the callback to the ASP.NET page and a context object created by ASP.NET. The native ASPCompatProcessRequest() function then calls a method that creates a COM+ activity and posts the activity to COM+.
- COM+ receives the request and binds the activity to an STA worker thread.
- Once the thread is bound to an activity, it calls the ASPCompatAsyncCall::OnCall() method, which initializes the intrinsics so they can be called from the ASP.NET (this is similar to classic ASP code). This function calls back into managed code so that the ProcessRequest() method of the page can continue executing.
- The page callback is invoked and the ProcessRequest() function on that page continues to run on that STA thread. This process reduces the number of thread switches required to run native COM components marked Apartment.
- Once the ASP.NET page finishes executing, it calls Page.ASPCompat EndProcessRequest() to complete the request.
Scenario: Unexpected Process Termination
Now that you know how COM threading models relate to ASP.NET, let's see how problems can occur when mixing native and managed code, and how to fix them.
In this scenario, a Server Application Unavailable error message is displayed in the browser, and there is an "aspnet_wp.exe stopped unexpectedly" entry in the application event log. Other processes continue to run, but the Aspnet_wp.exe process recycles, thus causing any related state or cache to be purged. These errors are not diagnostic—they don't point to the cause of the problem. It is known, however, that because the process terminated, something caused kernel32!ExitProcess() to be called. A snapshot of the process taken before it recycles may provide clues.
This scenario may present itself in many ways, including the following:
- Scenario 1: The big picture could be that a production Web site crashes and the connection to the browser client is lost.
- Scenario 2: A more specific example might be that a client browses to a page on a financial Web site and then tries to check stock quotes. The client makes an entry on the page, clicks Enter, and the Server Application Unavailable error message returns in the browser. The following entry appears in the Application log: "aspnet_wp.exe(PID:1234) stopped unexpectedly". Yet these messages are vague, and it's unclear what's wrong.
Breaking Down the Thought Processes
The following flowchart shows a walkthrough that helps discover more information about the problem so you can troubleshoot this scenario. The goal is to teach a core set of techniques that can be applied to similar production scenarios and architectures.
Figure 4.1. Thought Process Flowchart
Although the thought processes are described in the walkthrough that follows, let's look at the flowchart in more detail.
Is the Server in Production?
If the server is in production and the error is reproducible, you can create a memory dump to examine thread activity at the time the problem occurs. The amount of time before the problem reproduces dictates how long you have to wait for the dump to be created after you attach the debugger. If the server is not in production, debug with your favorite tool.
Run ADPlus to Monitor the Process for Exceptions
Run ADPlus in -crash mode against the Aspnet_wp.exe process, and then reproduce the problem to create a dump file.
ADPlus in -crash mode handles both first-chance and second-chance exceptions. The difference between the two is the stage at which they are handled.
- A first-chance exception may mean that something has gone wrong, but the application has handled the problem. ADPlus in -crash mode handles first-chance exceptions by outputting a stack trace to a log, creating a mini-dump, and continuing to execute.
- A second-chance exception is an unhandled first-chance exception. ADPlus handles second-chance exceptions by logging a stack trace, generating a full memory dump of the process, and terminating the debugger and process.
If ADPlus breaks in when the process is in the final stages of termination, the dump will only show one thread (why there is one thread is explained later in this chapter). You want to catch the thread activity before it terminates. A customized ADPlus.vbs script may accomplish this.
For more information on ADPlus, see article Q286350 "HOWTO: Use Autodump+ to Troubleshoot 'Hangs' and 'Crashes'" in the Microsoft Knowledge Base at https://support.microsoft.com/default.aspx?scid=kb;en-us;Q286350. Also, if you want to run ADPlus through Terminal Server, see article Q323478, "You Cannot Debug Through a Terminal Server Session" in the Microsoft Knowledge Base at https://support.microsoft.com/default.aspx?scid=kb;en-us;Q323478.
Examining the Dump File
If the name of the dump file does not contain the text Process_was_shutdown, examine the dump file with WinDbg and SOS.dll. If the name does contain the text Process_was_shutdown, check how many threads are in the dump file. If there is only one thread, run a modified version of ADPlus (described later in this chapter) to break in and create a dump before the process exits. Examine the dump file with WinDbg and SOS.dll. Consider what both the native and managed stack traces show. Determine why an exception or shutdown occurred and if the issue can be addressed with a workaround or fix.
Unexpected Process Termination Walkthrough
Now that you have seen how Server Application Unavailable errors can occur and how you can approach debugging them, here is a walkthrough that describes a simplified scenario of a realistic occurrence. The purpose is to illustrate troubleshooting techniques that help you solve the problem.
Here's the scenario: The Aspnet_wp.exe process has terminated, but the cause is not obvious. Error messages show, but they are not descriptive enough to solve the problem. You must trace the call of execution even if it hops across multiple threads. Ultimately, you discover that the process terminated because of the code. The managed code makes a native call that aborts the process.
The code also incorporates the concept of COM Interop and includes multiple ATL classes that use different threading models. The first button-click event calls an unconfigured COM component marked Apartment and the second button-click event calls an unconfigured COM component marked Free. The walkthrough concentrates on the former scenario and shows how the .NET framework handles the calls.
In this scenario, you will perform the following steps:
- Browse to the ASP.NET page and consider the values displayed in the processModel table.
- Follow the thought processes in the previous flowchart and find the errors presented both in the browser and event log.
- Run ADPlus to monitor the process, trap the error, and create a dump file.
- Look at the dump file name and determine if the process terminated or the process raised a second-chance exception.
- Use WinDbg and SOS.dll to peruse the data within the dump file, specifically the native and managed threads. By merging the managed and native call stacks, you see the managed code calling to the native code and point to the portion of the native code at fault.
Baseline View
First, gather some baseline data for comparisons with subsequent output.
To view Unexpected.aspx
- Open a command prompt window, and then type iisreset at the prompt to restart IIS.
- Open a browser window, and then browse to https://localhost/debugging/unexpected.aspx.
Figure 4.2 presents the baseline values with which to compare subsequent output. Note the information displayed, especially the data in the table. Some of the labels are self-explanatory. (For more information about these counters, see Chapter 2, "Debugging Memory Problems.") Here, the ProcessID, RequestCount, Status, and ShutdownReason fields change as the exercise continues. The RequestCount is zero because the value was queried while the first request was executing, so the request hasn't finished yet. RequestCount really means the number of requests completed. The Status is Alive, which indicates that the process is running.
Figure 4.2. Baseline data for Unexpected.aspx
You can explore the code and read through the comments by opening another instance of Unexpected.aspx.cs in Visual Studio .NET or Notepad. Notably, the STA button-click event handler is managed code that makes a call to native code and passes parameters that ultimately cause the application to abort or crash.
To start the exercise, click the Call STA COM Object button. The browser returns the following error message:
Server Application Unavailable
The web application you are attempting to access on this web server is currently unavailable. Please hit the "Refresh" button in your web browser to retry your request.
Administrator
Note: An error message detailing the cause of this specific request failure can be found in the system event log of the Web server. Please review this log entry to discover what caused this error to occur.
The application event log shows the following error:
Event Type: Error
Event Source: ASP.NET 1.0.3705.0
Event Category: None
Event ID: 1000
Date: 6/7/2002
Time: 12:59:54 PM
User: N/A
Computer: machineName
Description:
aspnet_wp.exe (PID: 2920) stopped unexpectedly.
Close and reopen the browser window, and then browse to https://localhost/debugging/unexpected.aspx. Figure 4.3 shows the tables that should be displayed in the browser.
Figure 4.3. Data from Unexpected.aspx following the crash
The RequestCount increments to two—one to browse the page for viewing and then one for the first button click. The browser shows the Status as Terminated and the ShutdownReason as Unexpected. As described in "IIS 5. x Process Model" in Chapter1, the processModel health monitoring launches a new Aspnet_wp.exe process after the first process terminates. A new process is accompanied by a new ProcessID.
Debugging a Dump File with WinDbg
Without looking at the code, you can speculate about what caused the crash—a stack overflow, code that called ProcessExit() or TerminateProcess(), or an external monitoring process that shut down the Aspnet_wp.exe process. To obtain a snapshot of what the threads are doing, you can run ADPlus in -crash mode against Aspnet_wp.exe using the following command line.
adplus.vbs -crash -pn aspnet_wp.exe -quiet
Note
The -quiet switch ensures that you can still create a dump file if the symbol path for ADPlus is not set.
To cause the failure, click the Call STA COM Object button again. A minimized window runs CDB.exe; the tool waits for a call to ProcessExit() and then creates the dump and the text files.
When the debugger window disappears, the full dump is created. You can check the C:\Debuggers directory for the most recent \Crash_Mode folder. The file's name will be similar to PID-1692__ASPNET_WP.EXE__Process_was_shutdown__full_2002-06-07_01-04-02-746_069C.dmp.
Exploring Further
Try these ideas to learn more:
- Use the debug build. In the examples in this document, a release build is used. Server messages and dump output differ with a debug build.
- Examine the log that ADPlus generates. Consider the history of what the system loads at the time of the crash.
Examining the Dump File
Examine the dump file that was created when Aspnet_wp.exe process was recycled.
To examine the dump file using WinDbg
- On the Start menu, point to Debugging Tools for Windows, and then click WinDbg.
- On the File menu, click Open Crash Dump.
- Select the appropriate dump file, and then click Open.
- If you are prompted to "Save Base Workspace Information," click No.
- If a Disassembly window pops up, close the window, and then on the Window menu, click Automatically Open Disassembly.
Symbol paths must be entered for the files that will be used by the debugger to analyze the dump file. The versions of symbol files needed on the debug computer are those that match the version on the system that produced the dump file. Include symbols from the .NET Framework. SDK, Visual Studio .NET, and symbols shipped with the following System32 folder: \%WINDIR%\system32.
To enter the symbol paths, you may do one of the following:
From the command line, type:
.sympath SRV*C:\symbols\debugginglabs*https://msdl.microsoft.com/download/symbols;C:\ symbols\debugginglabs;C:\Program Files\Microsoft Visual Studio .NET\FrameworkSDK\symbols;C:\windows\system32Note
If you have only installed the .NET Framework SDK (without Visual Studio .NET) then you need to replace the C:\Program Files\Microsoft Visual Studio .NET\FrameworkSDK\symbols with C:\Program Files\Microsoft.NET\FrameworkSDK\symbols.
Enter the symbol path for WinDbg with the _NT_SYMBOL_PATH environment variable.
On the File menu in WinDbg, click Symbol File Path, and then type:
SRV*C:\symbols\debugginglabs*https://msdl.microsoft.com/download/symbols;C:\ symbols\debugginglabs;C:\Program Files\Microsoft Visual Studio .NET\FrameworkSDK\symbols;C:\windows\system32
The "SRV" in the path tells WinDbg to go to an external symbol server and copy symbols into the local symbol cache.
To use these same symbol paths for other dumps
- On the File menu in WinDbg, click Save Workspace As, and then type a name for the saved paths, such as .NET Debugging Symbols.
As you look at the dump output, examine the call stacks by typing ~*kb (for example), and then verify that all symbols have been loaded successfully. For example, if you see the following message, you need to modify your symbol path to include the correct symbols.
040af6d4 10001017 MSVCR70!abort+0xe
*** WARNING: Unable to verify checksum for DebuggingCOM.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for
DebuggingCOM.dll -
WARNING: Stack unwind information not available. Following frames may be wrong.
040af6f4 77d281a5 DebuggingCOM+0x1017
To address this problem, modify the symbol path to include the symbols for the DLL in question. For example, use the following command:
.sympath+ c:\inetpub\wwwroot\Debugging\DebuggingCOM\Release
Then type .reload.
Examining the Native and Managed Output
Consider how many threads you are dealing with. Type tilde (~) in the command window to display all threads in the current process. In this example, there is one thread:
0:000> ~
0 Id: 7aa.208 Suspend: -1 Teb: 7ffd6000 Unfrozen
This thread is in Suspend state, unfrozen, which means that the system will run the thread when appropriate.
Look at the native call stack for the thread that was running when the dump occurred. The call stack tracks function calls and the parameters passed into these functions. The k command displays the stack frame of a given thread. If there were more than one thread, you would run the command ~*k 200 and list the call stacks for all threads loaded. Because there is only one thread, you use k 200 to display the entire call stack for this thread (given that this thread is under 200 stack frames in size).
0:000> k 200
ChildEBP RetAddr
0387ff44 77f7e76f SharedUserData!SystemCallStub+0x4
0387ff48 77e775b7 ntdll!NtDelayExecution+0xc
0387ffa0 792d05b2 kernel32!SleepEx+0x61
0387ffb4 77e802ed mscorwks!ThreadpoolMgr::TimerThreadStart+0x30
03880038 792067d5 kernel32!BaseThreadStart+0x37
00d1203b 000000ff mscorwks!CreateTypedHandle+0x16
Note
Your output may be different to the sample shown above because of timing and environment factors.
There is only one thread running because the process is in the final stages of termination. All threads except the last one have been destroyed, and managed threads have been cleaned up. If you were to run the SOS !threads command, you would see "XXX" for these managed thread IDs that do not map to native threads in the process.
Given the simplicity of the scenario, the thread that is output here may actually be the thread that calls kernel32 !ExitProcess, but this would be unusual in production mode. A production Web server most likely will serve many pages simultaneously and many threads will be running, thus complicating the troubleshooting process.
Also, because the process is in a shutdown state, managed structures in the process's memory may be inconsistent with native structures, or they may be unreliable. It is difficult to determine what thread terminated the process and whether this call was directly invoked from an ASP.NET page, so the next step is to attach a custom debugger script to the process and set a breakpoint on one of the functions in the call stack that is called before the process exits. Other threads (managed or native) may or may not provide more information.
Determining where to set a breakpoint depends on whether you have matching symbols for the modules in question. Sometimes having hotfixes installed can affect whether the symbols will resolve to the DLLs or not, and this may mean that you cannot set a breakpoint.
To create a dump for this exercise, use a customized version of ADPlus.vbs called ADPlus_KernelExit.vbs. (For download instructions, see Chapter 1, "Introduction to Production Debugging for .NET Framework Applications.") The changes to ADPlus are located within the CreateCDBScript() function, which creates the .cfg file that configures the CDB debugger when it attaches to the process.
Make sure that ADPlus_KernelExit.vbs is located in the C:\Debuggers folder. Symbols are required, so make sure that you are connected to the Internet or that you have downloaded the symbols for kernel32.pdb locally. Internet access is adequate because this tool sets up the path to the Microsoft external symbol server for you.
The following lines have been added to the ADPlus.vbs script:
objTextFile.Writeline "* -- Setup Symbol Path to use external symbol server ---"
objTextFile.Writeline ".sympath+ SRV*c:\symbols\debugginglabs*https://msdl.microsoft.com/download/symbols;c:\symbols
\debugginglabs"
objTextFile.Writeline "* ------------------------------------------------------"
objTextFile.Writeline "*"
objTextFile.Writeline "*"
objTextFile.Writeline "*"
objTextFile.Writeline "* -- Make sure the symbols are loaded for mscoree.dll ---"
objTextFile.Writeline "x kernel32!ExitProcess"
objTextFile.Writeline "* ------------------------------------------------------------"
objTextFile.Writeline "*"
objTextFile.Writeline "*"
objTextFile.Writeline "*"
'Since this is a breakpoint, any \ need to be escaped with \\
Dim EscapedCrashDir
EscapedCrashDir = Replace(ShortCrashDir, "\", "\\")
objTextFile.Writeline "* -- Add a breakpoint for CAsyncPipeManager::Close ---"
objTextFile.Writeline "bp kernel32!ExitProcess " & Chr(34) & ".echo Encountered
kernel32!ExitProcess breakpoint. Dumping process ...;.dump /mfh " &
EscapedCrashDir & "\\" & "PID-" & pid & "__" & packagename &
"__Kernel32ExitProcess__full.dmp;q" & Chr(34)
objTextFile.Writeline "* --------------------------------------------------------"
As noted, ADPlus_KernelExit.vbs sets the symbol path to use the Microsoft public symbol server. The tool verifies that the symbols can be loaded for the kernel32!ExitProcess() function. Because you know that the process terminates, the tool sets a breakpoint on kernel32!ExitProcess() to create a full dump file when the breakpoint is hit.
Along with a user dump, the ADPlus_KernelExit.vbs script also generates a text file that provides a history of exceptions, and dumps the native stack traces for those exceptions (the same as in the unmodified ADPlus.vbs). Keep in mind that throwing many exceptions over an extended period of time can produce a very large text file.
Throwing as few exceptions as possible is preferable because catching exceptions is extremely resource intensive. You can even disable the creation of the log file if it is too obtrusive. For more information, see "ASP.NET Performance Tips and Best Practices" on the GotDotNet Web site at https://code.msdn.microsoft.com/GotDotNet.aspx.
To prepare for debugging
Open a command prompt window, and then type iisreset at the prompt to restart IIS.
Open a new browser window, and then browse to the ASP.NET sample page at https://localhost/debugging/unexpected.aspx.
Run ADPlus_KernelExit.vbs in -crash mode and specify which Aspnet_wp.exe process to attach to.
In the command prompt window, change to the debuggers folder and type:
adplus_kernelexit.vbs -crash -pn aspnet_wp.exe -quietA minimized CDB.exe window shows on the task bar.
On the ASP.NET page, click Call STA COM Object and the CDB.exe window becomes active. The browser should display the following message:
Server Application Unavailable
The web application you are attempting to access on this web server is currently unavailable. Please hit the "Refresh" button in your web browser to retry your request.
Administrator Note An error message detailing the cause of this specific request failure can be found in the system event log of the Web server. Please review this log entry to discover what caused this error.
- Look in the C:\debuggers directory for the most recent \Crash_Mode folder that has a name similar to Crash_Mode__Date_06-10-2002__Time_15-56-01PM. The dump file has a name similar to PID-3968__ASPNET_WP.EXE__Kernel32ExitProcess__full.dmp.
- Open WinDbg, and then open the crash dump and set the symbol path like you did before.
- Select the Reload checkbox, or type .reload in the command window.
To examine the native threads
List the native threads using the tilde (~)command.
0:010> ~ 0 Id: ecc.8d8 Suspend: 1 Teb: 7ffde000 Unfrozen 1 Id: ecc.e4 Suspend: 1 Teb: 7ffdd000 Unfrozen 2 Id: ecc.d28 Suspend: 1 Teb: 7ffdc000 Unfrozen 3 Id: ecc.670 Suspend: 1 Teb: 7ffdb000 Unfrozen 4 Id: ecc.9e8 Suspend: 1 Teb: 7ffda000 Unfrozen 5 Id: ecc.d08 Suspend: 1 Teb: 7ffd9000 Unfrozen 6 Id: ecc.d88 Suspend: 1 Teb: 7ffd8000 Unfrozen 7 Id: ecc.55c Suspend: 1 Teb: 7ffd7000 Unfrozen 8 Id: ecc.c40 Suspend: 1 Teb: 7ffd6000 Unfrozen 9 Id: ecc.f54 Suspend: 1 Teb: 7ffd5000 Unfrozen 10 Id: ecc.59c Suspend: 1 Teb: 7ffd4000 Unfrozen 11 Id: ecc.f40 Suspend: 1 Teb: 7ffaf000 Unfrozen 12 Id: ecc.cfc Suspend: 1 Teb: 7ffae000 Unfrozen 13 Id: ecc.564 Suspend: 1 Teb: 7ffad000 UnfrozenThere are considerably more threads in this dump file than in previous dump. List the call stacks of the native threads using the ~*k 200 command.
0:010> ~*k 200 0 Id: ecc.8d8 Suspend: 1 Teb: 7ffde000 Unfrozen ChildEBP RetAddr 0012f874 77f7e76f SharedUserData!SystemCallStub+0x4 0012f878 77e775b7 ntdll!NtDelayExecution+0xc 0012f8d0 77e61bf1 kernel32!SleepEx+0x61 0012f8dc 00422c17 kernel32!Sleep+0xb 0012ff44 004237db aspnet_wp!wmain+0x30b [e:\dna\src\xsp\wp\main.cxx @ 222] 0012ffc0 77e7eb69 aspnet_wp!wmainCRTStartup+0x131 [f:\vs70builds\9111\vc\crtbld\crt\src\crtexe.c @ 379] 0012fff0 00000000 kernel32!BaseProcessStart+0x23 1 Id: ecc.e4 Suspend: 1 Teb: 7ffdd000 Unfrozen ChildEBP RetAddr 009df2dc 77f7f4af SharedUserData!SystemCallStub+0x4 009df2e0 77e7788b ntdll!NtWaitForSingleObject+0xc 009df344 77e79d6a kernel32!WaitForSingleObjectEx+0xa8 009df354 771da2e7 kernel32!WaitForSingleObject+0xf 009df370 772afc14 ole32!GetToSTA+0x6d 009df394 772af180 ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0xe2 009df464 771cf0af ole32!CRpcChannelBuffer::SendReceive2+0xa6 009df4ec 77cc28f7 ole32!CAptRpcChnl::SendReceive+0x95 009df514 77d28c20 RPCRT4!NdrClientInitializeNew+0x17 009df550 771c07c8 RPCRT4!NdrProxySendReceive+0x41 009df63c 771c955f ole32!CoTaskMemFree+0xf 009df654 792107dc ole32!CStdIdentity::CInternalUnk::Release+0x3b 009df684 00177226 mscorwks!SafeRelease+0x56 WARNING: Frame IP not in any known module. Following frames may be wrong. 00000000 00000000 0x177226 2 Id: ecc.d28 Suspend: 1 Teb: 7ffdc000 Unfrozen ChildEBP RetAddr 00adff04 77f7e76f SharedUserData!SystemCallStub+0x4 00adff08 77e775b7 ntdll!NtDelayExecution+0xc 00adff60 77e61bf1 kernel32!SleepEx+0x61 00adff6c 792d00bc kernel32!Sleep+0xb 00adffb4 77e802ed mscorwks!ThreadpoolMgr::GateThreadStart+0x4c 00adffec 00000000 kernel32!BaseThreadStart+0x37 3 Id: ecc.670 Suspend: 1 Teb: 7ffdb000 Unfrozen ChildEBP RetAddr 00ddfefc 77f7f4af SharedUserData!SystemCallStub+0x4 00ddff00 77e7788b ntdll!NtWaitForSingleObject+0xc 00ddff64 77e79d6a kernel32!WaitForSingleObjectEx+0xa8 00ddff74 0042289c kernel32!WaitForSingleObject+0xf 00ddff80 7c00fbab aspnet_wp!DoPingThread+0x10 [e:\dna\src\xsp\wp\main.cxx @ 412] 00ddffb4 77e802ed MSVCR70!_endthread+0xaa 00ddffec 00000000 kernel32!BaseThreadStart+0x37 4 Id: ecc.9e8 Suspend: 1 Teb: 7ffda000 Unfrozen ChildEBP RetAddr 011bfe80 77f7f49f SharedUserData!SystemCallStub+0x4 011bfe84 77e74bd8 ntdll!ZwWaitForMultipleObjects+0xc 011bff20 77e74c70 kernel32!WaitForMultipleObjectsEx+0x12c 011bff38 791cccf6 kernel32!WaitForMultipleObjects+0x17 011bffa0 791ccc6b mscorwks!DebuggerRCThread::MainLoop+0x90 011bffac 791ccc1e mscorwks!DebuggerRCThread::ThreadProc+0x55 011bffb4 77e802ed mscorwks!DebuggerRCThread::ThreadProcStatic+0xb 011bffec 00000000 kernel32!BaseThreadStart+0x37 5 Id: ecc.d08 Suspend: 1 Teb: 7ffd9000 Unfrozen ChildEBP RetAddr 0335ff08 77f7e76f SharedUserData!SystemCallStub+0x4 0335ff0c 77e775b7 ntdll!NtDelayExecution+0xc 0335ff64 77e61bf1 kernel32!SleepEx+0x61 0335ff70 791cde2c kernel32!Sleep+0xb 0335ffb4 77e802ed mscorwks!GCHeap::FinalizerThreadStart+0x1f9 0335ffec 00000000 kernel32!BaseThreadStart+0x37 6 Id: ecc.d88 Suspend: 1 Teb: 7ffd8000 Unfrozen ChildEBP RetAddr 0347ff4c 77f7ef9f SharedUserData!SystemCallStub+0x4 0347ff50 77e73b3f ntdll!ZwRemoveIoCompletion+0xc 0347ff7c 792cf8b8 kernel32!GetQueuedCompletionStatus+0x27 0347ffb4 77e802ed mscorwks!ThreadpoolMgr::CompletionPortThreadStart+0x46 0347ffec 00000000 kernel32!BaseThreadStart+0x37 7 Id: ecc.55c Suspend: 1 Teb: 7ffd7000 Unfrozen ChildEBP RetAddr 0379ff1c 77f7f4af SharedUserData!SystemCallStub+0x4 0379ff20 77e7788b ntdll!NtWaitForSingleObject+0xc 0379ff84 77e79d6a kernel32!WaitForSingleObjectEx+0xa8 0379ff94 792cf5dc kernel32!WaitForSingleObject+0xf 0379ffb4 77e802ed mscorwks!ThreadpoolMgr::WorkerThreadStart+0x2e 0379ffec 00000000 kernel32!BaseThreadStart+0x37 8 Id: ecc.c40 Suspend: 1 Teb: 7ffd6000 Unfrozen ChildEBP RetAddr 0394ff44 77f7e76f SharedUserData!SystemCallStub+0x4 0394ff48 77e775b7 ntdll!NtDelayExecution+0xc 0394ffa0 792d05b2 kernel32!SleepEx+0x61 0394ffb4 77e802ed mscorwks!ThreadpoolMgr::TimerThreadStart+0x30 0394ffec 00000000 kernel32!BaseThreadStart+0x37 9 Id: ecc.f54 Suspend: 1 Teb: 7ffd5000 Unfrozen ChildEBP RetAddr 03c7fe24 77f7efff SharedUserData!SystemCallStub+0x4 03c7fe28 77cc1ac9 ntdll!NtReplyWaitReceivePortEx+0xc 03c7ff90 77cc167e RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0xf6 03c7ff94 77cc1505 RPCRT4!RecvLotsaCallsWrapper+0x9 03c7ffac 77cc1670 RPCRT4!BaseCachedThreadRoutine+0x64 03c7ffb4 77e802ed RPCRT4!ThreadStartRoutine+0x16 03c7ffec 00000000 kernel32!BaseThreadStart+0x37 10 Id: ecc.59c Suspend: 1 Teb: 7ffd4000 Unfrozen ChildEBP RetAddr 03daf618 7923b594 kernel32!ExitProcess 03daf624 792176bb mscorwks!SafeExitProcess+0x38 03daf630 79210cdd mscorwks!ForceEEShutdown+0x3f 03daf644 7917b297 mscorwks!CorExitProcess+0x5a 03daf654 7c00bb88 mscoree!CorExitProcess+0x3d 03daf65c 7c00440a MSVCR70!__crtExitProcess+0x25 03daf668 7c00f791 MSVCR70!_cinit+0x101 03daf678 7c012ac8 MSVCR70!_exit+0xe 03daf6c0 7c012799 MSVCR70!raise+0xae 03daf6cc 10005877 MSVCR70!abort+0xe 03daf6d0 77cc2f58 DebuggingCOM!CSTA::RaiseError+0x17 [c:\inetpub\wwwroot\debugging\debuggingcom\sta.cpp @ 50] 03daf6f0 77d281a5 RPCRT4!Invoke+0x30 03dafad8 77d28d3e RPCRT4!NdrStubCall2+0x1fb 03dafb30 7713bb3d RPCRT4!CStdStubBuffer_Invoke+0x3f 03dafb90 772aec81 OLEAUT32!CUnivStubWrapper::Invoke+0xe1 03dafecc 77d43d6a ole32!StubInvoke+0xa5 03daff2c 77d43dd0 USER32!DispatchMessageWorker+0x2e9 03daff38 771e04c8 USER32!DispatchMessageW+0xb 03daff6c 771dbdc5 ole32!CDllHost::STAWorkerLoop+0x52 03daff8c 771dbd0e ole32!CDllHost::WorkerThread+0xb8 03daff90 771debb7 ole32!DLLHostThreadEntry+0x9 03daffa8 771dec16 ole32!CRpcThread::WorkerLoop+0x1e 03daffb4 77e802ed ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x1a 03daffec 00000000 kernel32!BaseThreadStart+0x37 11 Id: ecc.f40 Suspend: 1 Teb: 7ffaf000 Unfrozen ChildEBP RetAddr 03f0ff1c 77f7e76f SharedUserData!SystemCallStub+0x4 03f0ff20 77e775b7 ntdll!NtDelayExecution+0xc 03f0ff78 77e61bf1 kernel32!SleepEx+0x61 03f0ff84 771c987b kernel32!Sleep+0xb 03f0ff90 771debb7 ole32!CROIDTable::WorkerThreadLoop+0x12 03f0ffa8 771dec16 ole32!CRpcThread::WorkerLoop+0x1e 03f0ffb4 77e802ed ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x1a 03f0ffec 00000000 kernel32!BaseThreadStart+0x37 12 Id: ecc.cfc Suspend: 1 Teb: 7ffae000 Unfrozen ChildEBP RetAddr 0403ff20 77f7f4af SharedUserData!SystemCallStub+0x4 0403ff24 77e7788b ntdll!NtWaitForSingleObject+0xc 0403ff88 771debdf kernel32!WaitForSingleObjectEx+0xa8 0403ffa8 771dec16 ole32!CRpcThread::WorkerLoop+0x58 0403ffb4 77e802ed ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x1a 0403ffec 00000000 kernel32!BaseThreadStart+0x37 13 Id: ecc.564 Suspend: 1 Teb: 7ffad000 Unfrozen ChildEBP RetAddr 0413ff30 77f7e76f SharedUserData!SystemCallStub+0x4 0413ff34 77e775b7 ntdll!NtDelayExecution+0xc 0413ff8c 792d0444 kernel32!SleepEx+0x61 0413ffb4 77e802ed mscorwks!ThreadpoolMgr::WaitThreadStart+0x22 0413ffec 00000000 kernel32!BaseThreadStart+0x37
Thread Discussion
The following table summarizes the functions that each thread was executing at the point the dump file was generated. These functions were extracted from the previous thread output.
Table 4.1: Summary of thread state
| ThreadID | Purpose |
|---|---|
| 0 | aspnet_wp!wmain |
| 1 | ole32!GetToSTA |
| 2 | mscorwks!ThreadpoolMgr::GateThreadStart |
| 3 | aspnet_wp!DoPingThread |
| 4 | mscorwks!DebuggerRCThread::MainLoop |
| 5 | mscorwks!GCHeap::FinalizerThreadStart |
| 6 | mscorwks!ThreadpoolMgr::CompletionPortThreadStart |
| 7 | mscorwks!ThreadpoolMgr::WorkerThreadStart |
| 8 | mscorwks!ThreadpoolMgr::TimerThreadStart |
| 9 | RPCRT4!RecvLotsaCallsWrapper |
| 10 | ole32!DLLHostThreadEntry (EXIT THREAD) |
| 11 | ole32!CRpcThread::WorkerLoop |
| 12 | ole32!CRpcThread::WorkerLoop |
| 13 | mscorwks!ThreadpoolMgr::WaitThreadStart |
WinDbg without SOS.dll doesn't display the managed call stack; instead, it displays native call stacks. However, some of the managed calls are wrappers for native calls.
Note
When attaching to a process with Visual Studio .NET in mixed mode (native and managed), Visual Studio .NET displays both managed and native thread information in one window.
Let's examine more extensively what the threads are and what they're doing:
- Thread 0 is the ASP.NET main thread as shown by the call to aspnet_wp!wmain. This thread has initialized the runtime, connected the named pipes back to InetInfo.exe, and created the ping thread. Once it has completed the initialization, it loops, waiting for the process to exit.
- Thread 1 is a managed thread that has both managed and native code running on it, along with a call to the ole32 !GetToSTA function. To learn more about this thread, you'll use the SOS.dll and SieExtPub.dll managed extension, which is explained later in this chapter.
- Threads 2, 6, 7, 8, and 13 are ThreadpoolMgr threads. Note the calls to mscorwks!ThreadpoolMgr::GateThreadStart, mscorwks!ThreadpoolMgr::WorkerThreadStart, and mscorwks!ThreadpoolMgr::TimerThreadStart, respectively. These threads are present in ASP.NET (but not always in console applications) because of the ASP.NET use of the thread pool.
- Thread 2 is a GateThread that monitors the health of the thread pool. It is responsible for injecting and retiring worker and I/O threads based on indicators, such as CPU utilization, garbage-collection frequency, and thread starvation. There is only one thread of this type. It is created when the first I/O or work request is executed.
- Thread 7 is a managed Threadpool worker thread. This thread is waiting for work to be assigned to it.
- Thread 8 is a TimerThread that is used to manage timed callbacks specified using the System.Threading.Timer class. There is only one thread of this type, and it is created when the first System.Threading.Timer object is created.
- Thread 6 is an I/O completion port thread that is waiting for a managed or native work item.
- Thread 13 is a WaitThread that is used to wait on handles. When the wait is signaled or times out, a callback is executed on a worker thread. Each wait thread can wait on up to 64 distinct wait handles concurrently. New wait threads are created as necessary.
- Thread 3 is the ASP.NET ping thread as shown by the call to aspnet_wp!DoPingThread. This thread responds to the pings it receives from the health-monitoring thread in InetInfo.exe.
- Thread 4 is a debugger thread as shown by the call to mscorwks!DebuggerRCThread. All managed processes have a debugger thread, even if there is no debugger attached. There is always one debugger thread in a given process. This thread exists so that a managed debugger can attach and control the managed code. Threads react differently depending on whether a managed or native debugger attaches invasively or not.
- If you break into the process with a managed debugger (for example, CorDbg or Visual Studio .NET), it attaches noninvasively. The entire process is not frozen; instead, only the managed threads are frozen. The native threads that don't contain runtime references continue running. Options to use a managed debugger to attach invasively are available.
- If you use a native debugger, it can attach invasively or noninvasively. When you break into the process invasively and quit, the process terminates, and consequently all threads—managed and native—are shut down. If you attach noninvasively, the process (and threads) stop temporarily. The debugger pauses the native and managed code and provides a snapshot in time. It then can detach and the process resumes.
- In IIS 5.x, there is only one Aspnet_wp.exe worker process and one debugger thread. Consequently, only one debugger can be attached to the Aspnet_wp.exe process at a time. This can pose a problem if you're dealing with multiple Web applications on the same machine. In IIS 6.0, you can coerce an AppDomain to run in a separate application pool. (For more information, see "IIS 5.x Process Model" and "IIS 6.0 Process Model" in Chapter 1.) Separate application pools provide multiple W3wp.exe processes. Multiple debugger threads are created in these processes (one in each), allowing you to debug more efficiently.
- Thread 5 is a finalizer thread as shown by a call to mscorwks!GCHeap::FinalizerThreadStart. Just like the debugger thread, there is always one finalizer thread in .NET Framework version 1.0. The finalizer thread calls the Finalize() method of objects when the garbage collector (GC) determines that the object is unreachable. In Visual Basic .NET, you can implement a Finalize() method. The same function in the Visual C#™ development tool and managed extensions for the Visual C++® development system is performed by a destructor. To learn more, see the SDK for articles such as ".NET Framework General Reference" and "Implementing Finalize and Dispose to Clean Up Unmanaged Resources."
- Thread 9 is a lightweight remote procedure call (RPC) thread (note the call to RPCRT4!RecvLotsaCallsWrapper) used for local remote procedure call (LRPC) communication. It handles incoming RPC calls to this process and is waiting for work.
- Thread 10 is a HostThreadEntry thread that COM has created to instantiate the COM component marked Apartment. You need to discover why DebuggingCOM!CSTA::RaiseError calls abort() by checking the code, which is explained later in this chapter.
- Threads 11 and 12 are worker threads that are waiting for COM to dispatch work.
Checking the Code
The following shows the relevant code from STA.cpp. Based on the input parameters, this function offers three actions when called.
STDMETHODIMP CSTA::RaiseError(LONG errorid, LONG errorseverity)
{
switch(errorid)
{
case 1:
abort();
return E_FAIL;
case 0:
return E_FAIL;
default:
return S_OK;
}
}
The function CSTA::RaiseError() calls abort() when 1 is passed as the first parameter. If you ran ~*kb and looked at the call stack, you would see that 1 and 5 are indeed passed. The crash is caused by calling abort(), which eventually causes the process to exit. As a result, the Aspnet_wp.exe process terminates and a new process is created.
How is abort() being called? What called CSTA::RaiseError()? Can you prevent passing 1 and 5 to the function? Let's look at the managed threads.
You can use two extension DLLs to examine managed output in WinDbg: SOS.dll and SieExtPub.dll. The former provides most of the functionality you need for this walkthrough. You'll use the latter to examine MTA/STA thread switching.
To examine managed threads, you must load the SOS.dll debugger extension.
To examine the managed threads
In WinDbg, type**.load SOS\SOS.dll** to load the extension. Amend the path as necessary; in this case the location of SOS.dll is in the SOS folder.
Type !findtable to initialize SOS with the table information for the runtime version you are debugging.
Type !threads, and you should see results similar to the data shown below:
0:010> !threads ThreadCount: 5 UnstartedThread: 0 BackgroundThread: 5 PendingThread: 0 DeadThread: 01 5 6 7 10 ID f20 74c cc0 6c4 6c0 ThreadOBJ 00155008 00156df8 00171b68 00179650 001f9b58 State a220 b220 1800220 4220 220 PreEmptive GC Enabled Enabled Enabled Enabled Enabled GC Alloc Context 00000000: 00000000 00000000: 00000000 00000000: 00000000 00000000: 00000000 00000000: 00000000 Domain 001a68c0 00166030 00166030 00166030 00166030 Lock Count 1 0 0 1 1 Apt MTA MTA MTA STA Unk Exception (Finalizer) (Threadpool Worker) (GC)
Three out of five managed threads initialized to the MTA threading model. Thread 10 is initialized to STA and shows (GC). This means that the GC was run on the thread at some point. Four out of five threads are in the same AppDomain. Thread 1 is running in a different AppDomain because it's running in the ASP.NET virtual folder.
The following tables match the thread state to the values in the Appendix State chart and provide more information about the possible execution states for the thread. A thread can be in more than one state at a given time.
Table 4.2: Breakdown of state for thread 1, state value = 0xa220
| Symbol | Value | Description |
|---|---|---|
| TS_InMTA | 0x00008000 | Thread is part of the MTA |
| TS_CoInitialized | 0x00002000 | CoInitialize has been called for this thread |
| TS_Background | 0x00000200 | Thread is a background thread |
| TS_LegalToJoin | 0x00000020 | Indicates whether it is now legal to attempt a Join() |
Table 4.3: Breakdown of state for thread 6, state value = 0xb220
| Symbol | Value | Description |
|---|---|---|
| TS_InMTA | 0x00008000 | Thread is part of the MTA |
| TS_CoInitialized | 0x00002000 | CoInitialize has been called for this thread |
| TS_WeOwn | 0x00001000 | Exposed object initiated this thread |
| TS_Background | 0x00000200 | Thread is a background thread |
| TS_LegalToJoin | 0x00000020 | Indicates whether it is now legal to attempt a Join() |
Table 4.4: Breakdown of state for thread 7, state value = 0x01800220
| Symbol | Value | Description |
|---|---|---|
| TS_TPWorkerThread | 0x01000000 | Indicates that this a threadpool worker thread. (if not, it is a threadpool completionport thread) |
| TS_ThreadPoolThread | 0x00800000 | Indicates that this a threadpool thread |
| TS_Background | 0x00000200 | Thread is a background thread |
| TS_LegalToJoin | 0x00000020 | Indicates whether it is now legal to attempt a Join() |
Table 4.5: Breakdown of state for thread 10, state value = 0x4220
| Symbol | Value | Description |
|---|---|---|
| TS_InSTA | 0x00004000 | Thread hosts an STA |
| TS_Background | 0x00000200 | Thread is a background thread |
| TS_LegalToJoin | 0x00000020 | Indicates whether it is now legal to attempt a Join() |
Table 4.6: Breakdown of state for thread 12, state value = 0x220
| Symbol | Value | Description |
|---|---|---|
| TS_Background | 0x00000200 | Thread is a background thread |
| TS_LegalToJoin | 0x00000020 | Indicates whether it is now legal to attempt a Join() |
Display the managed call stacks by typing ~*e !clrstack. This command is similar to the ~*k WinDbg command, but it applies to managed calls. WinDbg now displays the current enter stack pointer (ESP), the current enter instruction pointer (EIP), and the method signature for each frame with a managed thread. The !clrstack command displays only the managed threads, not any native call-stack information.
Note
Unless otherwise specified, the dump files were created using release builds.
0:010> ~*e !clrstack
Thread 0
Not a managed thread.
Thread 1
ESP EIP
0096f95c 7ffe0304 [FRAME: ComPlusMethodFrameStandalone] [DEFAULT]
[hasThis] Void DebuggingCOMLib.STAClass.RaiseError(I4,I4)
0096f970 03a00e06 [DEFAULT] [hasThis] Void
Debugging.Unexpected.btnSTA_Click(Object,Class System.EventArgs)
0096f980 03ca2f55 [DEFAULT] [hasThis] Void
System.Web.UI.WebControls.Button.OnClick(Class System.EventArgs)
0096f994 03ca2d32 [DEFAULT] [hasThis] Void
System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandl
er.RaisePostBackEvent(String)
0096f9a4 03ca2ce3 [DEFAULT] [hasThis] Void
System.Web.UI.Page.RaisePostBackEvent(Class
System.Web.UI.IPostBackEventHandler,String)
0096f9ac 03ca2c42 [DEFAULT] [hasThis] Void
System.Web.UI.Page.RaisePostBackEvent(Class
System.Collections.Specialized.NameValueCollection)
0096f9bc 03a4a5e0 [DEFAULT] [hasThis] Void
System.Web.UI.Page.ProcessRequestMain()
0096f9f4 03a48317 [DEFAULT] [hasThis] Void
System.Web.UI.Page.ProcessRequest()
0096fa2c 03a47d7b [DEFAULT] [hasThis] Void
System.Web.UI.Page.ProcessRequest(Class System.Web.HttpContext)
0096fa34 03a47d54 [DEFAULT] [hasThis] Void
System.Web.HttpApplication/CallHandlerExecutionStep.Execute()
0096fa44 03a5a230 [DEFAULT] [hasThis] Class System.Exception
System.Web.HttpApplication.ExecuteStep(Class IExecutionStep,ByRef
Boolean)
0096fa8c 03a5997b [DEFAULT] [hasThis] Void
System.Web.HttpApplication.ResumeSteps(Class System.Exception)
0096fad0 03a59853 [DEFAULT] [hasThis] Class System.IAsyncResult
System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProce
ssRequest(Class System.Web.HttpContext,Class
System.AsyncCallback,Object)
0096faec 0377fed0 [DEFAULT] [hasThis] Void
System.Web.HttpRuntime.ProcessRequestInternal(Class
System.Web.HttpWorkerRequest)
0096fb28 0377fcbd [DEFAULT] Void
System.Web.HttpRuntime.ProcessRequest(Class
System.Web.HttpWorkerRequest)
0096fb34 0377b9af [DEFAULT] [hasThis] I4
System.Web.Hosting.ISAPIRuntime.ProcessRequest(I,I4)
0096fbf0 7920eab0 [FRAME: ContextTransitionFrame]
0096fcd0 7920eab0 [FRAME: ComMethodFrame]
Thread 2
Not a managed thread.
Thread 3
Not a managed thread.
Thread 4
Not a managed thread.
Thread 5
Not a managed thread.
Thread 6
ESP EIP
Thread 7
ESP EIP
Thread 8
Not a managed thread.
Thread 9
Not a managed thread.
Thread 10
ESP EIP
Thread 11
Not a managed thread.
Thread 12
ESP EIP
Thread 13
Not a managed thread.
Earlier you looked at the native output of one of the managed threads (as shown in thread 1). SOS.dll output fills in the missing information for the managed data. To build both managed and native portions of the stack, cut and paste the output from !clrstack into the middle portion where there were previously no mapped modules.
NATIVE
1 Id: f80.f20 Suspend: 1 Teb: 7ffdd000 Unfrozen
ChildEBP RetAddr
0096f2dc 77f7f4af SharedUserData!SystemCallStub+0x4
0096f2e0 77e7788b ntdll!NtWaitForSingleObject+0xc
0096f344 77e79d6a kernel32!WaitForSingleObjectEx+0xa8
0096f354 771da2e7 kernel32!WaitForSingleObject+0xf
0096f370 772afc14 ole32!GetToSTA+0x6d
0096f394 772af180 ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0xe2
0096f464 771cf0af ole32!CRpcChannelBuffer::SendReceive2+0xa6
0096f4ec 77cc28f7 ole32!CAptRpcChnl::SendReceive+0x95
0096f514 77d28c20 rpcrt4!NdrClientInitializeNew+0x17
0096f550 771c07c8 rpcrt4!NdrProxySendReceive+0x41
0096f63c 771c955f ole32!CoTaskMemFree+0xf
0096f654 792107dc ole32!CStdIdentity::CInternalUnk::Release+0x3b
0096f684 00174b7e mscorwks!SafeRelease+0x56
WARNING: Frame IP not in any known module. Following frames may be wrong.
00000000 00000000 0x174b7e
MANAGED
0096f95c 7ffe0304 [FRAME: ComPlusMethodFrameStandalone] [DEFAULT]
[hasThis] Void DebuggingCOMLib.STAClass.RaiseError(I4,I4)
0096f970 03a00e06 [DEFAULT] [hasThis] Void
Debugging.Unexpected.btnSTA_Click(Object,Class System.EventArgs)
0096f980 03ca2f55 [DEFAULT] [hasThis] Void
System.Web.UI.WebControls.Button.OnClick(Class System.EventArgs)
0096f994 03ca2d32 [DEFAULT] [hasThis] Void
System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandl
er.RaisePostBackEvent(String)
0096f9a4 03ca2ce3 [DEFAULT] [hasThis] Void
System.Web.UI.Page.RaisePostBackEvent(Class
System.Web.UI.IPostBackEventHandler,String)
0096f9ac 03ca2c42 [DEFAULT] [hasThis] Void
System.Web.UI.Page.RaisePostBackEvent(Class
System.Collections.Specialized.NameValueCollection)
0096f9bc 03a4a5e0 [DEFAULT] [hasThis] Void
System.Web.UI.Page.ProcessRequestMain()
0096f9f4 03a48317 [DEFAULT] [hasThis] Void
System.Web.UI.Page.ProcessRequest()
0096fa2c 03a47d7b [DEFAULT] [hasThis] Void
System.Web.UI.Page.ProcessRequest(Class System.Web.HttpContext)
0096fa34 03a47d54 [DEFAULT] [hasThis] Void
System.Web.HttpApplication/CallHandlerExecutionStep.Execute()
0096fa44 03a5a230 [DEFAULT] [hasThis] Class System.Exception
System.Web.HttpApplication.ExecuteStep(Class IExecutionStep,ByRef
Boolean)
0096fa8c 03a5997b [DEFAULT] [hasThis] Void
System.Web.HttpApplication.ResumeSteps(Class System.Exception)
0096fad0 03a59853 [DEFAULT] [hasThis] Class System.IAsyncResult
System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProce
ssRequest(Class System.Web.HttpContext,Class
System.AsyncCallback,Object)
0096faec 0377fed0 [DEFAULT] [hasThis] Void
System.Web.HttpRuntime.ProcessRequestInternal(Class
System.Web.HttpWorkerRequest)
0096fb28 0377fcbd [DEFAULT] Void
System.Web.HttpRuntime.ProcessRequest(Class
System.Web.HttpWorkerRequest)
0096fb34 0377b9af [DEFAULT] [hasThis] I4
System.Web.Hosting.ISAPIRuntime.ProcessRequest(I,I4)
0096fbf0 7920eab0 [FRAME: ContextTransitionFrame]
0096fcd0 7920eab0 [FRAME: ComMethodFrame]
The !clrstack output for Thread1 does not show all of the call-stack frames. When you use the !dumpstack command, the output shows all of the stack frames, including the origin of the thread and the reference to the type of thread you are dealing with, which is a completion port thread. The following example shows the end of the output from the !dumpstack command for thread 1.
Note
You must change to the appropriate thread before running the !dumpstack command. For example, to change to thread 1, execute ~1s.
0:01> !dumpstack
...
0096ff24 0042218c aspnet_wp!CAsyncPipeManager::ProcessCompletion+0x1d4
[e:\dna\src\xsp\wp\asyncpipemanager.cxx:516], calling
aspnet_wp!CAsyncPipeManager::ProcessMessage
[e:\dna\src\xsp\wp\asyncpipemanager.cxx:597]
0096ff40 7a13ecc2 aspnet_isapi!CorThreadPoolCompletionCallback+0x41
[e:\dna\src\xsp\isapi\threadpool.cxx:773]
0096ff84 792cf905 mscorwks!ThreadpoolMgr::CompletionPortThreadStart+0x93
0096ffb4 77e802ed kernel32!BaseThreadStart+0x37
In the managed stack, Debugging.Unexpected.btnSTA_Click calls DebuggingCOMLib.STAClass. Is there a specific call from one to the other in code? In the native stack, the calls work their way to ole32!GetToSTA.
Look at the code in the Debugging.Unexpected.btnSTA_Click event.
private void btnSTA_Click(object sender, System.EventArgs e)
{
DebuggingCOMLib.STAClass staobj = new DebuggingCOMLib.STAClass();
staobj.RaiseError(1,5);
Label1.Text += "STA Call Completed sucessfully";
}
If the source code is not available, you can examine the assembly by supplying the instruction pointer for the call-stack frame to the !u command. The instruction pointer can be retrieved from the !clrstack: output.
0096f970 03a00e06 [DEFAULT] [hasThis] Void
Debugging.Unexpected.btnSTA_Click(Object,Class System.EventArgs)
To disassemble this function, type !u 03a00e06.
0:010> !u 03a00e06
Normal JIT generated code
[DEFAULT] [hasThis] Void Debugging.Unexpected.btnSTA_Click(Object,Class
System.EventArgs)
Begin 03a00de0, size 54
03a00de0 57 push edi
03a00de1 56 push esi
03a00de2 8bf9 mov edi,ecx
03a00de4 b988c17503 mov ecx,0x375c188 (MT:
DebuggingCOMLib.STAClass)
03a00de9 e879147b75 call mscorwks!JIT_NewCrossContext (791b2267)
03a00dee 8bf0 mov esi,eax
03a00df0 8bce mov ecx,esi
03a00df2 ff15e4c17503 call dword ptr [0375c1e4]
(DebuggingCOMLib.STAClass..ctor)
03a00df8 6a05 push 0x5
03a00dfa 8bce mov ecx,esi
03a00dfc ba01000000 mov edx,0x1
03a00e01 8b01 mov eax,[ecx]
03a00e03 ff504c call dword ptr [eax+0x4c]
03a00e06 8b8fbc000000 mov ecx,[edi+0xbc]
03a00e0c 8bf1 mov esi,ecx
03a00e0e 8b01 mov eax,[ecx]
03a00e10 ff9090010000 call dword ptr [eax+0x190]
03a00e16 8bc8 mov ecx,eax
03a00e18 8b15a89c1702 mov edx,[02179ca8] ("STA Call Completed
sucessfully")
03a00e1e e83d3590ff call 03304360 (System.String.Concat)
03a00e23 8bd0 mov edx,eax
03a00e25 8bce mov ecx,esi
03a00e27 8b01 mov eax,[ecx]
03a00e29 ff9094010000 call dword ptr [eax+0x194]
03a00e2f 5e pop esi
03a00e30 5f pop edi
03a00e31 c20400 ret 0x4
The code in both C# and x86 assemblies shows the creation of a COM wrapper. It does this by making a call to DebuggingCOMLib.STAClass..ctor), which then calls a method on a staobj.RaiseError(1,5) COM object, which in turn calls MSVCR70!abort. This confirms the findings in the native code for thread 10.
You now have seen the code and have proven that it directly calls MSVCR70!abort(), which causes kernel32!ExitProcess() to be called and the Aspnet_wp.exe process to terminate. Although you have technically solved the problem, it would be helpful to learn more. Thread 10 contains a call to MSVCR70!abort(). If there are multiple calls to the DLL, how can you determine which call causes a problem? SOS.dll can provide you with information on the COM threading models for the managed threads using the !comstate command.
0:010> !comstate
ID TEB APT APTId CallerTID Context
0 230 7ffde000 MTA 6e 0 00145008
1 f20 7ffdd000 MTA 75 0 00145008
2 cdc 7ffdb000 Ukn
3 e80 7ffda000 Ukn
4 b94 7ffdc000 MTA 0 0 00145008
5 874 7ffd9000 Ukn
6 74c 7ffd8000 MTA 0 0 00145008
7 cc0 7ffd7000 MTA 0 0 00145008
8 764 7ffd6000 MTA 0 0 00145008
9 b1c 7ffd5000 Ukn
10 6c4 7ffd4000 STA 0 0 00145178
11 114 7ffaf000 Ukn
12 6c0 7ffae000 Ukn 0 0 00000000
13 524 7ffad000 Ukn
The !comstate output shows that the apartment type of each thread is either MTA, Ukn, or STA. The Ukn entry implies that the threads haven't been initialized to a particular threading model yet.
The other apartment type that stands out is STA, and as noted earlier, thread 10 is an STA thread. You can use two methods to determine on which thread this call continues to execute.
The first method is to guess. There is an entry for ole32!GetToSTA(), so you know that you need to switch to an STA thread in the process. The output of !comstate indicates that thread 10 is the only thread initialized to the STA threading model.
The second method is to troubleshoot using the SieExtPub.dll extension to WinDbg. Make sure that the SieExtPub.dll file is located in your C:\Debuggers or applicable folder. Type**.load sos\sieextpub.dll** to load the extension. Type !comcalls to display thread switching.
0:010> .load sos\sieextpub.dll
Thank you for using SIEExtPub.
0:010> !comcalls
Thread 1 - MTA
Target Process ID: 00000f80 = 3968
Target Thread ID: 000006c4 = 10 (STA)
This output shows that Thread 1, which is an MTA thread, calls Thread 10, which is an STA thread.
The previous example required an MTA/STA thread switch to allow the managed code to call the native code. This is because a component marked Apartment cannot be instantiated from an MTA thread with a thread switch due to COM marshalling. An alternative approach would be to avoid the marshalling overhead by using the ASPCompat page directive, which forces the page's ProcessRequest method to execute on a COM+ STA worker thread.
Using ASPCompat
To view the changes that ASPCompat can bring to the previous scenario, rerun the examples, but make one code change—add the following directive to the ASP.NET page. The Unexpectedcompat.aspx example page contains this page directive:
<%@ Page language="c#" ASPCompat="true" AutoEventWireup="false"
Inherits="Debugging.Unexpected" %>
This is the only change to the code. Because this code inherits from the same DLL as the previous ASP.NET page, recompiling the DLL is not necessary.
As noted earlier, when a managed object is created from a thread that is initialized to the MTA threading model, and it calls a COM component marked Apartment, it must follow COM threading rules. ASPCompat complies by running the page's ProcessRequest method on a thread initialized to STA threading model.
To start with a clean slate
- Open a command prompt window, and then type iisreset at the prompt to restart IIS.
- Browse to https://localhost/debugging/unexpectedcompat.aspx
The following figure shows a table similar to what the browser displays:
Figure 4.4. Baseline data for Unexpectedcompat.aspx
Run ADPlus_KernelExit.vbs in -crash mode from the command line:
ADPLUS_KernelExit.vbs -crash -pn aspnet_wp.exe -quiet
Click the Call STA COM Object button. The browser displays a Server Application Unavailable error and the application event log shows the following entry: "aspnet_wp.exe(PID:2992) stopped unexpectedly."
Look in the latest \Crash folder under C:\Debuggers for a file with a name similar to PID-2916__ASPNET_WP.EXE__Kernel32ExitProcess__full.dmp.
Open the file in WinDbg, and ensure the symbol path is set up correctly:
Enter the symbol path in the command line as it is shown below. Replace C:\Windows with your windows folder location.
srv*C:\symbols\debugginglabs*https://msdl.microsoft.com/download/symbols;C:\symbols
\debugginglabs;C:\program files\microsoft visual studio
.net\frameworksdk\symbols;C:\windows\system32;C:\inetpub\wwwroot\debugging\
debuggingCOM\release.
Dump the native threads using the tilde '~' command:
0:000> ~
0 Id: b64.db0 Suspend: 1 Teb: 7ffde000 Unfrozen
1 Id: b64.b98 Suspend: 1 Teb: 7ffdd000 Unfrozen
2 Id: b64.60c Suspend: 1 Teb: 7ffdc000 Unfrozen
3 Id: b64.d9c Suspend: 1 Teb: 7ffdb000 Unfrozen
4 Id: b64.b1c Suspend: 1 Teb: 7ffda000 Unfrozen
5 Id: b64.680 Suspend: 1 Teb: 7ffd9000 Unfrozen
6 Id: b64.d1c Suspend: 1 Teb: 7ffd8000 Unfrozen
7 Id: b64.3ac Suspend: 1 Teb: 7ffd7000 Unfrozen
8 Id: b64.184 Suspend: 1 Teb: 7ffd6000 Unfrozen
9 Id: b64.980 Suspend: 1 Teb: 7ffd5000 Unfrozen
10 Id: b64.d8 Suspend: 1 Teb: 7ffd4000 Unfrozen
11 Id: b64.c30 Suspend: 1 Teb: 7ffaf000 Unfrozen
12 Id: b64.f54 Suspend: 1 Teb: 7ffae000 Unfrozen
13 Id: b64.198 Suspend: 1 Teb: 7ffad000 Unfrozen
14 Id: b64.8b0 Suspend: 1 Teb: 7ffac000 Unfrozen
15 Id: b64.9c4 Suspend: 1 Teb: 7ffab000 Unfrozen
16 Id: b64.c5c Suspend: 1 Teb: 7ffaa000 Unfrozen
17 Id: b64.9a0 Suspend: 1 Teb: 7ffa9000 Unfrozen
18 Id: b64.da0 Suspend: 1 Teb: 7ffa8000 Unfrozen
19 Id: b64.934 Suspend: 1 Teb: 7ffa7000 Unfrozen
20 Id: b64.5c0 Suspend: 1 Teb: 7ffa6000 Unfrozen
Now dump the native call stacks.
0:000> ~*k 200
0 Id: b64.db0 Suspend: 1 Teb: 7ffde000 Unfrozen
ChildEBP RetAddr
0012f874 77f7e76f SharedUserData!SystemCallStub+0x4
0012f878 77e775b7 ntdll!NtDelayExecution+0xc
0012f8d0 77e61bf1 kernel32!SleepEx+0x61
0012f8dc 00422c17 kernel32!Sleep+0xb
0012ff44 004237db aspnet_wp!wmain+0x30b [e:\dna\src\xsp\wp\main.cxx @
222]
0012ffc0 77e7eb69 aspnet_wp!wmainCRTStartup+0x131
[f:\vs70builds\9111\vc\crtbld\crt\src\crtexe.c @ 379]
0012fff0 00000000 kernel32!BaseProcessStart+0x23
1 Id: b64.b98 Suspend: 1 Teb: 7ffdd000 Unfrozen
ChildEBP RetAddr
0086ff4c 77f7ef9f SharedUserData!SystemCallStub+0x4
0086ff50 77e73b3f ntdll!ZwRemoveIoCompletion+0xc
0086ff7c 792cf8b8 kernel32!GetQueuedCompletionStatus+0x27
0086ffb4 77e802ed mscorwks!ThreadpoolMgr::CompletionPortThreadStart+0x46
0086ffec 00000000 kernel32!BaseThreadStart+0x37
2 Id: b64.60c Suspend: 1 Teb: 7ffdc000 Unfrozen
ChildEBP RetAddr
0096ff04 77f7e76f SharedUserData!SystemCallStub+0x4
0096ff08 77e775b7 ntdll!NtDelayExecution+0xc
0096ff60 77e61bf1 kernel32!SleepEx+0x61
0096ff6c 792d00bc kernel32!Sleep+0xb
0096ffb4 77e802ed mscorwks!ThreadpoolMgr::GateThreadStart+0x4c
0096ffec 00000000 kernel32!BaseThreadStart+0x37
3 Id: b64.d9c Suspend: 1 Teb: 7ffdb000 Unfrozen
ChildEBP RetAddr
00c6fefc 77f7f4af SharedUserData!SystemCallStub+0x4
00c6ff00 77e7788b ntdll!NtWaitForSingleObject+0xc
00c6ff64 77e79d6a kernel32!WaitForSingleObjectEx+0xa8
00c6ff74 0042289c kernel32!WaitForSingleObject+0xf
00c6ff80 7c00fbab aspnet_wp!DoPingThread+0x10
[e:\dna\src\xsp\wp\main.cxx
@ 412]
00c6ffb4 77e802ed msvcr70!_threadstart+0x6c
[f:\vs70builds\9466\vc\crtbld\crt\src\thread.c @ 196]
00c6ffec 00000000 kernel32!BaseThreadStart+0x37
4 Id: b64.b1c Suspend: 1 Teb: 7ffda000 Unfrozen
ChildEBP RetAddr
0101fe80 77f7f49f SharedUserData!SystemCallStub+0x4
0101fe84 77e74bd8 ntdll!ZwWaitForMultipleObjects+0xc
0101ff20 77e74c70 kernel32!WaitForMultipleObjectsEx+0x12c
0101ff38 791cccf6 kernel32!WaitForMultipleObjects+0x17
0101ffa0 791ccc6b mscorwks!DebuggerRCThread::MainLoop+0x90
0101ffac 791ccc1e mscorwks!DebuggerRCThread::ThreadProc+0x55
0101ffb4 77e802ed mscorwks!DebuggerRCThread::ThreadProcStatic+0xb
0101ffec 00000000 kernel32!BaseThreadStart+0x37
5 Id: b64.680 Suspend: 1 Teb: 7ffd9000 Unfrozen
ChildEBP RetAddr
031bff08 77f7e76f SharedUserData!SystemCallStub+0x4
031bff0c 77e775b7 ntdll!NtDelayExecution+0xc
031bff64 77e61bf1 kernel32!SleepEx+0x61
031bff70 791cde2c kernel32!Sleep+0xb
031bffb4 77e802ed mscorwks!GCHeap::FinalizerThreadStart+0x1f9
031bffec 00000000 kernel32!BaseThreadStart+0x37
6 Id: b64.d1c Suspend: 1 Teb: 7ffd8000 Unfrozen
ChildEBP RetAddr
0354ff1c 77f7f4af SharedUserData!SystemCallStub+0x4
0354ff20 77e7788b ntdll!NtWaitForSingleObject+0xc
0354ff84 77e79d6a kernel32!WaitForSingleObjectEx+0xa8
0354ff94 792cf5dc kernel32!WaitForSingleObject+0xf
0354ffb4 77e802ed mscorwks!ThreadpoolMgr::WorkerThreadStart+0x2e
0354ffec 00000000 kernel32!BaseThreadStart+0x37
7 Id: b64.3ac Suspend: 1 Teb: 7ffd7000 Unfrozen
ChildEBP RetAddr
0364ff4c 77f7ef9f SharedUserData!SystemCallStub+0x4
0364ff50 77e73b3f ntdll!ZwRemoveIoCompletion+0xc
0364ff7c 792cf8b8 kernel32!GetQueuedCompletionStatus+0x27
0364ffb4 77e802ed mscorwks!ThreadpoolMgr::CompletionPortThreadStart+0x46
0364ffec 00000000 kernel32!BaseThreadStart+0x37
8 Id: b64.184 Suspend: 1 Teb: 7ffd6000 Unfrozen
ChildEBP RetAddr
0387ff44 77f7e76f SharedUserData!SystemCallStub+0x4
0387ff48 77e775b7 ntdll!NtDelayExecution+0xc
0387ffa0 792d05b2 kernel32!SleepEx+0x61
0387ffb4 77e802ed mscorwks!ThreadpoolMgr::TimerThreadStart+0x30
03880038 792067d5 kernel32!BaseThreadStart+0x37
00d1203b 000000ff mscorwks!CreateTypedHandle+0x16
9 Id: b64.980 Suspend: 1 Teb: 7ffd5000 Unfrozen
ChildEBP RetAddr
03bffe18 77f7f49f SharedUserData!SystemCallStub+0x4
03bffe1c 77e74bd8 ntdll!ZwWaitForMultipleObjects+0xc
03bffeb8 77d46db9 kernel32!WaitForMultipleObjectsEx+0x12c
03bfff14 77d46e5b user32!RealMsgWaitForMultipleObjectsEx+0x13c
03bfff30 757832b1 user32!MsgWaitForMultipleObjects+0x1d
03bfff80 77c37fb8 comsvcs!CSTAThread::WorkerLoop+0x1df
03bfffb4 77e802ed msvcrt!_endthreadex+0xa0
03bfffec 00000000 kernel32!BaseThreadStart+0x37
10 Id: b64.d8 Suspend: 1 Teb: 7ffd4000 Unfrozen
ChildEBP RetAddr
03cffe24 77f7efff SharedUserData!SystemCallStub+0x4
03cffe28 77cc1ac9 ntdll!NtReplyWaitReceivePortEx+0xc
03cfff90 77cc167e rpcrt4!LRPC_ADDRESS::ReceiveLotsaCalls+0xf6
03cfff94 77cc1505 rpcrt4!RecvLotsaCallsWrapper+0x9
03cfffac 77cc1670 rpcrt4!BaseCachedThreadRoutine+0x64
03cfffb4 77e802ed rpcrt4!ThreadStartRoutine+0x16
03cfffec 00000000 kernel32!BaseThreadStart+0x37
11 Id: b64.c30 Suspend: 1 Teb: 7ffaf000 Unfrozen
ChildEBP RetAddr
03dffe18 77f7f49f SharedUserData!SystemCallStub+0x4
03dffe1c 77e74bd8 ntdll!ZwWaitForMultipleObjects+0xc
03dffeb8 77d46db9 kernel32!WaitForMultipleObjectsEx+0x12c
03dfff14 77d46e5b user32!RealMsgWaitForMultipleObjectsEx+0x13c
03dfff30 757832b1 user32!MsgWaitForMultipleObjects+0x1d
03dfff80 77c37fb8 comsvcs!CSTAThread::WorkerLoop+0x1df
03dfffb4 77e802ed msvcrt!_endthreadex+0xa0
03dfffec 00000000 kernel32!BaseThreadStart+0x37
12 Id: b64.f54 Suspend: 1 Teb: 7ffae000 Unfrozen
ChildEBP RetAddr
03effe18 77f7f49f SharedUserData!SystemCallStub+0x4
03effe1c 77e74bd8 ntdll!ZwWaitForMultipleObjects+0xc
03effeb8 77d46db9 kernel32!WaitForMultipleObjectsEx+0x12c
03efff14 77d46e5b user32!RealMsgWaitForMultipleObjectsEx+0x13c
03efff30 757832b1 user32!MsgWaitForMultipleObjects+0x1d
03efff80 77c37fb8 comsvcs!CSTAThread::WorkerLoop+0x1df
03efffb4 77e802ed msvcrt!_endthreadex+0xa0
03efffec 00000000 kernel32!BaseThreadStart+0x37
13 Id: b64.198 Suspend: 1 Teb: 7ffad000 Unfrozen
ChildEBP RetAddr
03fffe18 77f7f49f SharedUserData!SystemCallStub+0x4
03fffe1c 77e74bd8 ntdll!ZwWaitForMultipleObjects+0xc
03fffeb8 77d46db9 kernel32!WaitForMultipleObjectsEx+0x12c
03ffff14 77d46e5b user32!RealMsgWaitForMultipleObjectsEx+0x13c
03ffff30 757832b1 user32!MsgWaitForMultipleObjects+0x1d
03ffff80 77c37fb8 comsvcs!CSTAThread::WorkerLoop+0x1df
03ffffb4 77e802ed msvcrt!_endthreadex+0xa0
03ffffec 00000000 kernel32!BaseThreadStart+0x37
14 Id: b64.8b0 Suspend: 1 Teb: 7ffac000 Unfrozen
ChildEBP RetAddr
040ffe18 77f7f49f SharedUserData!SystemCallStub+0x4
040ffe1c 77e74bd8 ntdll!ZwWaitForMultipleObjects+0xc
040ffeb8 77d46db9 kernel32!WaitForMultipleObjectsEx+0x12c
040fff14 77d46e5b user32!RealMsgWaitForMultipleObjectsEx+0x13c
040fff30 757832b1 user32!MsgWaitForMultipleObjects+0x1d
040fff80 77c37fb8 comsvcs!CSTAThread::WorkerLoop+0x1df
040fffb4 77e802ed msvcrt!_endthreadex+0xa0
040fffec 00000000 kernel32!BaseThreadStart+0x37
15 Id: b64.9c4 Suspend: 1 Teb: 7ffab000 Unfrozen
ChildEBP RetAddr
041ffe18 77f7f49f SharedUserData!SystemCallStub+0x4
041ffe1c 77e74bd8 ntdll!ZwWaitForMultipleObjects+0xc
041ffeb8 77d46db9 kernel32!WaitForMultipleObjectsEx+0x12c
041fff14 77d46e5b user32!RealMsgWaitForMultipleObjectsEx+0x13c
041fff30 757832b1 user32!MsgWaitForMultipleObjects+0x1d
041fff80 77c37fb8 comsvcs!CSTAThread::WorkerLoop+0x1df
041fffb4 77e802ed msvcrt!_endthreadex+0xa0
041fffec 00000000 kernel32!BaseThreadStart+0x37
16 Id: b64.c5c Suspend: 1 Teb: 7ffaa000 Unfrozen
ChildEBP RetAddr
042ff8e0 7923b594 kernel32!ExitProcess
042ff8ec 792176bb mscorwks!SafeExitProcess+0x38
042ff8f8 79210cdd mscorwks!ForceEEShutdown+0x3f
042ff90c 7917b297 mscorwks!CorExitProcess+0x5a
042ff91c 7c00bb88 mscoree!CorExitProcess+0x3d
042ff924 7c00440a msvcr70!__crtExitProcess+0x25
[f:\vs70builds\9466\vc\crtbld\crt\src\crt0dat.c @ 451]
042ff930 7c00f791 msvcr70!doexit+0xa9
[f:\vs70builds\9466\vc\crtbld\crt\src\crt0dat.c @ 402]
042ff940 7c012ac8 msvcr70!_exit+0xe
[f:\vs70builds\9466\vc\crtbld\crt\src\crt0dat.c @ 302]
042ff988 7c012799 msvcr70!raise+0xae
[f:\vs70builds\9466\vc\crtbld\crt\src\winsig.c @ 509]
042ff994 10005877 msvcr70!abort+0xe
[f:\vs70builds\9466\vc\crtbld\crt\src\abort.c @ 48]
17 Id: b64.9a0 Suspend: 1 Teb: 7ffa9000 Unfrozen
ChildEBP RetAddr
043ffe18 77f7f49f SharedUserData!SystemCallStub+0x4
043ffe1c 77e74bd8 ntdll!ZwWaitForMultipleObjects+0xc
043ffeb8 77d46db9 kernel32!WaitForMultipleObjectsEx+0x12c
043fff14 77d46e5b user32!RealMsgWaitForMultipleObjectsEx+0x13c
043fff30 757832b1 user32!MsgWaitForMultipleObjects+0x1d
043fff80 77c37fb8 comsvcs!CSTAThread::WorkerLoop+0x1df
043fffb4 77e802ed msvcrt!_endthreadex+0xa0
043fffec 00000000 kernel32!BaseThreadStart+0x37
18 Id: b64.da0 Suspend: 1 Teb: 7ffa8000 Unfrozen
ChildEBP RetAddr
044ffef0 77f7f4af SharedUserData!SystemCallStub+0x4
044ffef4 77e7788b ntdll!NtWaitForSingleObject+0xc
044fff58 77e79d6a kernel32!WaitForSingleObjectEx+0xa8
044fff68 75780195 kernel32!WaitForSingleObject+0xf
044fff80 77c37fb8
comsvcs!CSTAThreadPool::LoadBalanceThreadControlLoop+0x21
044fffb4 77e802ed msvcrt!_endthreadex+0xa0
044fffec 00000000 kernel32!BaseThreadStart+0x37
19 Id: b64.934 Suspend: 1 Teb: 7ffa7000 Unfrozen
ChildEBP RetAddr
045ffef0 77f7f4af SharedUserData!SystemCallStub+0x4
045ffef4 77e7788b ntdll!NtWaitForSingleObject+0xc
045fff58 77e79d6a kernel32!WaitForSingleObjectEx+0xa8
045fff68 7577fd59 kernel32!WaitForSingleObject+0xf
045fff80 77c37fb8 comsvcs!CSTAThreadPool::KillThreadControlLoop+0x21
045fffb4 77e802ed msvcrt!_endthreadex+0xa0
045fffec 00000000 kernel32!BaseThreadStart+0x37
20 Id: b64.5c0 Suspend: 1 Teb: 7ffa6000 Unfrozen
ChildEBP RetAddr
0485fee0 77f7f49f SharedUserData!SystemCallStub+0x4
0485fee4 77e74bd8 ntdll!ZwWaitForMultipleObjects+0xc
0485ff80 792d0467 kernel32!WaitForMultipleObjectsEx+0x12c
0485ffb4 77e802ed mscorwks!ThreadpoolMgr::WaitThreadStart+0x45
0485ffec 00000000 kernel32!BaseThreadStart+0x37
Note
Your output may be different from the sample shown above because of the differences in the names of function calls between Windows 2000 and Windows XP.
Thread Discussion
The following table summarizes the functions that were being executed by the threads at the point the dump was created.
Table 4.7: Summary of thread state information
| Thread ID | Purpose |
|---|---|
| 0 | aspnet_wp!wmain |
| 1 | mscorwks!ThreadpoolMgr::CompletionPortThreadStart |
| 2 | mscorwks!ThreadpoolMgr::GateThreadStart |
| 3 | aspnet_wp!DoPingThread |
| 4 | mscorwks!DebuggerRCThread::MainLoop |
| 5 | mscorwks!GCHeap::FinalizerThreadStart |
| 6 | mscorwks!ThreadpoolMgr::WorkerThreadStart |
| 7 | mscorwks!ThreadpoolMgr::CompletionPortThreadStart |
| 8 | mscorwks!ThreadpoolMgr::TimerThreadStart |
| 9 | comsvcs!CSTAThread::WorkerLoop |
| 10 | rpcrt4!RecvLotsaCallsWrapper |
| 11 | comsvcs!CSTAThread::WorkerLoop |
| 12 | comsvcs!CSTAThread::WorkerLoop |
| 13 | comsvcs!CSTAThread::WorkerLoop |
| 14 | comsvcs!CSTAThread::WorkerLoop |
| 15 | comsvcs!CSTAThread::WorkerLoop |
| 16 | comsvcs!CSTAThread::WorkerLoop (EXIT) |
| 17 | comsvcs!CSTAThread::WorkerLoop |
| 18 | comsvcs!CSTAThreadPool::LoadBalanceThreadControlLoop |
| 19 | comsvcs!CSTAThreadPool::KillThreadControlLoop |
| 20 | mscorwks!ThreadpoolMgr::WaitThreadStart |
There are eight comsvcs!CSTAThread::WorkerLoop threads—seven threads, plus one thread for each CPU on the machine. There are also two comsvcs!CSTAThreadPool threads, five mscorwks!ThreadpoolMgr threads, and one rpcrt4!RecvLotsaCallsWrapper thread. For more information about the COM+ thread pool, see article Q282490, "INFO: Thread Pool Differences Between COM+ and MTS" in the Microsoft Knowledge Base at https://support.microsoft.com/default.aspx?scid=kb;en-us;Q282490.
Examine the threading models by using the SOS.dll. Load the extension and initialize it using the !findtable command. Then type !comstate to dump information on the threading models of the COM threads:
0:000> !comstate
ID TEB APT APTId CallerTID Context
0 db0 7ffde000 MTA 80 0 00145008
1 b98 7ffdd000 MTA 0 0 00145008
2 60c 7ffdc000 Ukn
3 d9c 7ffdb000 Ukn
4 b1c 7ffda000 Ukn
5 680 7ffd9000 MTA 0 0 00145008
6 d1c 7ffd8000 MTA 0 0 00145008
7 3ac 7ffd7000 MTA 0 0 00145008
8 184 7ffd6000 MTA 0 0 00145008
9 980 7ffd5000 STA 81 0 00145178
10 d8 7ffd4000 Ukn
11 c30 7ffaf000 STA 0 0 00145230
12 f54 7ffae000 STA 0 0 001452e8
13 198 7ffad000 STA 0 0 001453a0
14 8b0 7ffac000 STA 0 0 00145458
15 9c4 7ffab000 STA 0 0 00145510
16 c5c 7ffaa000 STA 0 0 001457f0
17 9a0 7ffa9000 STA 0 0 00145680
18 da0 7ffa8000 Ukn
19 934 7ffa7000 Ukn
20 5c0 7ffa6000 Ukn
In this aspnet_wp process, there are eight threads initialized to the STA threading model. These are the comsvcs!CSTAThread::WorkerLoop threads that were mentioned previously.
Examining the Managed Threads
Use the !threads command to dump information on the managed threads:
0:000> !threads
ThreadCount: 5
UnstartedThread: 0
BackgroundThread: 5
PendingThread: 0
DeadThread: 0
| 1 | 5 | 6 | 17 | 16 | |
| ID | b98 | 680 | d1c | 9a0 | c5c |
| ThreadOBJ | 00153a68 | 00156c10 | 00174a30 | 001785c8 | 001ac710 |
| State | a220 | b220 | 1800220 | 4220 | 4220 |
| PreEmptive GC | Enabled | Enabled | Enabled | Enabled | Enabled |
| GC Alloc Context | 00000000: 00000000 | 00000000: 00000000 | 00000000: 00000000 | 00000000: 00000000 | 00000000: 00000000 |
| Domain | 00166030 | 00166030 | 00166030 | 00166030 | 001a6918 |
| Lock Count | 0 | 0 | 0 | 0 | 2 |
| Apt | MTA | MTA | MTA | STA | STA |
| Exception | (Finalizer) | (Threadpool Worker) | (GC) |
The following tables decode the state field for each managed thread:
Table 4.8: Breakdown of state for thread 1, state value = 0xa220
| Symbol | Value | Description |
|---|---|---|
| TS_InMTA | 0x00008000 | Thread is part of the MTA |
| TS_CoInitialized | 0x00002000 | CoInitialize has been called for this thread |
| TS_Background | 0x00000200 | Thread is a background thread |
| TS_LegalToJoin | 0x00000020 | Indicates whether it is now legal to attempt a Join() |
Table 4.9: Breakdown of state for thread 5, state value = 0xb220
| Symbol | Value | Description |
|---|---|---|
| TS_InMTA | 0x00008000 | Thread is part of the MTA |
| TS_CoInitialized | 0x00002000 | CoInitialize has been called for this thread |
| TS_WeOwn | 0x00001000 | Exposed object initiated this thread |
| TS_Background | 0x00000200 | Thread is a background thread |
| TS_LegalToJoin | 0x00000020 | Indicated whether it is now legal to attempt a Join() |
Table 4.10: Breakdown of state for thread 6, state value = 0x01800220
| Symbol | Value | Description |
|---|---|---|
| TS_TPWorkerThread | 0x01000000 | Indicates that this a threadpool worker thread. (if not, it is a threadpool completionport thread) |
| TS_ThreadPoolThread | 0x00800000 | Indicates that this is a threadpool thread |
| TS_Background | 0x00000200 | Thread is a background thread |
| TS_LegalToJoin | 0x00000020 | Indicates whether it is now legal to attempt a Join() |
Table 4.11: Breakdown of state for threads 16 and 17, state value = 0x4220
| Symbol | Value | Description |
|---|---|---|
| TS_InSTA | 0x00004000 | Thread hosts an STA |
| TS_Background | 0x00000200 | Thread is a background thread |
| TS_LegalToJoin | 0x00000020 | Indicates whether it is now legal to attempt a Join() |
Threads 16 and 17 are STA threads created by COM+, but they are initialized by runtime thread 17, which is waiting for either managed or native work. As seen in the native thread output, Thread 16 called msvcr70!abort, which called mscorwks!CorExitProcess. Then mscorwks!CorExitProcess made a call to mscorwks!ForceEEShutdown, which finally called kernel32!ExitProcess. To examine this and other managed threads more closely, type ~*e !clrstack.
0:000> ~*e !clrstack
Thread 0
Not a managed thread.
Thread 1
ESP EIP
Thread 2
Not a managed thread.
Thread 3
Not a managed thread.
Thread 4
Not a managed thread.
Thread 5
ESP EIP
Thread 6
ESP EIP
Thread 7
Not a managed thread.
Thread 8
Not a managed thread.
Thread 9
Not a managed thread.
Thread 10
Not a managed thread.
Thread 11
Not a managed thread.
Thread 12
Not a managed thread.
Thread 13
Not a managed thread.
Thread 14
Not a managed thread.
Thread 15
Not a managed thread.
Thread 16
ESP EIP
042ff9c8 77e75cb5 [FRAME: ComPlusMethodFrameStandalone] [DEFAULT]
[hasThis] Void DebuggingCOMLib.STAClass.RaiseError(I4,I4)
042ff9dc 03a00e86 [DEFAULT] [hasThis] Void
Debugging.Unexpected.btnSTA_Click(Object,Class System.EventArgs)
042ff9ec 04748b6d [DEFAULT] [hasThis] Void
System.Web.UI.WebControls.Button.OnClick(Class System.EventArgs)
042ffa00 0474894a [DEFAULT] [hasThis] Void
System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandl
er.RaisePostBackEvent(String)
042ffa10 047488fb [DEFAULT] [hasThis] Void
System.Web.UI.Page.RaisePostBackEvent(Class
System.Web.UI.IPostBackEventHandler,String)
042ffa18 0474885a [DEFAULT] [hasThis] Void
System.Web.UI.Page.RaisePostBackEvent(Class
System.Collections.Specialized.NameValueCollection)
042ffa28 04620ac0 [DEFAULT] [hasThis] Void
System.Web.UI.Page.ProcessRequestMain()
042ffa60 03a9f09f [DEFAULT] [hasThis] Void
System.Web.UI.Page.ProcessRequest()
042ffa98 03a9f012 [DEFAULT] [hasThis] Void System.Web.Util.ASPCompat
ApplicationStep.System.Web.HttpApplication+IExecutionStep.Execute()
042ffa9c 03a5a6f0 [DEFAULT] [hasThis] Class System.Exception
System.Web.HttpApplication.ExecuteStep(Class IExecutionStep,ByRef
Boolean)
042ffae4 03a9ef8c [DEFAULT] [hasThis] Void System.Web.Util.ASPCompat
ApplicationStep.ExecuteASPCompat Code()
042ffb10 03a9ee54 [DEFAULT] [hasThis] Void System.Web.Util.ASPCompat
ApplicationStep.OnASPCompat Execution()
042ffb7c 001ec90a [FRAME: ContextTransitionFrame]
Thread 17
ESP EIP
Thread 18
Not a managed thread.
Thread 19
Not a managed thread.
Thread 20
Not a managed thread.
Thread 16 is the only managed thread that is executing .NET code. You can combine Thread 16's managed and native frames together to see how one talks to the other.
16 Id: b64.c5c Suspend: 1 Teb: 7ffaa000 Unfrozen
ChildEBP RetAddr
042ff8e0 7923b594 kernel32!ExitProcess
042ff8ec 792176bb mscorwks!SafeExitProcess+0x38
042ff8f8 79210cdd mscorwks!ForceEEShutdown+0x3f
042ff90c 7917b297 mscorwks!CorExitProcess+0x5a
042ff91c 7c00bb88 mscoree!CorExitProcess+0x3d
042ff924 7c00440a msvcr70!__crtExitProcess+0x25
[f:\vs70builds\9466\vc\crtbld\crt\src\crt0dat.c @ 451]
042ff930 7c00f791 msvcr70!doexit+0xa9
[f:\vs70builds\9466\vc\crtbld\crt\src\crt0dat.c @ 402]
042ff940 7c012ac8 msvcr70!_exit+0xe
[f:\vs70builds\9466\vc\crtbld\crt\src\crt0dat.c @ 302]
042ff988 7c012799 msvcr70!raise+0xae
[f:\vs70builds\9466\vc\crtbld\crt\src\winsig.c @ 509]
042ff994 10005877 msvcr70!abort+0xe
[f:\vs70builds\9466\vc\crtbld\crt\src\abort.c @ 48]
Thread 16
ESP EIP
042ff9c8 77e75cb5 [FRAME: ComPlusMethodFrameStandalone] [DEFAULT]
[hasThis] Void DebuggingCOMLib.STAClass.RaiseError(I4,I4)
042ff9dc 03a00e86 [DEFAULT] [hasThis] Void
Debugging.Unexpected.btnSTA_Click (Object,Class System.EventArgs)
042ff9ec 04748b6d [DEFAULT] [hasThis] Void
System.Web.UI.WebControls.Button.OnClick(Class System.EventArgs)
042ffa00 0474894a [DEFAULT] [hasThis] Void
System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandl
er.RaisePostBackEvent(String)
042ffa10 047488fb [DEFAULT] [hasThis] Void
System.Web.UI.Page.RaisePostBackEvent(Class
System.Web.UI.IPostBackEventHandler,String)
042ffa18 0474885a [DEFAULT] [hasThis] Void
System.Web.UI.Page.RaisePostBackEvent(Class
System.Collections.Specialized.NameValueCollection)
042ffa28 04620ac0 [DEFAULT] [hasThis] Void
System.Web.UI.Page.ProcessRequestMain()
042ffa60 03a9f09f [DEFAULT] [hasThis] Void
System.Web.UI.Page.ProcessRequest()
042ffa98 03a9f012 [DEFAULT] [hasThis] Void System.Web.Util.ASPCompat
ApplicationStep.System.Web.HttpApplication+IExecutionStep.Execute(
)
042ffa9c 03a5a6f0 [DEFAULT] [hasThis] Class System.Exception
System.Web.HttpApplication.ExecuteStep(Class IExecutionStep,ByRef
Boolean)
042ffae4 03a9ef8c [DEFAULT] [hasThis] Void System.Web.Util.ASPCompat
ApplicationStep.ExecuteASPCompat Code()
042ffb10 03a9ee54 [DEFAULT] [hasThis] Void System.Web.Util.ASPCompat
ApplicationStep.OnASPCompat Execution()
042ffb7c 001ec90a [FRAME: ContextTransitionFrame]
...
this is an extract of the output from !dumpstack to show the starting frames of
the thread's call stack
042fff2c 75782f91 comsvcs!CSTAThread::ProcessQueueWork+0x32, calling
comsvcs!CSTAThread::DoWork
042fff44 75783251 comsvcs!CSTAThread::WorkerLoop+0x17f, calling
comsvcs!CSTAThread::ProcessQueueWork
042fff7c 77c2ab33 msvcrt!free+0xc8, calling msvcrt!_SEH_epilog
042fff80 77c37fb8 msvcrt!_endthreadex+0xa0
042fff8c 77f52dbb ntdll!RtlAllocateHeap+0x81f, calling
ntdll!RtlpUpdateIndexInsertBlock
042fffb4 77e802ed kernel32!BaseThreadStart+0x37
Note
You must change to the appropriate thread before running the !dumpstack command. For example, to change to thread 16, execute ~16s.
When you use ASPCompat, the process terminates on one thread instead of multiple threads. In the previous example, the call from Debugging.Unexpected.btnSTA_Click() to DebuggingCOMLib.STAClass.RaiseError() occurred on thread 1, and the call to the msvcr70!abort() and ultimately kernel32!ExitProcess() occurred on thread 10. Here, all calls take place on thread 16 because ASPCompat forces the ProcessRequest method to execute in a thread that has been initialized to the STA threading model.
For the reasons shown above, Microsoft recommends that you use the supported scenario of ASPCompat when calling COM components marked Apartment from ASP.NET pages. This scenario has proven to perform better and be more reliable in Microsoft stress labs.
Another server-based technology, XML Web services, doesn't support ASPCompat. See article Q303375, "INFO: XML Web Services and Apartment Objects" in the Microsoft Knowledge Base at https://support.microsoft.com/default.aspx?scid=kb;en-us;Q303375. The component must be registered in COM+ as a library application, and Synchronization must be set to Required so that it runs on a COM+ STA worker thread similarly to the examples above.
Exploring Further
This walkthrough includes code to call COM components marked either Apartment or Free. Try instantiating other components marked either Both or Neutral. This can be accomplished by creating DebuggingComLib, BothClass, or DebuggingComLib.NeutralClass objects from the ASP.NET pages.
Try these ideas to learn more:
- Consider how the walkthrough results differ if you click the second button-click event, which calls an MTA COM object.
- Create an unconfigured COM component marked Apartment with a method that calls Sleep(). Then make another request to create another component instance of the same type and see if the object creation fails or succeeds.
Conclusion
This walkthrough has shown you how to use Microsoft tools, in particular WinDbg, the SOS extension DLL, and ADPlus, to investigate a crash in an ASP.NET application. In this scenario, the crash was caused by a call to ExitProcess, and as a result the process exited.
Using the dump files produced by ADPlus, you have discovered how to find information about the threads executing within a process, and about how to use call-stack information to determine which methods are executing on each thread.
You have also learned about the ASPCompat attribute, which provides better and more reliable performance when calling COM components marked Apartment from ASP.NET pages.