Production Debugging for .NET Framework Applications

 

Debugging Contention Problems

November 2002

Summary: This chapter describes how to approach debugging when ASP.NET fails to respond to browser requests. Causes may include a long-running request, or threads that deadlock or contend for a resource. Focusing on the latter, a walkthrough shows how to debug in a scenario with some controls that wait on a shared event. To set the scene, "ASP.NET Thread Pools" discusses .NET threading, the common language runtime thread pool, and how ASP.NET uses the thread pool. After the WinDbg walkthrough, there is an example that uses the Microsoft® Visual Studio® .NET debugger. For information about using CorDbg, see the Appendix.

Although reproducing this problem on your machine may not give you the exact same results because of differing operating systems and runtime versions, the output should be similar, and the debugging concepts are the same. Also, because all .NET Framework applications use the same memory architecture, you can apply what you learn about memory management in an ASP.NET context to other .NET architectures, such as console applications, Microsoft Windows® operating system applications, and Windows services.

Contents

ASP.NET Thread Pools

Scenario: Contention or Deadlock Symptoms

Conclusion

ASP.NET Thread Pools

ASP.NET takes advantage of the .NET thread pool, thus efficiently using a pool of threads managed by the common language runtime. This section briefly describes the building blocks that ASP.NET needs to use that thread pool. First, managed threads and application domains (or AppDomains) are defined. Then the .NET thread pool, the ThreadPool manager, and various thread types involved are discussed. Understanding the thread types helps you to interpret the dump files shown in the subsequent walkthrough. As the information unfolds, you see how ASP.NET uses the .NET thread pool and handles synchronous and asynchronous calls.

Managed Threads and AppDomains

.NET threading builds upon the existing implementation of native threads and processes by using managed threads and AppDomains. Managed threads are .NET common-language runtime classes that are bound to native threads. Not all native threads in a process can run managed code; consequently, managed threads are a subset of the native threads in a process. AppDomains can be thought of as lightweight processes that isolate the managed code running in them. A native process can contain multiple AppDomains, and each AppDomain contains assemblies and managed threads. For more information, read the .NET Framework SDK Help entries for System.AppDomain and System.Theading.Thread. Figure 3.1 shows how multiple threads can exist in an AppDomain, and how a process consists of one or more AppDomains.

Figure 3.1. A managed process

The .NET ThreadPool

A thread pool is a mechanism that allows the queuing and processing of work items using a dynamically load-balanced group of threads. Using a thread pool might be more efficient than creating a thread for each individual work item because the time needed to create, start, and tear down the thread when it exits is relatively expensive. Note that there is one thread pool per AppDomain.

In addition, managing the thread's lifetime and deciding when to add new threads can be a complex task. .NET provides built-in support for using a thread pool from managed code. Developers can write code to implement callback functions that process one work item at a time using the thread pool. For more information, see "Thread Pooling" in the .NET Framework Developer's Guide on the MSDN Web site at https://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpguide/html/cpconthreadpooling.asp?frame=true.

.NET implements the entire pooling and load-balancing mechanism using the native ThreadPoolMgr class. The ThreadPoolMgr class has five main thread types, and only the first two of these can run managed code. The ThreadPoolMgr types are:

• Completion port thread
• Worker thread
• Gate thread
• Wait thread
• Timer thread

A completion port thread executes the callback function after an item is queued by QueueUserWorkItem() (for more details, see the .NET Framework SDK). If the thread pool manager determines that there are not enough completion port threads available, the work item is sent to the work request queue. A worker thread services the work request queue, dequeues items, and executes the work item. To ensure that there are enough free worker threads available, a gate thread monitors the health of the thread pool. The gate thread is responsible for injecting and retiring worker and input/output (I/O) threads based on factors such as CPU utilization, garbage collection frequency, and work queue size.

The thread pool manager uses a wait thread to wait on a synchronization object. To use a wait thread, a developer can call ThreadPool.RegisterWaitForSingleObject(). This function attempts to find a wait thread and creates one if it does not exist. A common example of .NET Framework base classes that uses wait threads to monitor synchronization objects is the System.Threading.WaitHandle class.

.NET uses timer threads internally to implement the timer callback functionality of the System.Threading.Timer class.

ASP.NET Thread Pool

The wmain() function marks the beginning and end of the Aspnet_wp.exe process execution. As the entry point into Aspnet_wp.exe, the wmain() function first initializes the .NET thread pool. To do this, wmain() calls functions that choose the common language runtime flavor (which can be either workstation or server), load the common language runtime, and obtain an interface to the thread pool interface. Once this function obtains the interface for the thread pool, it can configure the number of completion port threads and the number of worker threads by querying the <processModel> and setting the appropriate values for maxIoThreads and maxWorkerThreads.

Once the process initializes the common language runtime and the thread pool, the wmain() function sets up the named pipes that it uses to communicate with Microsoft Internet Information Services (IIS). ASP.NET uses two named pipes to communicate with IIS: a synchronous pipe and an asynchronous pipe. The synchronous pipe makes blocking calls back to IIS. It is primarily used as a channel to obtain Internet Server Application Programming Interface (ISAPI) server variables from IIS.

The primary function of the asynchronous pipe is to transport requests from IIS to ASP.NET. In order to accomplish this, the initialization routine links the asynchronous pipe handle to a completion port callback within Aspnet_isapi.dll. When work arrives for this pipe, the .NET common language runtime calls the function called aspnet_isapi!CorThreadPoolCompletionCallback(). This function can be seen when viewing the native stack trace of multiple threads in the debugger.

CorThreadPoolCompletionCallback() invokes the callback function using the parameters passed. In the asynchronous pipe manager class, a completion callback function calls a function to process the message sent from IIS. The message processing function examines the request and locates the AppDomain that the request should be routed to. Once located, the AppDomain is queried for a pointer to the managed HttpRuntime object that corresponds to the current AppDomain. Once a pointer is obtained, the HttpRuntime.ProcessRequest() function is called and the request enters managed execution.

The HttpRuntime object processes the worker request. It decides whether it will queue the request for a worker thread or execute the request on the current thread that it is running. It does this based on conditions such as "Is this request a local request?", "Are there free completion port threads available?", and "Is this the first request to this AppDomain?"

If this is the first request, the ASP.NET request queue needs to be initialized based on the values for minFreeThreads, minLocalRequestFreeThreads, and appRequestQueueLimit. These values are specified in the machine.config or web.config files for the Web application.

Scenario: Contention or Deadlock Symptoms

Now that you understand some details of .NET threading and the thread pool, and how ASP.NET uses those services provided by the common language runtime, let's look at how to debug cases where ASP.NET fails to respond to browser requests. At first, debugging "hang" symptoms appears to be nebulous. You usually don't know the immediate cause of the hang, so it's more difficult to determine what led to the results. You look at the big picture first and then narrow down the possibilities. This scenario considers what aspects of the managed code could be causing this behavior, looks at both the native and managed threads, and then focuses on the latter. Tools such as System Monitor (known as Performance Monitor in Windows 2000) may help diagnose the problem, but to debug the problem, analyzing the dump file and the source code proves most efficient.

Sometimes you may observe behavior that displays deadlock symptoms (for example, when requests do not complete because they are waiting on external resources). However, a true deadlock occurs when one thread obtains a lock and requests another lock that has already been obtained by a second thread. The second thread is waiting for the lock that the first thread owns. What a dilemma! Neither thread can execute. Both threads stop responding and appear to hang. This walkthrough focuses on problems that may not be "true" deadlock scenarios.

Note: For more information on deadlocks, see article Q317723, "What Are Race Conditions and Deadlocks?" in the Microsoft Knowledge Base at https://support.microsoft.com/default.aspx?scid=kb;en-us;Q317723.

Here are some real customer scenarios that could indicate contention problems:

• Scenario 1: A customer hosts ASP.NET sites. Their client says that pages are not returning, and/or they are monitoring the application event logs and notice that they are continually getting errors because the Aspnet_wp.exe process is recycled.
• Scenario 2: There is one ASP.NET application on one server. The application is facing deadlock symptoms when customers run a particular function on a page (for example, clicking the logon button on the logon page). Some internal logic causes the page to be unavailable.

Breaking Down the Thought Processes

The following flowchart presents common thought processes that you can follow when troubleshooting resource contention problems.

Figure 3.2. Thought flowchart

Many questions can arise. Are you dealing with hanging requests or processes? Will you be notified? With ASP.NET, you should see an error in the application event log. With other technologies, you may or may not see an error posted to the log.

Although the thought processes are illustrated in the walkthrough that follows, let's look at the flowchart in more detail.

Error in the Browser and/or Application Event Log

Errors may surface in both the application event logs and the browser. The responseDeadlockInterval attribute is a configuration setting in the machine.config file set to the default interval of three minutes. If the Aspnet_wp.exe process has deadlocked and the three-minute time limit has passed, the deadlock-detection mechanism in IIS recycles the process and posts the following error in the application event log: "aspnet_wp.exe was recycled because it was suspected to be in a deadlocked state. It did not send any responses for pending requests in the last 180 seconds." A Server Application Unavailable error is often received by the browser because Aspnet_wp.exe is not responding.

Deadlock Error Occurs: Apply .NET Framework Service Pack 2

When a deadlock error surfaces, make sure that the .NET Framework Service Pack 2 (SP2) has been applied. SP2 addresses a problem that causes the ASP.NET health monitor to think a process is executing when it's not. This service pack fixes the request processing logic so there isn't a phantom request.

For more information, see article Q321792, "ASP.NET Worker Process (Aspnet_wp.exe) Is Recycled Unexpectedly*"* in the Microsoft Knowledge Base at https://support.microsoft.com/default.aspx?scid=kb;en-us;Q321792.

Deadlock Error Still Occurs

If the .NET Framework SP2 has been applied and the error still occurs, this means that the ASP.NET process is hung and you need to get a dump file. If the project is in a controlled environment (development or stress mode) and you can reproduce the problem, debugging choices include performing a live debug with CorDbg, Content Debugger (CDB), and SOS.dll, or remotely using Visual Studio .NET.

If the project is in a production environment and you can reproduce the problem, you can run Autodump+ (ADPlus) in–hang mode to produce a full memory dump of the Aspnet_wp.exe process. The problem with this approach is that you may not know how much time will pass before the error happens. Also, once the error does happen, you only have three minutes to run ADPlus. To allow more time to break in on the problem, you can extend the responseDeadlockInterval attribute of the <processModel> element in machine.config to a longer time period (maybe nine minutes instead of three). Keep in mind that the longer the time period for responseDeadlockInterval, the longer a deadlocked server can appear to be unavailable. Choose a time period that makes sense for the application. Also, if you want to disable the process model deadlock detection mechanism, set responseDeadlockInterval and responseRestartDeadlockInterval to infinite.

.NET Framework version 1.1 includes a DebugOnDeadlock registry setting that breaks into Aspnet_wp.exe and/or InetInfo.exe. This makes it easier to run ADPlus in –crash mode and create a dump file (the DebugOnDeadlock setting would be 1). This feature is also available through a hotfix. For more information, see article Q325947, "Event ID 1003 with ASP.NET Deadlock" in the Microsoft Knowledge Base at https://support.microsoft.com/default.aspx?scid=kb;en-us;Q325947.

If you can't apply this hotfix, an alternative is to use ADPlus_AspNet.vbs to create a full dump when a deadlock occurs. ADPlus_AspNet.vbs is a customized version of ADPlus that sets a breakpoint and creates a dump file before the process is killed. In this example, AdPlus_AspNet.vbs runs in -crash mode when the process is told to recycle by the process model health monitoring.

If you use ADPlus through Terminal Server, consider the following: A native debugger cannot attach to a process running in another WinStation. Consequently, ADPlus cannot run in -crash mode because it scripts the CDB.exe debugger. Instead, you can use Task Scheduler to create a Remote.exe command window that has access to the system processes. For more information, see article Q323478, "You Cannot Debug Through a Terminal Server Session" in the Microsoft Knowledge Base at https://support.microsoft.com/default.aspx?scid=kb;en-us;Q323478.

Process Appears to Hang, but There Is No Error

In many scenarios, you may have deadlock symptoms, but no error. For example, the server may hang and not respond. If this is the case, follow normal troubleshooting procedures first. For example, ask what else is not working. Is the request blocked in IIS or is it blocked because of third-party code running in InetInfo? If it is one of these problems, create a dump file and either examine it, or send it to Microsoft Product Support to examine.

If the problem is that the ASP.NET pages are not rendering, consider the reasons why an error message does not show. Here are some possibilities:

• ASP.NET may not know that the deadlock symptoms occurred because the worker process has not sent back a response. The process may hang before the time-out occurs.
• The debugger attached to the Aspnet_wp.exe process and disabled the deadlock detection.
• The responseDeadlockInterval was altered and set too high.

Examining the Dump Files

To examine dump files, you need to start WinDbg, set up your symbols as appropriate (for more information, see "Debugging" later in this chapter), and then analyze the dump file. While debugging, examine managed threads, native threads, and call stacks. If threads are processing requests at the time of the dump, examine those threads to discover why the request is not responding.

Using the SOS.dll extension, consider the thread pool information, including CPU utilization and the number of worker and completion port threads available. Identify the calling functions in the stack traces and determine if they are blocked.

Contention Walkthrough

The following walkthrough describes a simplified, realistic scenario. The purpose is to illustrate techniques to help you troubleshoot a contention problem. A production environment scenario may not be as clear cut as the contention walkthrough scenario, but you can apply similar techniques to help identify the causes of contention.

The example that is shown in this walkthrough demonstrates threads waiting on a shared resource. More specifically, this code blocks a thread by calling WaitHandle.WaitOne() with an infinite time-out. The object contains static members, but it could have been shared in session or application scope. For simplicity, all shown requests come from one browser. A more realistic scenario would include requests from multiple browsers and multiple clients.

In this scenario, you will perform the following steps:

1. Browse to the ASP.NET page at https://localhost/debugging/contention.aspx and consider the values displayed in the process model table.
2. Follow the thought processes in the flowchart and find the errors presented both in the browser and the application event log.
3. Create a dump file and examine the dump data using WinDbg and SOS.dll.
4. Examine the native threads, managed threads, and call stacks.
5. Consider which threads are bound to which AppDomain, and the different stages of execution.
6. After you've found evidence in the dumps for possible problem areas in the code, look at the source code.

Baseline View

The following browser view provides a baseline from which to compare data after a deadlock occurs.

Figure 3.3. Baseline browser data

Note the information displayed, especially the data in the table. Some of the fields are self-explanatory, but descriptions are provided in the following table.

Table 3.1: Description of fields in contention.aspx

Field	Description
StartTime	The time at which this Aspnet_wp.exe process started
Age	The length of time the process has been running
ProcessID	The ID assigned to the process
RequestCount	Number of completed requests, which is initially zero
Status	The current status of the process: if Alive, the process is running; if Shutting Down, the process has begun to shut down; if ShutDown, the process has shut down normally after receiving a shut down message from the IIS process; if Terminated, the process was forced to terminate by the IIS process
ShutdownReason	The reason why a process shut down: if Unexpected, the process shut down unexpectedly; if Requests Limit, requests executed by the process exceeded the allowable limit; if Request Queue Limit, requests assigned to the process exceeded the allowable number in the queue; if Timeout, the process restarted because it was alive longer than allowed; if Idle Timeout, the process exceeded the allowable idle time; if Memory Limit Exceeded, the process exceeded the per-process memory limit
PeakMemoryUsed	The maximum amount of memory the process has used. This value maps onto the Private Bytes (maximum amount) count in System Monitor

You can explore the code and read through the comments by opening another instance of Contention.aspx.cs in Visual Studio .NET or Notepad.

The code behind each Obtain Shared Resource button on the user interface calls a function that blocks a thread by a call to WaitHandle.WaitOne() with an infinite time-out. More specifically, the code first creates a ManualResetEvent object in a nonsignaled state. Each time you click an Obtain Shared Resource button, code for SharedResource.Wait() is invoked, which calls ManualResetEvent.WaitOne(). The call to WaitOne() blocks the current thread until the WaitHandle() function receives a signal.

Clicking the Free Shared Resource button invokes SharedResource.Free(), which calls ManualResetEvent.Set(), which in turn sets the state of the specified event to signaled. Because Free() leaves ManualResetEvent in a signaled state, you won't be able to reproduce the blocking scenario. To be able to restart the test, the Free Shared Resource button also calls SharedResource.Reset(), which sets the state of the ManualResetEvent to nonsignaled.

To create and view the error

1. Open a browser window, and then browse to https://localhost/Debugging/contention.aspx.

2. Click each Obtain Shared Resource button in order.

3. After three minutes, you should see an error appear in the browser window and in the application event log.

   The server returns the following error message:

Server Application Unavailable 
  

The web application you are attempting to access on this web server is currently unavailable. Please hit the "Refresh" button in your web browser to retry your request.

Administrator

   

Note: An error message detailing the cause of this specific request failure can be found in the application event log of the Web server. Please review this log entry to discover what caused this error to occur.

1. Close the browser window. The application event log contains the following error, which you can display using the event viewer:

   Event Type:  Error
   Event Source:  ASP.NET 1.0.3705.0
   Event Category:  None
   Event ID:    1003
   Date:    6/5/2002
   Time:    9:32:49 PM
   User:    N/A
   Computer:    machineName
   Description:
   aspnet_wp.exe  (PID: 2248) was recycled because it was suspected to be in a 
   deadlocked state. It did not send any responses for pending requests in the 
   last 180 seconds.
   
   

   More specifically, Aspnet_isapi.dll (running in InetInfo.exe) reports the error that the browser and the application event log display. Requests that wait for a response cause the error.

2. Open a new browser window, and then browse to https://localhost/debugging/contention.aspx.

3. Click each Obtain Shared Resource button again.

The data in your browser should resemble the following table:

Table 3.2: Contention.aspx results

Field	Baseline value	New value
StartTime	07/08/2002 16:58:51	07/08/2002 17:05:32
Age	00:00:02.0588432	00:06.43.0472853
ProcessID	2,792	2,792
RequestCount	0	4
Status	Alive	Terminated
ShutdownReason	N/A	Deadlock Suspected
PeakMemoryUsed	10,656	11,248

The RequestCount increments to four—one to browse the page for viewing and then three for the button clicks. The buttons are ASP.NET server controls that execute code on the server. Every button click is a round trip to the server, and you can see this happening when you examine the call stacks later in this walkthrough.

The ShutdownReason field shows that a deadlock is suspected. As described in the "IIS 5. x Process Model" section of Chapter 1, "Introduction to Production Debugging for .NET Framework Applications," the process model health monitoring launches a new Aspnet_wp.exe process and terminates the old process.

Before troubleshooting further, restart IIS (by running IISReset.exe) in the command window to reinitialize the PMInfo counters.

Debugging a Dump File with WinDbg

The goal is to diagnose why the process was restarted. There may have been queued requests and/or a lack of response during the default three-minute interval set by responseDeadlockInterval. A dump file provides a snapshot of what occurs when the process is recycled. Because the interval may allow minimal time for the creation of the dump file, you need to modify the registry to allow debugging before the process recycles.

The ASP.NET deadlock detection mechanism is automatically disabled when a native debugger is attached to Aspnet_wp.exe, but it would be beneficial to obtain a dump before the process recycles. To circumvent this problem, you must first modify the registry.

Note

   

The following exercise requires that you make changes to the registry. Use caution—it is always recommended that you back up the registry before making changes. In addition, when you have finished debugging, it is recommended that you delete the DWORDs created below. These registry DWORDs are undocumented, and should only be used during specific debugging sessions.

To modify the registry

1. Run Regedit, and then locate the key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ASP.NET.

2. Right-click ASP.NET, and then select New DWORD Value from the shortcut menu.

3. Under Name, change the new New Value #1 entry to UnderDebugger. Keep this data value as zero.

   When UnderDebugger is zero, the deadlock detection mechanism works normally.

Figure 3.4 shows how the new value should appear.

Figure 3.4. New registry key

To create a dump for this exercise, you use a customized version of ADPlus.vbs called ADPlus_AspNet.vbs. The changes to ADPlus are located in the CreateCDBScript() function. This is the function that is run to create the .cfg file that in turn configures the CDB debugger when it attaches to the process.

ADPlus_AspNet.vbs includes the following changes to the ADPlus.vbs script:

objTextFile.Writeline "* -- Setup Symbol Path to use external symbol server ---"
objTextFile.Writeline ".sympath+ SRV*c:\ 
symbols\debugginglabs*https://msdl.microsoft.com/download/symbols;C:\symbols\
debugginglabs"
objTextFile.Writeline "* ------------------------------------------------------"
objTextFile.Writeline "*"
objTextFile.Writeline "*"
objTextFile.Writeline "*"
objTextFile.Writeline "* -- Make sure the symbols are loaded for aspnet_wp.exe ---"
objTextFile.Writeline "x aspnet_wp!CAsyncPipeManager::Close"
objTextFile.Writeline "* ------------------------------------------------------------"
objTextFile.Writeline "*"
objTextFile.Writeline "*"
objTextFile.Writeline "*"
    
'Since this is a breakpoint, any \ need to be escaped with \\
Dim EscapedCrashDir
EscapedCrashDir = Replace(ShortCrashDir, "\", "\\")
    
objTextFile.Writeline "* -- Add a breakpoint for CAsyncPipeManager::Close ---"
objTextFile.Writeline "bp aspnet_wp!CAsyncPipeManager::Close " & Chr(34) & _ 
".echo Encountered aspnet_wp!CAsyncPipeManager::Close breakpoint. Dumping process 
...;.dump /mfh "  _ 
& EscapedCrashDir & "\\" & "PID-" & pid & "__" & packagename & 
"__Recycle__full.dmp;q" & Chr(34)
  

The changes to ADPlus.vbs show how easy it is to adapt this file to your needs. If you do make these changes, a good practice is to rename the new script file to differentiate it from ADPlus.vbs.

Make sure that ADPlus_AspNet.vbs is located in the folder where you installed the debugging toolkit; by default, this is C:\Debuggers. Symbols are required, so make sure that you are connected to the Internet or that you have downloaded the symbols for ASP.NET locally, especially Aspnet_wp.pdb. Internet access is adequate because this tool sets up the path to the Microsoft external symbol server for you.

As noted in the changes, ADPlus_AspNet.vbs sets the symbol path to use the Microsoft public symbol server. The tool verifies that the symbols can be loaded for aspnet_wp!CAsyncPipeManager::Close.

This function is called when the process is shut down because of one of the conditions specified by the process model (for deadlocks). The tool sets a breakpoint on aspnet_wp! CAsyncPipeManager::Close() to create a full dump file when the breakpoint is hit.

Note

   

The WinDbg and CDB debuggers can set breakpoints on addresses, function names, or source file lines. The syntax to set a breakpoint on a function is bp module_or_exe_name!function_name.

Using this automated version of ADPlus is easier in development mode or in a controlled environment, such as stress testing. If you did not use this automated version, you would have to monitor the Aspnet_wp.exe process continually until the recycle occurs. In that case, you would run ADPlus in -hang mode, and timing would be tricky.

The use of the customized ADPlus script is also easier to use in production mode. You don't have to continually monitor the process; it does this for you, and it creates a dump just before the process terminates. However, if you run in -crash mode, which is invasive, the process is terminated when the debugger exits.

Along with a user dump, the ADPlus_AspNet.vbs script also generates a text file of the exception history, dumping the native stack traces for those exceptions (like traditional ADPlus.vbs). The default configuration of ADPlus means that for every exception, a minidump file is generated and a stack trace is logged. Obviously, when more exceptions are thrown, a larger amount of text is generated. Users have been known to run out of disk space when running this tool.

If running in production mode, throwing as few exceptions as possible is recommended. You can even disable the creation of the log file if it is too obtrusive. For more information, see "ASP.NET Performance Tips and Best Practices" on the GotDotNet Web site at https://code.msdn.microsoft.com/GotDotNet.aspx.

To start the exercise again

1. Open a command prompt window, and then type iisreset at the prompt to restart IIS.

2. Open a new browser window, and then browse to https://localhost/debugging/contention.aspx.

3. After Aspnet_wp.exe loads, run ADPlus_AspNet.vbs against Aspnet_wp.exe in -quiet mode at the command line using the following command:

   adplus_aspnet.vbs -pn aspnet_wp.exe -crash -quiet
   
   

   Note

      

   The -quiet switch ensures that you can still create a dump file if the symbol path for ADPlus is not set.

You should see the following output:

C:\debuggers>adplus_aspnet.vbs -pn aspnet_wp.exe -crash -quiet
The '-crash' switch was used, Autodump+ is running in 'crash' mode.
The '-quiet' switch was used, Autodump+ will not display any modal dialog boxes.

Monitoring the processes that were specified on the command line
for unhandled exceptions.
----------------------------------------------------------------------
Attaching the CDB debugger to: Process - ASPNET_WP.EXE with PID–360
  

You should also see a minimized window running Cdb.exe. The tool waits until the deadlock is detected and then creates the dump and text files. As noted in the "IIS 5.x Process Model" discussion on ASP.NET architecture in Chapter 1, ASP.NET has a standard ISAPI extension (Aspnet_isapi.dll) that runs in-proc within InetInfo.exe and uses named pipes to connect to an ASP.NET worker process (Aspnet_wp.exe). When a deadlock is detected, InetInfo.exe closes down the named pipes to Aspnet_wp.exe. When this pipe is closed, the custom breakpoint on CAsychPipeManager::Close is hit. The breakpoint command is then executed and the log entry and a full dump file is generated.

To create the dump file

1. Switch to the browser that displays Contention.aspx and click each Obtain Shared Resource button in order.

2. Wait three minutes for the browser errors and event-log errors to be generated. When the debugger window disappears, the full dump is created.

   You can check the debugging toolkit installation folder for the most recent \Crash_Mode folder. The file's name will be similar to PID-360__ASPNET_WP.EXE__Recycle__full.dmp.

To examine the dump file using WinDbg

1. On the Start menu, point to Debugging Tools for Windows, and then click WinDbg.
2. On the File menu, click Open Crash Dump.
3. Select the appropriate dump file, and then click Open.
4. If you are prompted to "Save Base Workspace Information," click No.
5. If a Disassembly window pops up, close the window, and then on the Window menu, click Automatically Open Disassembly.

Symbol paths must be entered for the files that are used by the debugger to analyze the dump file. The symbol file versions that are needed on the debug computer should match the version on the system that produced the dump file. Include symbols from the .NET Framework SDK, Visual Studio .NET, and symbols shipped with the following System32 folder: \%WINDIR%\system32.

To enter the symbol paths, do one of the following:

• From the WinDbg command line, type:

  .sympath SRV*C:\symbols\debugginglabs*https://msdl.microsoft.com/download/symbols;C:\
  symbols\debugginglabs;C:\Program Files\Microsoft Visual Studio 
  .NET\FrameworkSDK\symbols;C:\windows\system32 
  
  

  Note

     

  If you have only installed the .NET Framework SDK (without Visual Studio .NET), then you need to replace C:\Program Files\Microsoft Visual Studio .NET\FrameworkSDK\symbols with C:\Program Files\Microsoft.NET\FrameworkSDK\symbols.

• Create a new environment variable called _NT_SYMBOL_PATH with a value of:

  C:\symbols\debugginglabs*https://msdl.microsoft.com/download/symbols; 
  C:\symbols\debugginglabs;C:\Program Files\Microsoft Visual Studio 
  .NET\FrameworkSDK\symbols;C:\windows\system32
  
  

• On the File menu in WinDbg, click Symbol File Path, and then type:

  SRV*C:\symbols\debugginglabs*https://msdl.microsoft.com/download/symbols;C:\
  symbols\debugginglabs;C:\Program Files\Microsoft Visual Studio 
  .NET\FrameworkSDK\symbols;C:\windows\system32
  
  

The "SRV" in the path tells WinDbg to go to an external symbol server and copy symbols into the local symbol cache.

To use these same symbol paths for other dumps

• On the File menu in WinDbg, click Save Workspace As, and then type a name for the saved paths, such as**.NET Debugging Symbols**.

Examining Native Data

First, consider how many threads you are dealing with. In WinDbg, type tilde (~) in the command line; this displays all threads in the current process. Here you have twelve threads, but you don't know yet how many are native and how many are managed.

Each thread is in Suspend state, and is unfrozen. The WinDbg online Help explains that "Each thread has a suspend count associated with it. If it is one or higher, the system will not run the thread. If it is zero or lower, the system will run the thread when appropriate." The following are the twelve threads:

0:000> ~
.  0  id: 168.248   Suspend: 1 Teb 7ffde000 Unfrozen
   1  id: 168.420   Suspend: 1 Teb 7ffdd000 Unfrozen
   2  id: 168.ab0   Suspend: 1 Teb 7ffdc000 Unfrozen
   3  id: 168.a18   Suspend: 1 Teb 7ffdb000 Unfrozen
   4  id: 168.9c4   Suspend: 1 Teb 7ffda000 Unfrozen
   5  id: 168.d84   Suspend: 1 Teb 7ffd9000 Unfrozen
   6  id: 168.2ec   Suspend: 1 Teb 7ffd8000 Unfrozen
   7  id: 168.7b0   Suspend: 1 Teb 7ffd7000 Unfrozen
   8  id: 168.3c8   Suspend: 1 Teb 7ffd6000 Unfrozen
   9  id: 168.a4    Suspend: 1 Teb 7ffd5000 Unfrozen
  10  id: 168.eb8   Suspend: 1 Teb 7ffd4000 Unfrozen
  11  id: 168.11c   Suspend: 1 Teb 7ffaf000 Unfrozen
  12  id: 168.894   Suspend: 1 Teb 7ffae000 Unfrozen
  13  id: 168.e94   Suspend: 1 Teb 7ffad000 Unfrozen
  

The WinDbg online Help also says "A thread can also be frozen by the debugger. This is similar to suspending the thread in some ways. However, it is solely a debugger setting; nothing in Windows itself will recognize anything different about this thread. All threads are unfrozen by default. When the debugger causes a process to execute, threads that are frozen will not execute. However, if the debugger detaches from the process, all threads will unfreeze."

Look at the native call stacks for all threads that were running when the dump occurred. The call stack is a data structure that tracks function calls and the parameters passed into these functions. The ~*k command shows 20 stack frames by default, the ~* command displays all threads, and the k command displays the stack frame of the given thread.

Type ~*k 200 to display the entire call stacks for all twelve threads (given that each has under 200 stack frames).

0:000> ~*k 200

.  0  id: 168.248   Suspend: 1 Teb 7ffde000 Unfrozen
ChildEBP RetAddr  
0012f874 77f7e76f SharedUserData!SystemCallStub+0x4
0012f878 77e775b7 ntdll!NtDelayExecution+0xc
0012f8d0 77e61bf1 kernel32!SleepEx+0x61
0012f8dc 00422c17 kernel32!Sleep+0xb
0012ff44 004237db aspnet_wp!wmain+0x30b [e:\dna\src\xsp\wp\main.cxx @ 
222]
0012ffc0 77e7eb69 aspnet_wp!wmainCRTStartup+0x131 
[f:\vs70builds\9111\vc\crtbld\crt\src\crtexe.c @ 379]
0012fff0 00000000 kernel32!BaseProcessStart+0x23

   1  id: 168.420   Suspend: 1 Teb 7ffdd000 Unfrozen
ChildEBP RetAddr  
0086f7ac 77f7f49f SharedUserData!SystemCallStub+0x4
0086f7b0 77e74bd8 ntdll!ZwWaitForMultipleObjects+0xc
0086f84c 79281971 kernel32!WaitForMultipleObjectsEx+0x12c
0086f87c 79282444 mscorwks!Thread::DoAppropriateWaitWorker+0xc1
0086f8d0 79317c0a mscorwks!Thread::DoAppropriateWait+0x46
0086f918 036545f4 mscorwks!WaitHandleNative::CorWaitOneNative+0x6f
0086f9ec 03c6c90f 0x36545f4
0086fa24 03c6c373 0x3c6c90f
0086fa80 03daf66b 0x3c6c373
0086fac8 03daf543 0x3daf66b
0086fb20 0377f42d 0x3daf543
0086fba4 792e0069 0x377f42d
0086fbe4 792830c1 mscorwks!ComCallMLStubCache::CompileMLStub+0x1af
0086fc2c 79260b19 mscorwks!Thread::DoADCallBack+0x5c
0086fc94 00cea179 mscorwks!ComCallMLStubCache::CompileMLStub+0x2c2
0086ff24 0042218c 0xcea179
0086ff40 7a13ecc2 aspnet_wp!CAsyncPipeManager::ProcessCompletion+0x1d4 
[e:\dna\src\xsp\wp\asyncpipemanager.cxx @ 516]
0086ff84 792cf905 aspnet_isapi!CorThreadPoolCompletionCallback+0x41 
[e:\dna\src\xsp\isapi\threadpool.cxx @ 773]
0086ffb4 77e802ed mscorwks!ThreadpoolMgr::CompletionPortThreadStart+0x93
0086ffec 00000000 kernel32!BaseThreadStart+0x37

   2  id: 168.ab0   Suspend: 1 Teb 7ffdc000 Unfrozen
ChildEBP RetAddr  
0096ff04 77f7e76f SharedUserData!SystemCallStub+0x4
0096ff08 77e775b7 ntdll!NtDelayExecution+0xc
0096ff60 77e61bf1 kernel32!SleepEx+0x61
0096ff6c 792d00bc kernel32!Sleep+0xb
0096ffb4 77e802ed mscorwks!ThreadpoolMgr::GateThreadStart+0x4c
0096ffec 00000000 kernel32!BaseThreadStart+0x37

   3  id: 168.a18   Suspend: 1 Teb 7ffdb000 Unfrozen
ChildEBP RetAddr  
00c6fefc 77f7f4af SharedUserData!SystemCallStub+0x4
00c6ff00 77e7788b ntdll!NtWaitForSingleObject+0xc
00c6ff64 77e79d6a kernel32!WaitForSingleObjectEx+0xa8
00c6ff74 0042289c kernel32!WaitForSingleObject+0xf
00c6ff80 7c00fbab aspnet_wp!DoPingThread+0x10 [e:\dna\src\xsp\wp\main.cxx 
@ 412]
00c6ffb4 77e802ed MSVCR70!_threadstart+0x6c 
[f:\vs70builds\9466\vc\crtbld\crt\src\thread.c @ 196]
00c6ffec 00000000 kernel32!BaseThreadStart+0x37

   4  id: 168.9c4   Suspend: 1 Teb 7ffda000 Unfrozen
ChildEBP RetAddr  
0101fe80 77f7f49f SharedUserData!SystemCallStub+0x4
0101fe84 77e74bd8 ntdll!ZwWaitForMultipleObjects+0xc
0101ff20 77e74c70 kernel32!WaitForMultipleObjectsEx+0x12c
0101ff38 791cccf6 kernel32!WaitForMultipleObjects+0x17
0101ffa0 791ccc6b mscorwks!DebuggerRCThread::MainLoop+0x90
0101ffac 791ccc1e mscorwks!DebuggerRCThread::ThreadProc+0x55
0101ffb4 77e802ed mscorwks!DebuggerRCThread::ThreadProcStatic+0xb
0101ffec 00000000 kernel32!BaseThreadStart+0x37

   5  id: 168.d84   Suspend: 1 Teb 7ffd9000 Unfrozen
ChildEBP RetAddr  
031bfe98 77f7f49f SharedUserData!SystemCallStub+0x4
031bfe9c 77e74bd8 ntdll!ZwWaitForMultipleObjects+0xc
031bff38 77e74c70 kernel32!WaitForMultipleObjectsEx+0x12c
031bff50 791d3f35 kernel32!WaitForMultipleObjects+0x17
031bff70 791cdd05 mscorwks!WaitForFinalizerEvent+0x5a
031bffb4 77e802ed mscorwks!GCHeap::FinalizerThreadStart+0x96
031bffec 00000000 kernel32!BaseThreadStart+0x37

   6  id: 168.2ec   Suspend: 1 Teb 7ffd8000 Unfrozen
ChildEBP RetAddr  
0353ff1c 77f7f4af SharedUserData!SystemCallStub+0x4
0353ff20 77e7788b ntdll!NtWaitForSingleObject+0xc
0353ff84 77e79d6a kernel32!WaitForSingleObjectEx+0xa8
0353ff94 792cf5dc kernel32!WaitForSingleObject+0xf
0353ffb4 77e802ed mscorwks!ThreadpoolMgr::WorkerThreadStart+0x2e
0353ffec 00000000 kernel32!BaseThreadStart+0x37

   7  id: 168.7b0   Suspend: 1 Teb 7ffd7000 Unfrozen
ChildEBP RetAddr  
0363f7b0 77f7f49f SharedUserData!SystemCallStub+0x4
0363f7b4 77e74bd8 ntdll!ZwWaitForMultipleObjects+0xc
0363f850 79281971 kernel32!WaitForMultipleObjectsEx+0x12c
0363f880 79282444 mscorwks!Thread::DoAppropriateWaitWorker+0xc1
0363f8d4 79317c0a mscorwks!Thread::DoAppropriateWait+0x46
0363f91c 036545f4 mscorwks!WaitHandleNative::CorWaitOneNative+0x6f
0363f9ec 03c6c90f 0x36545f4
0363fa24 03c6c373 0x3c6c90f
0363fa80 03daf66b 0x3c6c373
0363fac8 03daf543 0x3daf66b
0363fb20 0377f42d 0x3daf543
0363fba4 792e0069 0x377f42d
0363fbe4 792830c1 mscorwks!ComCallMLStubCache::CompileMLStub+0x1af
0363fc2c 79260b19 mscorwks!Thread::DoADCallBack+0x5c
0363fc94 00cea179 mscorwks!ComCallMLStubCache::CompileMLStub+0x2c2
0363ff24 0042218c 0xcea179
0363ff40 7a13ecc2 aspnet_wp!CAsyncPipeManager::ProcessCompletion+0x1d4 
[e:\dna\src\xsp\wp\asyncpipemanager.cxx @ 516]
0363ff84 792cf905 aspnet_isapi!CorThreadPoolCompletionCallback+0x41 
[e:\dna\src\xsp\isapi\threadpool.cxx @ 773]
0363ffb4 77e802ed mscorwks!ThreadpoolMgr::CompletionPortThreadStart+0x93
0363ffec 00000000 kernel32!BaseThreadStart+0x37

   8  id: 168.3c8   Suspend: 1 Teb 7ffd6000 Unfrozen
ChildEBP RetAddr  
0387ff44 77f7e76f SharedUserData!SystemCallStub+0x4
0387ff48 77e775b7 ntdll!NtDelayExecution+0xc
0387ffa0 792d05b2 kernel32!SleepEx+0x61
0387ffb4 77e802ed mscorwks!ThreadpoolMgr::TimerThreadStart+0x30
03880038 792067d5 kernel32!BaseThreadStart+0x37
00d1203b 000000ff mscorwks!CreateTypedHandle+0x16

   9  id: 168.a4   Suspend: 1 Teb 7ffd5000 Unfrozen
ChildEBP RetAddr  
03eafe24 77f7efff SharedUserData!SystemCallStub+0x4
03eafe28 77cc1ac9 ntdll!NtReplyWaitReceivePortEx+0xc
03eaff90 77cc167e RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0xf6
03eaff94 77cc1505 RPCRT4!RecvLotsaCallsWrapper+0x9
03eaffac 77cc1670 RPCRT4!BaseCachedThreadRoutine+0x64
03eaffb4 77e802ed RPCRT4!ThreadStartRoutine+0x16
03eaffec 00000000 kernel32!BaseThreadStart+0x37

  10  id: 168.eb8   Suspend: 1 Teb 7ffd4000 Unfrozen
ChildEBP RetAddr  
040af7b0 77f7f49f SharedUserData!SystemCallStub+0x4
040af7b4 77e74bd8 ntdll!ZwWaitForMultipleObjects+0xc
040af850 79281971 kernel32!WaitForMultipleObjectsEx+0x12c
040af880 79282444 mscorwks!Thread::DoAppropriateWaitWorker+0xc1
040af8d4 79317c0a mscorwks!Thread::DoAppropriateWait+0x46
040af91c 036545f4 mscorwks!WaitHandleNative::CorWaitOneNative+0x6f
040af9ec 03c6c90f 0x36545f4
040afa24 03c6c373 0x3c6c90f
040afa80 03daf66b 0x3c6c373
040afac8 03daf543 0x3daf66b
040afb20 0377f42d 0x3daf543
040afba4 792e0069 0x377f42d
040afbe4 792830c1 mscorwks!ComCallMLStubCache::CompileMLStub+0x1af
040afc2c 79260b19 mscorwks!Thread::DoADCallBack+0x5c
040afc94 00cea179 mscorwks!ComCallMLStubCache::CompileMLStub+0x2c2
040aff24 0042218c 0xcea179
040aff40 7a13ecc2 aspnet_wp!CAsyncPipeManager::ProcessCompletion+0x1d4 
[e:\dna\src\xsp\wp\asyncpipemanager.cxx @ 516]
040aff84 792cf905 aspnet_isapi!CorThreadPoolCompletionCallback+0x41 
[e:\dna\src\xsp\isapi\threadpool.cxx @ 773]
040affb4 77e802ed mscorwks!ThreadpoolMgr::CompletionPortThreadStart+0x93
040affec 00000000 kernel32!BaseThreadStart+0x37

  11  id: 168.11c   Suspend: 1 Teb 7ffaf000 Unfrozen
ChildEBP RetAddr  
041aff28 004221ab aspnet_wp!CAsyncPipeManager::Close 
[e:\dna\src\xsp\wp\asyncpipemanager.cxx @ 122]
041aff40 7a13ecc2 aspnet_wp!CAsyncPipeManager::ProcessCompletion+0x1f3 
[e:\dna\src\xsp\wp\asyncpipemanager.cxx @ 535]
041aff84 792cf905 aspnet_isapi!CorThreadPoolCompletionCallback+0x41 
[e:\dna\src\xsp\isapi\threadpool.cxx @ 773]
041affb4 77e802ed mscorwks!ThreadpoolMgr::CompletionPortThreadStart+0x93
041affec 00000000 kernel32!BaseThreadStart+0x37

  12  id: 168.894   Suspend: 1 Teb 7ffae000 Unfrozen
ChildEBP RetAddr  
042afe24 77f7efff SharedUserData!SystemCallStub+0x4
042afe28 77cc1ac9 ntdll!NtReplyWaitReceivePortEx+0xc
042aff90 77cc167e RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0xf6
042aff94 77cc1505 RPCRT4!RecvLotsaCallsWrapper+0x9
042affac 77cc1670 RPCRT4!BaseCachedThreadRoutine+0x64
042affb4 77e802ed RPCRT4!ThreadStartRoutine+0x16
042affec 00000000 kernel32!BaseThreadStart+0x37

  13  id: 168.e94   Suspend: 1 Teb 7ffad000 Unfrozen
ChildEBP RetAddr  
00000000 00000000 kernel32!BaseThreadStartThunk
  

WinDbg doesn't display the managed call stack. It displays native call stacks, but some of the managed calls are wrappers for native calls. The middle section of this stack (without correlated modules and functions) represents managed threads. Threads 1, 7, and 10 have managed code on them. You can identify them by the unmapped code regions between the native resolved function names. This is JIT-compiled code, and it doesn't map to an address range for a loaded module. You will extract the managed thread data shortly.

Note

   

Experienced debuggers scan through the threads and examine the threads that are questionable. However, many people wonder how one knows which threads are questionable. The answer is experience. You could examine these threads, conclude that three threads are each waiting and have similar call stacks, and decide to check out only those threads. In this example, however, each thread is examined, and the differences and similarities of the dumps are considered. Then, next time you look at a similar dump, you too might go directly to those irregular threads.

Thread 0 is the ASP.NET main thread as shown by the call to aspnet_wp!wmain(). This thread has initialized the common language runtime, connected the named pipes back to InetInfo.exe, and created the ping thread. Once it has completed the initialization, it loops, waiting for the process to exit.

The listings for threads 7 and 10 look similar to thread 1, which is shown in the following code sample. Each of these managed threads shows a native call to mscorwks!WaitHandleNative::CorWaitOneNative() and appears to be waiting for a response. As noted, the middle section is managed thread information that WinDbg cannot display on its own. You can enlist the help of SOS.dll to interpret this section. Also note that these three threads are completion port threads (see mscorwks!ThreadpoolMgr::CompletionPortThreadStart() below). As noted in the discussion on .NET thread pools at the beginning of this chapter, ASP.NET first runs its pages on I/O completion port threads, and then if they get used up, they run on worker threads.

   1  id: 168.420   Suspend: 1 Teb 7ffdd000 Unfrozen
ChildEBP RetAddr  
0086f7ac 77f7f49f SharedUserData!SystemCallStub+0x4
0086f7b0 77e74bd8 ntdll!ZwWaitForMultipleObjects+0xc
0086f84c 79281971 kernel32!WaitForMultipleObjectsEx+0x12c
0086f87c 79282444 mscorwks!Thread::DoAppropriateWaitWorker+0xc1
0086f8d0 79317c0a mscorwks!Thread::DoAppropriateWait+0x46
0086f918 036545f4 mscorwks!WaitHandleNative::CorWaitOneNative+0x6f
0086f9ec 03c6c90f 0x36545f4
0086fa24 03c6c373 0x3c6c90f
0086fa80 03daf66b 0x3c6c373
0086fac8 03daf543 0x3daf66b
0086fb20 0377f42d 0x3daf543
0086fba4 792e0069 0x377f42d
0086fbe4 792830c1 mscorwks!ComCallMLStubCache::CompileMLStub+0x1af
0086fc2c 79260b19 mscorwks!Thread::DoADCallBack+0x5c
0086fc94 00cea179 mscorwks!ComCallMLStubCache::CompileMLStub+0x2c2
0086ff24 0042218c 0xcea179
0086ff40 7a13ecc2 aspnet_wp!CAsyncPipeManager::ProcessCompletion+0x1d4 
[e:\dna\src\xsp\wp\asyncpipemanager.cxx @ 516]
0086ff84 792cf905 aspnet_isapi!CorThreadPoolCompletionCallback+0x41 
[e:\dna\src\xsp\isapi\threadpool.cxx @ 773]
0086ffb4 77e802ed mscorwks!ThreadpoolMgr::CompletionPortThreadStart+0x93
0086ffec 00000000 kernel32!BaseThreadStart+0x37
  

Threads 2, 6, and 8 are ThreadpoolMgr threads. Note the calls to mscorwks!ThreadpoolMgr::GateThreadStart, mscorwks!ThreadpoolMgr::WorkerThreadStart, and mscorwks!ThreadpoolMgr::TimerThreadStart, respectively. These threads are present in ASP.NET (but not always in console applications) because of the ASP.NET use of the thread pool.

• Thread 2 is a GateThread, which monitors the health of the thread pool. It is responsible for injecting and retiring worker and I/O threads based on indicators, such as CPU utilization, garbage-collection frequency, and thread starvation. There is only one thread of this type. It is created when the first I/O or work request is executed.
• Thread 6 is a worker thread as shown by the call to mscorwks!ThreadpoolMgr::WorkerThreadStart() in the listing. This thread is waiting for work to be assigned to it.
• Thread 8 is a TimerThread, which is used to manage timed callbacks specified using the System.Threading.Timer class. There is only one thread of this type, which is created when the first System.Threading.Timer object is created.

The threads discussed above are managed threads and thread-pool threads. Let's examine some of the other threads:

• Thread 3 is the ASP.NET ping thread as shown by the call to aspnet_wp!DoPingThread(). This thread responds to the pings it receives from the health monitoring thread in InetInfo.exe. The UnderDebugger registry change you made has disabled InetInfo from pinging this worker process.

• Thread 4 is a debugger thread as shown by the call to mscorwks!DebuggerRCThread(). All managed processes have a debugger thread, even if there is no debugger attached. There is only one debugger thread in a given process. This thread exists so that a managed debugger can attach and control the managed code. Threads react differently based on whether or not a managed or native debugger attaches invasively.

  If you break into the process with a managed debugger (CorDbg or Visual Studio .NET in common language runtime mode), it attaches noninvasively. The entire process is not frozen; instead, only the managed threads are frozen. The native threads that don't contain common language runtime references will continue to run. Options that use a managed debugger to attach invasively are available, but they are beyond the scope of this guide.

  If you use a native debugger, it can attach invasively or noninvasively. When you break into the process invasively and quit, the process terminates. Consequently all threads—managed and native—are shut down. If you attach noninvasively, the process (and threads) stop temporarily. The debugger pauses the native and managed code and provides a snapshot. It then can detach and the process resumes.

  In IIS 5. x, there is only one Aspnet_wp.exe worker process and one debugger thread. Consequently, only one debugger can be attached to the Aspnet_wp.exe process at a time. This can pose a problem if you're dealing with multiple Web applications on the same computer. In IIS 6.0, you can coerce an AppDomain to run in a separate application pool. For more information, see "IIS 5. x Process Model" and "IIS 6.0 Process Model" in Chapter 1" Separate application pools provide multiple W3wp.exe processes. Multiple debugger threads are created in these processes (one in each), allowing you to debug more efficiently.

• Thread 5 is a finalizer thread as shown by the call to mscorwks!GCHeap::FinalizerThreadStart. Just like the debugger thread, there is always one finalizer thread per process in version 1.0 of the .NET Framework. The finalizer thread calls the Finalize() method of applicable objects when the GC determines that the object is unreachable (not rooted). In Visual Basic .NET, you can implement a finalize method. The same function in the Visual C#™ development tool and managed extensions for the Visual C++® development system is performed by a destructor, which maps onto a finalize method. For more information, see the following articles in the .NET Framework SDK:

  • .NET Framework General Reference
  • Implementing Finalize
  • Dispose to Clean Up Unmanaged Resources.

  Other sources include:

  • Applied Microsoft .NET Framework Programming by Jeffrey Richter (Microsoft Press, 2002).
  • "Garbage Collection: Automatic Memory Management in the Microsoft .NET Framework", by Jeffrey Richter, MSDN Magazine, November 2000. (https://msdn.microsoft.com/en-us/magazine/bb985010.aspx).
  • "Garbage Collection—Part 2: Automatic Memory Management in the Microsoft .NET Framework", by Jeffrey Richter, MSDN Magazine, December 2000. (https://msdn.microsoft.com/en-us/magazine/bb985011.aspx).
  • Chapter 19, "Automatic Memory Management (Garbage Collection)" in Applied Microsoft .NET Framework Programming by Jeffrey Richter (Microsoft Press, 2002).

• Thread 9 is a lightweight remote procedure call (RPC) thread used for local remote procedure call (LRPC) communication. Here it handles incoming RPC calls to this process.

• Thread 11 is the thread that AdPlus_AspNet.vbs broke in on when Aspnet_isapi.dll (running in InetInfo.exe) was about to shut down the process. Remember that the tool set a breakpoint on aspnet_wp!CAsyncPipeManager::Close and dumped the process when this code was triggered. This thread is also a completion port thread.

You'll find several features common to all ASP.NET dumps: the main thread, the ping thread, the debugger thread, the ThreadpoolMgr thread(s), and the finalizer thread.

Examining Managed Data

The native output has set the stage, but you want to see where the action is. To examine the managed code in WinDbg, you must load the SOS.dll debugger extension.

To load the debugger extension

1. In WinDbg, type .load SOS\SOS.dll to load the extension (where "SOS" is the location of SOS.dll).
2. Type !findtable to initialize SOS with the table information for the runtime version you are debugging.

WinDbg displays the following:

0:000> .load sos\sos.dll
0:000> !findtable
Args: ''
Attempting data table load from SOS.WKS.4.3705.209.BIN
Failed timestamp verification.
  Binary time: 05:30:57, 2002/1/5.
  Table time: 03:04:42, 2002/2/15
Attempting data table load from SOS.WKS.4.3705.BIN
Loaded Son of Strike data table version 4 from "SOS.WKS.4.3705.BIN"
  

• To list and examine the managed threads and their functions, type !threads.

There are five managed threads, which include the finalizer thread and the thread pool worker threads. Examine threads 1, 7, and 10. These are also the same threads whose call stacks made their last managed native call to CorWaitOneNative() Note the lock count values; for this scenario, the number of threads with lock counts greater than 0 will most likely equal the number of times you clicked the Obtain Shared Resource buttons. The following shows three threads with a lock count of 1:

ThreadCount: 5
UnstartedThread: 0
BackgroundThread: 5
PendingThread: 0
DeadThread: 0
  

	1	5	6	7	10
ID	420	d84	2ec	7b0	eb8
ThreadOBJ	00154008	00155d80	00174a30	0017d060	001f1618
State	200a220	b220	1800220	2000220	2000220
PreEmptive GC	Enabled	Enabled	Enabled	Enabled	Enabled
GC Alloc Context	00000000: 00000000	00000000: 00000000	00000000: 00000000	00000000: 00000000	00000000: 00000000
Domain	001a58b8	00165058	00165058	001a58b8	001a58b8
Lock Count	1	0	0	1	1
Apt	MTA	MTA	MTA	MTA	MTA
Exception		(Finalizer)	(Threadpool Worker)

The SOS !threads command provides other information that may be useful, such as the threading model, AppDomain, and thread state. In this case, all threads are initialized to multithreaded apartment (MTA) threads, and there are two active domains. Every ASP.NET application always has at least two AppDomains—the default domain and the virtual folder's application domain. The default domain houses threads that aren't running managed code in a specific AppDomain. Threads 1, 7, and 10 are each in the same domain (001a58b8). The thread states are discussed later in this section.

According to the .NET Framework SDK, an AppDomain is "an isolated environment where applications execute. It provides isolation, unloading, and security boundaries for executing managed code."

The AppDomain information can provide answers for questions like these:

• There are three blocked threads. Are they all in the same AppDomain? If so, what else is loaded in that AppDomain?
• What threads are in the default domain? These threads aren't running managed code. What are they doing? The default domain threads include the finalizer thread and the worker thread, which hasn't been given work to do yet. Once the worker thread has some work, it moves out of the default domain.
• What application is using up all the threads? If you see 25 threads in one AppDomain, that could help answer why there might not be other threads available to process requests.

For specific information on the AppDomain, type !dumpdomain <addr>, where <addr> is the domain address listed by the !threads command. In this scenario, typing !dumpdomain 1a58b8 lists the assemblies and modules in the domain specified.

0:000> !dumpdomain 001a58b8
Domain: 001a58b8
LowFrequencyHeap: 001a591c
HighFrequencyHeap: 001a5970
StubHeap: 001a59c4
Name: /LM/W3SVC/1/ROOT/Debugging-1-126678048540984363
Assembly: 0017d240 [System, Version=1.0.3300.0, Culture=neutral, 
PublicKeyToken=b77a5c561934e089]
ClassLoader: 00177e90
  Module Name
00178a68 c:\windows\assembly\gac\system\1.0.3300.0__b77a5c561934e089\system.dll

Assembly: 0017f830 [System.Drawing, Version=1.0.3300.0, Culture=neutral, 
PublicKeyToken=b03f5f7f11d50a3a]
ClassLoader: 00177f08
  Module Name
0017f8f8 c:\windows\assembly\gac\system.drawing\1.0.3300.0__b03f5f7f11d50a3a\syste
m.drawing.dll
  

Note the AppDomain name for these three threads. In this case, it is:

/LM/W3SVC/1/ROOT/Debugging-1-126678048540984363.
  

Now check the name of the AppDomain that threads 5 and 6 are in (165058). Remember that thread 5 is a finalizer thread and thread 6 is a ThreadpoolWorker wait thread. Use the command !dumpdomain 165058. Again, 165058 is the domain address listed by the !threads command.

0:000> !dumpdomain 165058
Domain: 00165058
LowFrequencyHeap: 001650bc
HighFrequencyHeap: 00165110
StubHeap: 00165164
Name: DefaultDomain
Assembly: 0017d240 [System, Version=1.0.3300.0, Culture=neutral, 
PublicKeyToken=b77a5c561934e089]
ClassLoader: 00177e90
  Module Name
00178a68 c:\windows\assembly\gac\system\1.0.3300.0__b77a5c561934e089\system.dll

Assembly: 0017f830 [System.Drawing, Version=1.0.3300.0, Culture=neutral, 
PublicKeyToken=b03f5f7f11d50a3a]
ClassLoader: 00177f08
  Module Name
0017f8f8 c:\windows\assembly\gac\system.drawing\1.0.3300.0__b03f5f7f11d50a3a\syste
m.drawing.dll
  

The name of this AppDomain is DefaultDomain. AppDomains own resources held in memory, and in order to unload a module or assembly, the AppDomain must be unloaded. The default AppDomain cannot be unloaded until the application is no longer running.

This AppDomain information did not reveal anything significant. That happens sometimes—you check out different paths, and some reveal more than others.

The SOS !threads command also provided thread state information. The state column indicates what the thread is doing, for example, waiting or running. The values correspond to byte representations of bit flags set for a particular state. You'll look at some threads' states here. To explore more, match the values to the state table in the Appendix.

Looking at the !threads output, you can see that thread 1has the state value of 0x0200a220. The following interpretation applies:

Table 3.3: Thread state for thread 1 is 0x0200a220

Mnemonic	Value	Description
TS_Interruptible	0x02000000	Sitting in either a Sleep(), Wait(), or Join()
TS_InMTA	0x00008000	Thread is part of the MTA
TS_CoInitialized	0x00002000	CoInitialize has been called for this thread
TS_Background	0x00000200	Thread is a background thread
TS_LegalToJoin	0x00000020	Is it now legal to attempt a Join()?

Threads 7 and 10 are similar, with the state value of 0x02000220, except that they exclude TS_InMTA and TS_CoInitialized. A thread will not live in an apartment until after CoInitialize is called. CoInitialize has not been called yet for these two threads because they're waiting for the resource that thread 1 accessed first.

Again, these threads are created by clicking the Obtain Shared Resource buttons. They each have a lock count of 1 and made their last managed native call to CorWaitOneNative().

Let's look at the thread state values for some the other managed threads. Again, this output provides more information on the threads.

Table 3.4: Thread state for thread 5 is 0x0000b220

Mnemonic	Value	Description
TS_InMTA	0x00008000	Thread is part of the MTA
TS_CoInitialized	0x00002000	CoInitialize has been called for this thread
TS_WeOwn	0x00001000	Exposed object initiated this thread
TS_Background	0x00000200	Thread is a background thread
TS_LegalToJoin	0x00000020	Indicates if it now legal to attempt a Join()

Table 3.5: Thread state for thread 6 is 0x1800220

Mnemonic	Value	Description
TS_TPWorkerThread	0x01000000	Is this a thread pool worker thread? (If not, it is a thread pool CompletionPort thread.)
TS_ThreadPoolThread	0x00800000	Is this a thread pool thread?
TS_Background	0x00000200	Thread is a background thread
TS_LegalToJoin	0x00000020	Indicates if it now legal to attempt a Join()

ASP.NET uses thread pooling to provide the application with a pool of worker threads that are managed by the system. There is only one ThreadPool object per process. The !threadpool command confirms that CPU utilization is minimal. There is one worker thread and five completion port threads. In this example, the maximum worker thread is 25, which implies that there is one CPU.

0:000> !threadpool
CPU utilization 2%
Worker Thread: Total: 1 Running: 1 Idle: 1 MaxLimit: 25 MinLimit: 1
Work Request in Queue: 0
--------------------------------------
Number of Timers: 0
--------------------------------------
Completion Port Thread: Total: 5 Free: 1 MaxFree: 2 CurrentLimit: 5 MaxLimit: 25 
MinLimit: 1
  

The worker thread is managed thread 6 waiting for work, and the completion port threads are runtime threads 1, 7, 10, 11, and 12. Threads 1, 7, and 10 are managed; they have work to do, but are waiting for a shared resource. Threads 11 and 12 are native. Thread 11 is the thread on which we set a breakpoint with ADPlus_AspNet.vbs. It is a completion port thread whose work item is to cleanly shut down the worker process. Thread 12 is just waiting for work as shown by the "Free:1" output from the command.

When using the SOS.dll extension, the output for some commands varies depending on whether you are working with a debug build or a release build:

• Debug build. The debug build aids in testing and debugging. This build contains error-checking, argument-verification, and debugging information that is not available in the release build. Consequently, the debug build is larger, slower, and uses more memory. Performance is also slower because the executables contain symbolic debugging information. Additional code paths are executed because of parameter checking and output of diagnostic messages for debugging purposes.
• Release build. The release build is smaller and faster, and it uses less memory than the debug build. It is fully optimized, debugging asserts are disabled, and debugging information is stripped from the binaries. Because the compiler repositions and reorganizes instructions, the debugger can't always identify the source code that corresponds to the instructions.
• Release build with .ini files. If the release build is coupled with applicable .ini files, the results simulate the debug build. The .ini files must have the same name as the assembly and be in the same folder from where the assembly was loaded. The .ini file includes the AllowOptimize and GenerateTrackingInfo settings, which are set to 0 and 1, respectively. For more information, see the .NET Framework SDK article "Making An Image Easier To Debug" on the MSDN Web site at https://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpguide/html/cpconmakingimageeasiertodebug.asp.

For more information on compiling options, see the .NET Framework SDK. Unless otherwise specified, release build examples from the dump file are used here.

You can examine the managed trace by typing ~*e !clrstack, as shown in the following command sample. This command is similar to the ~*k WinDbg command noted earlier, but it applies to managed calls. WinDbg now displays the current enter stack pointer (ESP), the current enter instruction pointer (EIP), and the method signature for each frame with a managed thread. The !clrstack command would show only the managed threads.

0:000> ~*e !clrstack
Thread 0
Not a managed thread.
Thread 1
ESP       EIP     
0086f948  7ffe0304 [FRAME: ECallMethodFrame] [DEFAULT] Boolean 
System.Threading.WaitHandle.WaitOneNative(I,UI4,Boolean)
0086f95c  03ce73f6 [DEFAULT] [hasThis] Boolean 
System.Threading.WaitHandle.WaitOne(I4,Boolean)
0086f970  03c410e2 [DEFAULT] Boolean Debugging.SharedResource.Wait()
0086f974  03c41059 [DEFAULT] [hasThis] Void 
Debugging.Contention.btnObtainShared1_Click(Object,Class System.EventArgs)
0086f980  03ce7375 [DEFAULT] [hasThis] Void 
System.Web.UI.WebControls.Button.OnClick(Class System.EventArgs)
0086f994  03ce7152 [DEFAULT] [hasThis] Void 
System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandl
er.RaisePostBackEvent(String)
0086f9a4  03ce7103 [DEFAULT] [hasThis] Void 
System.Web.UI.Page.RaisePostBackEvent(Class 
System.Web.UI.IPostBackEventHandler,String)
0086f9ac  03ce7062 [DEFAULT] [hasThis] Void 
System.Web.UI.Page.RaisePostBackEvent(Class System.Collections.Specialized.NameValueCollection)
0086f9bc  03c6ebd8 [DEFAULT] [hasThis] Void 
System.Web.UI.Page.ProcessRequestMain()
0086f9f4  03c6c90f [DEFAULT] [hasThis] Void 
System.Web.UI.Page.ProcessRequest()
0086fa2c  03c6c373 [DEFAULT] [hasThis] Void 
System.Web.UI.Page.ProcessRequest(Class System.Web.HttpContext)
0086fa34  03c6c34c [DEFAULT] [hasThis] Void 
System.Web.HttpApplication/CallHandlerExecutionStep.Execute()
0086fa44  03c600b0 [DEFAULT] [hasThis] Class System.Exception 
System.Web.HttpApplication.ExecuteStep(Class IExecutionStep,ByRef 
Boolean)
0086fa8c  03daf66b [DEFAULT] [hasThis] Void 
System.Web.HttpApplication.ResumeSteps(Class System.Exception)
0086fad0  03daf543 [DEFAULT] [hasThis] Class System.IAsyncResult 
System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProce
ssRequest(Class System.Web.HttpContext,Class 
System.AsyncCallback,Object)
0086faec  0377f648 [DEFAULT] [hasThis] Void 
System.Web.HttpRuntime.ProcessRequestInternal(Class 
System.Web.HttpWorkerRequest)
0086fb28  0377f42d [DEFAULT] Void 
System.Web.HttpRuntime.ProcessRequest(Class 
System.Web.HttpWorkerRequest)
0086fb34  0377b127 [DEFAULT] [hasThis] I4 
System.Web.Hosting.ISAPIRuntime.ProcessRequest(I,I4)
0086fbf0  7920eab0 [FRAME: ContextTransitionFrame] 
0086fcd0  7920eab0 [FRAME: ComMethodFrame] 
Thread 2
Not a managed thread.
Thread 3
Not a managed thread.
Thread 4
Not a managed thread.
Thread 5
ESP       EIP     
Thread 6
ESP       EIP     
Thread 7
ESP       EIP     
0363f94c  7ffe0304 [FRAME: ECallMethodFrame] [DEFAULT] Boolean 
System.Threading.WaitHandle.WaitOneNative(I,UI4,Boolean)
0363f960  03ce73f6 [DEFAULT] [hasThis] Boolean 
System.Threading.WaitHandle.WaitOne(I4,Boolean)
0363f974  03c41115 [DEFAULT] [hasThis] Void 
Debugging.Contention.btnObtainShared2_Click(Object,Class 
System.EventArgs)
0363f980  03ce7375 [DEFAULT] [hasThis] Void 
System.Web.UI.WebControls.Button.OnClick(Class System.EventArgs)
0363f994  03ce7152 [DEFAULT] [hasThis] Void 
System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandl
er.RaisePostBackEvent(String)
0363f9a4  03ce7103 [DEFAULT] [hasThis] Void 
System.Web.UI.Page.RaisePostBackEvent(Class 
System.Web.UI.IPostBackEventHandler,String)
0363f9ac  03ce7062 [DEFAULT] [hasThis] Void 
System.Web.UI.Page.RaisePostBackEvent(Class 
System.Collections.Specialized.NameValueCollection)
0363f9bc  03c6ebd8 [DEFAULT] [hasThis] Void 
System.Web.UI.Page.ProcessRequestMain()
0363f9f4  03c6c90f [DEFAULT] [hasThis] Void 
System.Web.UI.Page.ProcessRequest()
0363fa2c  03c6c373 [DEFAULT] [hasThis] Void 
System.Web.UI.Page.ProcessRequest(Class System.Web.HttpContext)
0363fa34  03c6c34c [DEFAULT] [hasThis] Void 
System.Web.HttpApplication/CallHandlerExecutionStep.Execute()
0363fa44  03c600b0 [DEFAULT] [hasThis] Class System.Exception 
System.Web.HttpApplication.ExecuteStep(Class IExecutionStep,ByRef 
Boolean)
0363fa8c  03daf66b [DEFAULT] [hasThis] Void 
System.Web.HttpApplication.ResumeSteps(Class System.Exception)
0363fad0  03daf543 [DEFAULT] [hasThis] Class System.IAsyncResult 
System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProce
ssRequest(Class System.Web.HttpContext,Class 
System.AsyncCallback,Object)
0363faec  0377f648 [DEFAULT] [hasThis] Void 
System.Web.HttpRuntime.ProcessRequestInternal(Class 
System.Web.HttpWorkerRequest)
0363fb28  0377f42d [DEFAULT] Void 
System.Web.HttpRuntime.ProcessRequest(Class 
System.Web.HttpWorkerRequest)
0363fb34  0377b127 [DEFAULT] [hasThis] I4 
System.Web.Hosting.ISAPIRuntime.ProcessRequest(I,I4)
0363fbf0  7920eab0 [FRAME: ContextTransitionFrame] 
0363fcd0  7920eab0 [FRAME: ComMethodFrame] 
Thread 8
Not a managed thread.
Thread 9
Not a managed thread.
Thread 10
ESP       EIP     
040af94c  7ffe0304 [FRAME: ECallMethodFrame] [DEFAULT] Boolean 
System.Threading.WaitHandle.WaitOneNative(I,UI4,Boolean)
040af960  03ce73f6 [DEFAULT] [hasThis] Boolean 
System.Threading.WaitHandle.WaitOne(I4,Boolean)
040af974  03c4116d [DEFAULT] [hasThis] Void 
Debugging.Contention.btnObtainShared3_Click(Object,Class 
System.EventArgs)
040af980  03ce7375 [DEFAULT] [hasThis] Void 
System.Web.UI.WebControls.Button.OnClick(Class System.EventArgs)
040af994  03ce7152 [DEFAULT] [hasThis] Void 
System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandl
er.RaisePostBackEvent(String)
040af9a4  03ce7103 [DEFAULT] [hasThis] Void 
System.Web.UI.Page.RaisePostBackEvent(Class 
System.Web.UI.IPostBackEventHandler,String)
040af9ac  03ce7062 [DEFAULT] [hasThis] Void 
System.Web.UI.Page.RaisePostBackEvent(Class 
System.Collections.Specialized.NameValueCollection)
040af9bc  03c6ebd8 [DEFAULT] [hasThis] Void 
System.Web.UI.Page.ProcessRequestMain()
040af9f4  03c6c90f [DEFAULT] [hasThis] Void 
System.Web.UI.Page.ProcessRequest()
040afa2c  03c6c373 [DEFAULT] [hasThis] Void 
System.Web.UI.Page.ProcessRequest(Class System.Web.HttpContext)
040afa34  03c6c34c [DEFAULT] [hasThis] Void 
System.Web.HttpApplication/CallHandlerExecutionStep.Execute()
040afa44  03c600b0 [DEFAULT] [hasThis] Class System.Exception 
System.Web.HttpApplication.ExecuteStep(Class IExecutionStep,ByRef 
Boolean)
040afa8c  03daf66b [DEFAULT] [hasThis] Void 
System.Web.HttpApplication.ResumeSteps(Class System.Exception)
040afad0  03daf543 [DEFAULT] [hasThis] Class System.IAsyncResult 
System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProce
ssRequest(Class System.Web.HttpContext,Class 
System.AsyncCallback,Object)
040afaec  0377f648 [DEFAULT] [hasThis] Void 
System.Web.HttpRuntime.ProcessRequestInternal(Class 
System.Web.HttpWorkerRequest)
040afb28  0377f42d [DEFAULT] Void 
System.Web.HttpRuntime.ProcessRequest(Class 
System.Web.HttpWorkerRequest)
040afb34  0377b127 [DEFAULT] [hasThis] I4 
System.Web.Hosting.ISAPIRuntime.ProcessRequest(I,I4)
040afbf0  7920eab0 [FRAME: ContextTransitionFrame] 
040afcd0  7920eab0 [FRAME: ComMethodFrame] 
Thread 11
Not a managed thread.
Thread 12
Not a managed thread.
Thread 13
Not a managed thread.
  

Because this is a release build, the call stack may not report as many method calls as expected. Code transformations, such as inlining, occur during optimization. The following shows how the code has been optimized by the JIT compiler. Managed threads 1, 7, and 10 each call the same function. Thread 1 ran first and included Debugging.SharedResource.Wait(). The other two frames don't include this call.

Thread 1
ESP       EIP     
0086f948  7ffe0304 [FRAME: ECallMethodFrame] [DEFAULT] Boolean 
System.Threading.WaitHandle.WaitOneNative(I,UI4,Boolean)
0086f95c  03cd77be [DEFAULT] [hasThis] Boolean 
System.Threading.WaitHandle.WaitOne(I4,Boolean)
0086f970  03c310e2 [DEFAULT] Boolean Debugging.SharedResource.Wait()
0086f974  03c31059 [DEFAULT] [hasThis] Void 
Debugging.Contention.btnObtainShared1_Click(Object,Class 
System.EventArgs)
0086f980  03cd7735 [DEFAULT] [hasThis] Void 
System.Web.UI.WebControls.Button.OnClick(Class System.EventArgs)

Thread 7
ESP       EIP     
0363f94c  7ffe0304 [FRAME: ECallMethodFrame] [DEFAULT] Boolean 
System.Threading.WaitHandle.WaitOneNative(I,UI4,Boolean)
0363f960  03cd77be [DEFAULT] [hasThis] Boolean 
System.Threading.WaitHandle.WaitOne(I4,Boolean)
0363f974  03c31115 [DEFAULT] [hasThis] Void 
Debugging.Contention.btnObtainShared2_Click(Object,Class 
System.EventArgs)
0363f980  03cd7735 [DEFAULT] [hasThis] Void 
System.Web.UI.WebControls.Button.OnClick(Class System.EventArgs)

Thread 10
ESP       EIP     
040af94c  7ffe0304 [FRAME: ECallMethodFrame] [DEFAULT] Boolean 
System.Threading.WaitHandle.WaitOneNative(I,UI4,Boolean)
040af960  03cd77be [DEFAULT] [hasThis] Boolean 
System.Threading.WaitHandle.WaitOne(I4,Boolean)
040af974  03c3116d [DEFAULT] [hasThis] Void 
Debugging.Contention.btnObtainShared3_Click(Object,Class 
System.EventArgs)
040af980  03cd7735 [DEFAULT] [hasThis] Void 
System.Web.UI.WebControls.Button.OnClick(Class System.EventArgs)
  

If you use ILDasm with the release build and examine the Microsoft intermediate language (MSIL) for each button-click event, you will find that each function's IL code is practically identical. But if you disassemble the code, you will see differences between the JIT-compiled (optimized) and non-JIT compiled code. Keep in mind that JIT-compiled managed code is converted to x86 assembly (native code).

Use the WinDbg !u command to disassemble the code for the thread 1 button-click event. You pass in the EIP, and the disassembly begins at the current address.

From the SOS !clrstack command output for thread 1, use the EIP value as the address that is passed to the !u command:

0086f974  03c31059 [DEFAULT] [hasThis] Void 
Debugging.Contention.btnObtainShared1_Click(Object,Class 
System.EventArgs)

0:010> !u 03c31059 
Normal JIT generated code
[DEFAULT] [hasThis] Void 
Debugging.Contention.btnObtainShared1_Click(Object,Class 
System.EventArgs)
Begin 03c31050, size 3b
03c31050 56               push    esi
03c31051 8bf1             mov     esi,ecx
03c31053 ff1554c07503     call    dword ptr [0375c054] 
(Debugging.SharedResource.Wait)  //call Wait() function
03c31059 25ff000000       and     eax,0xff 
03c3105e 8b8ec8000000     mov     ecx,[esi+0xc8]
03c31064 8bf1             mov     esi,ecx
03c31066 8b01             mov     eax,[ecx]
03c31068 ff9090010000     call    dword ptr [eax+0x190]
03c3106e 8bc8             mov     ecx,eax
03c31070 8b15d49c0702     mov     edx,[02079cd4] ("<BR>Obtained Shared 
Resource 1")
03c31076 e8e5325dff       call    03204360 (System.String.Concat)
03c3107b 8bd0             mov     edx,eax
03c3107d 8bce             mov     ecx,esi
03c3107f 8b01             mov     eax,[ecx]
03c31081 ff9094010000     call    dword ptr [eax+0x194]
03c31087 5e               pop     esi
03c31088 c20400           ret     0x4
  

The first button-click event calls SharedResource.Wait(). This code path has not been JIT-compiled before. The JIT compiler compiles the method call the first time it is called, and it includes a call to Debugging.SharedResource.Wait(). You can use the !u command to determine where this call goes as shown in the following example:

0:010> !u  03c310e2  
  

In this case, 03c310e2 is the EIP address for the Debugging.SharedResource.Wait() function, as you can see from the !clrstack output for thread 1.

As shown in the code below, the Wait() method calls ManualResetEvent(). For the second or third button click, take the EIP value as the address passed to the !u command. In this example, the Thread 10s value is used.

040af974  03c3116d [DEFAULT] [hasThis] Void 
Debugging.Contention.btnObtainShared3_Click(Object,Class 
System.EventArgs)

0:010> !u 03c3116d
Normal JIT generated code
[DEFAULT] [hasThis] Void 
Debugging.Contention.btnObtainShared3_Click(Object,Class 
System.EventArgs)
Begin 03c31158, size 4c
03c31158 56               push    esi
03c31159 8bf1             mov     esi,ecx
//code runs inline, after JITting
03c3115b 6a00             push    0x0
03c3115d 8b0d98910702     mov     ecx,[02079198] (Object: 
System.Threading.ManualResetEvent)   //moves the address of the 
object into ecx
03c31163 baffffffff       mov     edx,0xffffffff  //pass in -1 to 
ManualResetEvent (infinite wait)
03c31168 8b01             mov     eax,[ecx]  //deference ecx (function 
pointer)  
03c3116a ff5050           call    dword ptr [eax+0x50]  //call eax 
plus 50 (offset in the ManualResetEvent to WaitOne())
03c3116d b801000000       mov     eax,0x1  
03c31172 25ff000000       and     eax,0xff  
03c31177 8b8ec8000000     mov     ecx,[esi+0xc8]  
03c3117d 8bf1             mov     esi,ecx
03c3117f 8b01             mov     eax,[ecx]
03c31181 ff9090010000     call    dword ptr [eax+0x190]  
03c31187 8bc8             mov     ecx,eax
03c31189 8b15dc9c0702     mov     edx,[02079cdc] ("<BR>Obtained Shared 
Resource 3")
03c3118f e8cc315dff       call    03204360 (System.String.Concat)
03c31194 8bd0             mov     edx,eax
03c31196 8bce             mov     ecx,esi
03c31198 8b01             mov     eax,[ecx]
03c3119a ff9094010000     call    dword ptr [eax+0x194]
03c311a0 5e               pop     esi
03c311a1 c20400           ret     0x4
  

When the other click events call the method again, the optimized code does not perform an "external" call to SharedResource.Wait(); rather, it runs the code inline.

Look again at threads 5 and 6 and note that the !clrstack output does not include any stack data for these managed threads. Managed code can run on these threads, but isn't currently active. The example below shows the relevant lines extracted from the !clrstack and !threads outputs.

Output from !clrstack
Thread 5
ESP       EIP     
Thread 6
ESP       EIP  
   
Output from !threads
  5   454 00156c10      b220 Enabled  00000000:00000000 00166030     0 MTA 
(Finalizer)
  6   5e8 00174a30   1800220 Enabled  00000000:00000000 00166030     0 MTA 
(Threadpool Worker)
  

Again, you can see threads 1, 7, and 10 performing similar actions. Earlier you looked at the native output of one of these managed threads (thread 7). SOS.dll output fills in the missing information for the managed data. To build both managed and native portions of the stack, use Notepad or a similar editor to cut and paste the output from !clrstack into the middle portion where there were previously no mapped modules.

   1  id: 580.ffc   Suspend: 1 Teb 7ffdd000 Unfrozen
NATIVE
ChildEBP RetAddr  
0086f7ac 77f7f49f SharedUserData!SystemCallStub+0x4
0086f7b0 77e74bd8 ntdll!ZwWaitForMultipleObjects+0xc
0086f84c 79281971 kernel32!WaitForMultipleObjectsEx+0x12c
0086f87c 79282444 mscorwks!Thread::DoAppropriateWaitWorker+0xc1
0086f8d0 79317c0a mscorwks!Thread::DoAppropriateWait+0x46
0086f918 036545f4 mscorwks!WaitHandleNative::CorWaitOneNative+0x6f

MANAGED
ESP       EIP     
0086f948  7ffe0304 [FRAME: ECallMethodFrame] [DEFAULT] Boolean 
System.Threading.WaitHandle.WaitOneNative (I,UI4,Boolean)
0086f95c  03cd77be [DEFAULT] [hasThis] Boolean 
System.Threading.WaitHandle.WaitOne (I4,Boolean)
0086f970  03c310e2 [DEFAULT] Boolean Debugging.SharedResource.Wait ()
0086f974  03c31059 [DEFAULT] [hasThis] Void 
Debugging.Contention.btnObtainShared1_Click (Object,Class 
System.EventArgs)
0086f980  03cd7735 [DEFAULT] [hasThis] Void 
System.Web.UI.WebControls.Button.OnClick(Class System.EventArgs)
0086f994  03cd7512 [DEFAULT] [hasThis] Void 
System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandl
er.RaisePostBackEvent(String)
0086f9a4  03cd74c3 [DEFAULT] [hasThis] Void 
System.Web.UI.Page.RaisePostBackEvent(Class 
System.Web.UI.IPostBackEventHandler,String)
0086f9ac  03cd7422 [DEFAULT] [hasThis] Void 
System.Web.UI.Page.RaisePostBackEvent(Class 
System.Collections.Specialized.NameValueCollection)
0086f9bc  03c5f050 [DEFAULT] [hasThis] Void 
System.Web.UI.Page.ProcessRequestMain()
0086f9f4  03c5cd7f [DEFAULT] [hasThis] Void 
System.Web.UI.Page.ProcessRequest()
0086fa2c  03c5c7eb [DEFAULT] [hasThis] Void 
System.Web.UI.Page.ProcessRequest(Class System.Web.HttpContext)
0086fa34  03c5c7c4 [DEFAULT] [hasThis] Void 
System.Web.HttpApplication/CallHandlerExecutionStep.Execute()
0086fa44  03c50528 [DEFAULT] [hasThis] Class System.Exception 
System.Web.HttpApplication.ExecuteStep(Class IExecutionStep,ByRef 
Boolean)
0086fa8c  03dafb2b [DEFAULT] [hasThis] Void 
System.Web.HttpApplication.ResumeSteps(Class System.Exception)
0086fad0  03daf9fb [DEFAULT] [hasThis] Class System.IAsyncResult 
System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProce
ssRequest(Class System.Web.HttpContext,Class 
System.AsyncCallback,Object)
0086faec  0377f5b8 [DEFAULT] [hasThis] Void 
System.Web.HttpRuntime.ProcessRequestInternal(Class 
System.Web.HttpWorkerRequest)
0086fb28  0377f39d [DEFAULT] Void 
System.Web.HttpRuntime.ProcessRequest(Class 
System.Web.HttpWorkerRequest)
0086fb34  0377b097 [DEFAULT] [hasThis] I4 
System.Web.Hosting.ISAPIRuntime.ProcessRequest(I,I4)
0086fbf0  7920eab0 [FRAME: ContextTransitionFrame] 
0086fcd0  7920eab0 [FRAME: ComMethodFrame]

NATIVE
0086fbe4 792830c1 mscorwks!ComCallMLStubCache::CompileMLStub+0x1af
0086fc2c 79260b19 mscorwks!Thread::DoADCallBack+0x5c
0086fc94 00cea179 mscorwks!ComCallMLStubCache::CompileMLStub+0x2c2
0086ff24 0042218c 0xcea179
0086ff40 7a13ecc2 aspnet_wp!CAsyncPipeManager::ProcessCompletion+0x1d4 
[e:\dna\src\xsp\wp\asyncpipemanager.cxx @ 516]
0086ff84 792cf905 aspnet_isapi!CorThreadPoolCompletionCallback+0x41 
[e:\dna\src\xsp\isapi\threadpool.cxx @ 773]
0086ffb4 77e802ed mscorwks!ThreadpoolMgr::CompletionPortThreadStart+0x93
0086ffec 00000000 kernel32!BaseThreadStart+0x37
  

Read through the various events and calls. Note that the ButtonObtainShared1_Click() event calls SharedResource.Wait(), which calls System.Threading.WaitHandle.WaitOne(). This thread is waiting—it makes a runtime call to CorWaitOneNative(), which in turn makes an operating system call to WaitForMultipleObjectsEx(). This is seen in the three of the managed threads (except the thread1 call to Wait() as mentioned before). This gives a great picture of what is causing the deadlock!

Using the Debug Build

The debug build output for the managed threads also includes the source file and line number at the time of the dump. You can use the !clrstack command to view this information. For example, in the following debug build frame, the !clrstack output tells you to go to the SharedResource.cs file and check out line 55.

0373f940  03df1303 [DEFAULT] Boolean Debugging.SharedResource.Wait()
  at [+0x5b] [+0x2d] C:\Inetpub\wwwroot\Debugging\SharedResource.cs:55
0373f960  03df136b [DEFAULT] [hasThis] Void 
Debugging.Contention.btnObtainShared1_Click(Object,Class 
System.EventArgs)
  at [+0x13] [+0x5] c:\inetpub\wwwroot\debugging\contention.aspx.cs:120

  

Provided that the debug build is used, you can also check out other !clrstack options, such as -l and -p. For example, type ~1s (to go to that thread), and then type !clrstack -p to get the following results:

0373f960  03df136b [DEFAULT] [hasThis] Void 
Debugging.Contention.btnObtainShared1_Click (Object,Class System.EventArgs)
  at [+0x13] [+0x5] c:\inetpub\wwwroot\debugging\contention.aspx.cs:120
    PARAM: this: 0x011ce548 (ASP.Contention_aspx)
    PARAM: class System.EventArgs sender: 0x011d2f60 
    PARAM: unsigned int8 e: 0x12186b0
  

What jumps out is the PARAM information in the ASP.Contention_aspx btnObtainShared1_Click frame. You can use the !dumpobj command to discover more about this parameter. The !DumpObj <addr> command dumps an object on the managed heap and displays information about the object's fields. Type !dumpobj 0x011ce548. You can then dump other objects using the same syntax.

For the same thread, you can examine the locals for the frames that contain debugging information. Type !clrstack -l and see if the LOCAL information is helpful.

0373f940  03df1303 [DEFAULT] Boolean Debugging.SharedResource.Wait()
  at [+0x5b] [+0x2d] C:\Inetpub\wwwroot\Debugging\SharedResource.cs:55
    LOCAL: bool CS$00000003$00000000: false
    LOCAL: int32 CS$00000002$00000001: 142
    LOCAL: int32 CS$00000002$00000002: 0

  

Reviewing What You've Learned

In this scenario, the dumps have indicated that it's time to look at the source code. The ~*k 200 WinDbg command showed three managed threads with the last managed native call to mscorwks !WaitHandleNative::CorWaitOneNative().

You loaded SOS.dll and used the !threads command to list and examine the managed threads and their functions. The same three threads show up in the same AppDomain, with lock counts of one, and with state of Wait. The !clrstack command output shows a call to the ButtonObtainShared1_Click() event, which calls SharedResource.Wait(), which in turn calls System.Threading.WaitHandle.WaitOne(). If you were using the debug build, the output even points to the page and line of code in question: C:\Inetpub\wwwroot\Debugging\SharedResource.cs:55.

Looking at the Source Code

Now let's discover what the code is waiting on. The btnObtainShared1_Click() event calls Debugging.SharedResource.Wait(). Look at the source code for Debugging.SharedResource.Wait().

//From SharedResource.cs
public static bool Wait()
{
   System.Diagnostics.Debug.WriteLine("About to wait : " + 
         Thread.CurrentThread.GetHashCode().ToString());
   _sharedEvent.WaitOne(-1, false);
   System.Diagnostics.Debug.WriteLine("Finished Waiting: " + 
         Thread.CurrentThread.GetHashCode().ToString());
   return true;
}
  

The SDK documentation for the WaitHandle.WaitOne() method says that when this method is "overridden in a derived class, [it] blocks the current thread until the current WaitHandle receives a signal." The documentation also says that because -1 was passed as the first parameter, the thread waits indefinitely until a signal is received.

Look at the _sharedEvent object to determine its scope.

//From SharedResource.cs
public class SharedResource
{
   static SharedResource()
   {
      _sharedEvent = new ManualResetEvent(false);

   }
   private static ManualResetEvent _sharedEvent;
  

Note that there is a static constructor and a static field called _sharedEvent. This means that there is only one copy of this class per AppDomain.

To reiterate, the code first creates a ManualResetEvent in a nonsignaled state. Each time you click an Obtain Shared Resource button, code for SharedResource.Wait() is invoked, which calls ManualResetEvent.WaitOne(). The threads are waiting for a shared resource, not their own copy. WaitOne() blocks the current thread until the WaitHandle receives a signal. You don't provide the signal within the three-minute interval, so the Aspnet_wp.exe process recycles.

If you don't have the source, you can use ILDasm and examine the MSIL code.

To change which thread is being examined, use the ~ command, passing it the thread number you want to use. For example, to view the call stack for thread 1, use ~1s.

0:000> ~1s
  

Now look back at the native call stack using the kb command. In the examples shown below, you are only interested in the first six frames so you can pass 6 to the kb command. You want to verify that each thread is waiting on the same event.

0:001> kb 6
ChildEBP RetAddr  Args to Child              
0086f7ac 77f7f49f 77e74bd8 00000001 0086f7f8 SharedUserData!SystemCallStub+0x4
0086f7b0 77e74bd8 00000001 0086f7f8 00000000 ntdll!ZwWaitForMultipleObjects+0xc
0086f84c 79281971 00000001 0086f92c 00000001 
kernel32!WaitForMultipleObjectsEx+0x12c
0086f87c 79282444 00000001 0086f92c 00000001 
mscorwks!Thread::DoAppropriateWaitWorker+0xc1
0086f8d0 79317c0a 00000001 0086f92c 00000001 
mscorwks!Thread::DoAppropriateWait+0x46
0086f918 036545f4 0086f924 00000000 ffffffff 
mscorwks!WaitHandleNative::CorWaitOneNative+0x6f
  

Examine the kernel32!WaitForMultipleObjectsEx() function. You can look up the function definition on the MSDN Web site at https://msdn.microsoft.com/en-us/library/default.aspxdefault.asp by searching for WaitForMultipleObjectsEx . Here's the definition:

 
DWORD WaitForMultipleObjectsEx(
  DWORD nCount,             // number of handles in array
  CONST HANDLE *lpHandles,  // object-handle array
  BOOL bWaitAll,            // wait option
  DWORD dwMilliseconds,     // time-out interval
  BOOL bAlertable           // alertable option
);
  

In the above listing, the first argument passed has the value 1, while the second is the address of the object handle array. Because you know there is one item in the handle array, you can dump the second parameter with the dd <address> l<number of DWORDs to dump> command; for example, use dd 0086f92c l1.

If you check the other threads, you can see that they are all waiting on the same handle, which has the value 250.

0:001> dd 0086f92c l1
0086f92c  00000250
0:007> ~7s
eax=000000c0 ebx=0363f7fc ecx=000003ff edx=00000000 esi=00000000 edi=7ffdf000
eip=7ffe0304 esp=0363f7b4 ebp=0363f850 iopl=0         nv up ei pl nz na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000202
SharedUserData!SystemCallStub+4:
7ffe0304 c3               ret
0:007> kb 6
ChildEBP RetAddr  Args to Child              
0363f7b0 77f7f49f 77e74bd8 00000001 0363f7fc SharedUserData!SystemCallStub+0x4
0363f7b4 77e74bd8 00000001 0363f7fc 00000000 ntdll!ZwWaitForMultipleObjects+0xc
0363f850 79281971 00000001 0363f930 00000001 
kernel32!WaitForMultipleObjectsEx+0x12c
0363f880 79282444 00000001 0363f930 00000001 
mscorwks!Thread::DoAppropriateWaitWorker+0xc1
0363f8d4 79317c0a 00000001 0363f930 00000001 
mscorwks!Thread::DoAppropriateWait+0x46
0363f91c 036545f4 0363f928 00000000 ffffffff 
mscorwks!WaitHandleNative::CorWaitOneNative+0x6f
0:007> dd 0363f7fc l1
0363f7fc  00000250
0:007> ~10s
eax=000000c0 ebx=040af7fc ecx=000003ff edx=00000000 esi=00000000 edi=7ffdf000
eip=7ffe0304 esp=040af7b4 ebp=040af850 iopl=0         nv up ei pl nz na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000202
SharedUserData!SystemCallStub+4:
7ffe0304 c3               ret
0:010> kb 6
ChildEBP RetAddr  Args to Child              
040af7b0 77f7f49f 77e74bd8 00000001 040af7fc SharedUserData!SystemCallStub+0x4
040af7b4 77e74bd8 00000001 040af7fc 00000000 ntdll!ZwWaitForMultipleObjects+0xc
040af850 79281971 00000001 040af930 00000001 
kernel32!WaitForMultipleObjectsEx+0x12c
040af880 79282444 00000001 040af930 00000001 mscorwks!Thread::DoAppropriateWaitWorker+0xc1
040af8d4 79317c0a 00000001 040af930 00000001 
mscorwks!Thread::DoAppropriateWait+0x46
040af91c 036545f4 040af928 00000000 ffffffff 
mscorwks!WaitHandleNative::CorWaitOneNative+0x6f
0:010> dd 040af930  l1
040af930  00000250

  

Exploring Further

Try these ideas to learn more:

• Although each Obtain Shared Resource button invokes the functions with the same code, the output for the call stacks shows multiple code paths. You may be dealing with the same code, but not the same function.

  Click the same Obtain Shared Resource button three times and compare these results to those that occur when you click three different Obtain Shared Resource buttons. The output should show the same call stack three times. Each thread would be obtaining the same resource.

• Click an Obtain Shared Resource button at least 25 times and consider any thread pool information gathered from either System Monitor or the dumps and SOS.dll. What happens when the thread pool is exhausted?

• Compare the results when you use AutoResetEvent instead of ManualResetEvent. From the .NET Framework SDK: "You use the AutoResetEvent class to make a thread wait until some event puts it in the signaled state by calling AutoResetEvent.Set. Unlike the ManualResetEvent, the AutoResetEvent is automatically reset to nonsignaled by the system after a single waiting thread has been released. If no threads are waiting, the event object's state remains signaled."

Debugging with Visual Studio .NET

Visual Studio .NET contains an excellent debugger that can be used to display detailed information about both managed and native code. This section demonstrates how you can use Visual Studio .NET to obtain much of the same information you can obtain using WinDbg, SOS.dll, and CorDbg.

Dumping Process Information

Before you create a dump, you must ensure that the registry and the application are correctly configured.

To configure the registry and the application

1. Run Regedit, and then locate the key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ASP.NET.

2. Look at the UnderDebugger key, and make sure that its data value is set to 1.

3. Start the Visual Studio .NET IDE, and then open the DebuggingWeb project. Make sure that the active configuration is the Debug configuration. To change the configuration:

   1. Right-click the solution in Solution Explorer, and then click Configuration Manager.
   2. Set the Active Solution Configuration to Debug.

4. In Solution Explorer, right-click the project, and then click Set as Startup Project.

5. Right-click Contention.aspx, and then click Set as Start Page.

6. Press F5 to start the application.

7. When the form displays in the browser window, click each Obtain Shared Resource button once.

8. Close all dockable windows.

9. On the Debug menu, click Break All, or press CTRL+ALT+BREAK. On the Debug menu, point to Windows, and then click Disassembly.

   You should see disassembly and source windows displayed. Figure 3.5 shows the source window:

   Figure 3.5. Source code window

10. To open the thread window, on the Debug menu, point to Windows, and then click Threads.

In Visual Studio .NET, you can also issue debugger commands from the command line.

To issue debugger commands from the command line

1. On the View menu, point to Other Windows, and then select Command Window.

   You can use the tilde (~) command to list the current threads:

   >~
    Index Id     Name                           Location
   --------------------------------------------------------------------------------
   *1     3584   <No Name>       System.Threading.WaitHandle::WaitOne
    2     2740   <No Name>       System.Threading.WaitHandle::WaitOne
    3     3360   <No Name>       System.Threading.WaitHandle::WaitOne
   
   

2. To open a window that displays the call stack, on the Debug menu, point to Windows, and then select Call Stack.

   Figure 3.6. Call Stack window

3. To open the Macro Explorer window, on the Tools menu, point to Macros, and then click Macro Explorer.

4. Expand Samples, and then expand VSDebugger.

5. Right-click DumpStacks, and the click Run.

The output from the macro appears in the command window and is similar to the following example:

>kb
 Index  Function
--------------------------------------------------------------------------------
 1      mscorlib.dll!System.Threading.WaitHandle::WaitOne(__int32 
millisecondsTimeout = -1, bool exitContext = false)
*2      debugging.dll!Debugging.SharedResource.Wait()
 3      
debugging.dll!Debugging.Contention.btnObtainShared1_Click(System.O
bject sender = {System.Web.UI.WebControls.Button}, 
System.EventArgs e = {System.EventArgs})
 4      
system.web.dll!System.Web.UI.WebControls.Button::OnClick(System.Ev
entArgs e = {System.EventArgs})
 5      
system.web.dll!System.Web.UI.WebControls.Button::System.Web.UI.IPo
stBackEventHandler.RaisePostBackEvent(String* eventArgument = 
null)
 6      
system.web.dll!System.Web.UI.Page::RaisePostBackEvent(System.Web.U
I.IPostBackEventHandler sourceControl = 
{System.Web.UI.WebControls.Button}, String* eventArgument = null)
 7      
system.web.dll!System.Web.UI.Page::RaisePostBackEvent(System.Colle
ctions.Specialized.NameValueCollection postData = 
{System.Web.HttpValueCollection})
 8      system.web.dll!System.Web.UI.Page::ProcessRequestMain()
 9      system.web.dll!System.Web.UI.Page::ProcessRequest()
 10     
system.web.dll!System.Web.UI.Page::ProcessRequest(System.Web.HttpC
ontext context = {System.Web.HttpContext})
 11     system.web.dll!CallHandlerExecutionStep::Execute()
 12     
system.web.dll!System.Web.HttpApplication::ExecuteStep(System.Web.
HttpApplication.IExecutionStep step = 
{System.Web.HttpApplication.CallHandlerExecutionStep}, bool 
completedSynchronously = true)
 13     
system.web.dll!System.Web.HttpApplication::ResumeSteps(System.
Exception error = null)
 14     
system.web.dll!System.Web.HttpApplication::System.Web.
IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext context = 
{System.Web.HttpContext}, System.AsyncCallback cb = 
{System.AsyncCallback}, System.Object extraData = 
{System.Web.HttpContext})
 15     
system.web.dll!System.Web.HttpRuntime::ProcessRequestInternal(Syst
em.Web.HttpWorkerRequest wr)
 16     
system.web.dll!System.Web.HttpRuntime::ProcessRequest(System.Web.H
ttpWorkerRequest wr)
 17     
system.web.dll!System.Web.Hosting.ISAPIRuntime::ProcessRequest(__i
nt32 ecb, __int32 iWRType)
  

Switch the thread to 2740 to display similar information for the next thread. Only the third frame differs from the first thread's call stack.

>kb
 Index  Function
--------------------------------------------------------------------------------
 1      mscorlib.dll!System.Threading.WaitHandle::WaitOne(__int32 
millisecondsTimeout = -1, bool exitContext = false)
*2      debugging.dll!Debugging.SharedResource.Wait()
 3      
debugging.dll!Debugging.Contention.btnObtainShared2_Click(System.O
bject sender = {System.Web.UI.WebControls.Button}, 
System.EventArgs e = {System.EventArgs})
  

Switch the thread to 3360 to display similar information for the next thread. Only the third frame differs from the first thread's call stack.

>kb
 Index  Function
--------------------------------------------------------------------------------
 1      mscorlib.dll!System.Threading.WaitHandle::WaitOne(__int32 
millisecondsTimeout = -1, bool exitContext = false)
*2      debugging.dll!Debugging.SharedResource.Wait()
 3      
debugging.dll!Debugging.Contention.btnObtainShared3_Click(System.O
bject sender = {System.Web.UI.WebControls.Button}, 
System.EventArgs e = {System.EventArgs})
  

You can see that this information is very similar to that obtained using WinDbg and the SOS extension.

Debugging a Hung Process with Visual Studio .NET

Complete the following steps to attach to the process.

To debug a hung process

1. Open a command prompt window, and then type iisreset at the prompt to restart IIS.
2. Right-click Contention.aspx in Visual Studio .NET, and then click View in Browser.
3. On the Debug menu, click Processes. A window that displays process information for local or remote machines appears.
4. In the Processes dialog box, select the Show system processes and Show processes in all sessions check boxes.
5. Select aspnet_wp.exe, and then click Attach. The Attach to Process dialog box appears.
6. Make sure that the Common Language Runtime item is selected, and then click OK.
7. In the Processes dialog box, make sure that the Aspnet_wp.exe process is listed in the Debugged Processes list, and then click Close.

You are now attached to the process and can use the debugger windows to display information about the process.

Conclusion

In this chapter, you have been introduced to some of the problems that can arise when executing multithreaded code. You have also discovered how managed and unmanaged threads are used within ASP.NET.

Using either WinDbg with the SOS.dll extension or Visual Studio .NET, you have been able to trace the execution of the threads within an ASP.NET worker process, and use a dump file to pinpoint the position in the code where contention occurred.