Production Debugging for .NET Framework Applications

 

Debugging Memory Problems

Summary: This chapter describes how to approach debugging memory consumption problems that users might experience when using ASP.NET applications. First, the chapter discusses how memory is managed in .NET, and in particular how the garbage collector (GC) reclaims unused memory. A walkthrough then shows how to debug a memory consumption scenario.

Although reproducing this problem on your machine might not give you the exact same results because of differing operating systems and runtime versions, the output should be similar, and the debugging concepts are the same. Also, because all .NET Framework applications use the same memory architecture, you can apply what you learn about memory management in an ASP.NET context to other .NET environments, such as console applications, Microsoft® Windows® operating system applications, and Windows services.

Contents

.NET Memory Management and Garbage Collection

Scenario: Memory Consumption

Conclusion

.NET Memory Management and Garbage Collection

C and C++ programs have traditionally been prone to memory leaks because developers had to manually allocate and free memory. In Microsoft® .NET, it is not necessary to do this because .NET uses garbage collection to automatically reclaim unused memory. This makes memory usage safer and more efficient.

The garbage collector (GC) uses references to keep track of objects that occupy blocks of memory. When an object is set to null or is no longer in scope, the GC marks the object as reclaimable. The GC can return the blocks of memory referenced by these reclaimable objects to the operating system.

The performance benefit of a GC arises from deferring the collection of objects, as well as from performing a large number of object collections at once. GCs tend to use more memory than typical memory management routines, such as those used by the Windows-based operating systems.

The GC in .NET uses the Microsoft Win32® VirtualAlloc() application programming interface (API) to reserve a block of memory for its heap. A .NET managed heap is a large, contiguous region of virtual memory. The GC first reserves virtual memory, and then commits the memory as the managed heap grows. The GC keeps track of the next available address at the end of the managed heap and places the next allocation request at this location. Thus, all .NET managed memory allocations are placed in the managed heap one after another. This vastly improves allocation time because it isn't necessary for the GC to search through a free list or a linked list of memory blocks for an appropriately sized free block, as normal heap managers do. Over time, holes begin to form in the managed heap as objects are deleted. When garbage collection occurs, the GC compacts the heap and fills the holes by moving allocations using a straight memory copy. Figure 2.1 shows how this works.

Figure 2.1. How the garbage collector compacts the heap

For more details on the .NET garbage collection mechanism, see the following references:

• "Garbage Collection: Automatic Memory Management in the Microsoft .NET Framework", by Jeffrey Richter, MSDN Magazine, November 2000. (https://msdn.microsoft.com/en-us/magazine/bb985010.aspx).
• "Garbage Collection—Part 2: Automatic Memory Management in the Microsoft .NET Framework", by Jeffrey Richter, MSDN Magazine, December 2000. (https://msdn.microsoft.com/en-us/magazine/bb985011.aspx).
• Chapter 19, "Automatic Memory Management (Garbage Collection)" in Applied Microsoft .NET Framework Programming by Jeffrey Richter (Microsoft Press, 2002).

Generations

The GC improves memory management performance by dividing objects into generations based on age. When a collection occurs, objects in the youngest generation are collected. If this does not free enough memory, successively older generations can also be collected. The use of generations means that the GC only has to work with a subset of the allocated objects at any one time.

The GC currently uses three generations, numbered 0, 1, and 2. Allocated objects start out belonging to generation 0. Collections can have a depth of 0, 1, or 2. All objects that exist after a collection with a depth of 0 are promoted to generation 1. Objects that exist after a collection with a depth of 1, which will collect both generation 0 and 1, move into generation 2. Figure 2.2 shows how the migration between generations occurs.

Figure 2.2. Migration between generations during multiple collections

Over time, the higher generations are filled with the oldest objects. These higher generations should be more stable and require fewer collections; therefore, fewer memory copies occur in the higher generations.

Collection for a specific generation occurs when the memory threshold for that generation is hit. In the implementation of .NET Version 1.0, the initial thresholds for generations 0, 1, and 2 are 256 kilobytes (KB), 2 megabytes (MB), and 10 MB, respectively. Note that the GC can adjust these thresholds dynamically based on an application's patterns of allocation. Objects larger than 85 KB are automatically placed in the large object heap, which is discussed later in this chapter.

Roots

The GC uses object references to determine whether or not a specific block of memory in the managed heap can be collected. Unlike other GC implementations, there is not a heap flag on each allocated block indicating whether or not the block can be collected. For each application, the GC maintains a tree of references that tracks the objects referenced by the application. Figure 2.3 shows this tree.

Figure 2.3. Root reference tree

The GC considers an object to be rooted if the object has at least one parent object that holds a reference to it. Every application in .NET has a set of roots, which includes global and static objects, as well as associated thread stacks and dynamically instantiated objects. Before performing a garbage collection, the GC starts from the roots and works downward to build a tree of all variable references. The GC builds a master list of all live objects, and then walks the managed heap looking for objects that are not in this live object list.

This would appear to be an expensive way of determining whether or not an object is alive, compared with using a simple flag in a memory block header or a reference counter, but it does ensure complete accuracy. For example, an object reference counter could be mistakenly over-referenced or under-referenced, and a heap flag could be mistakenly set as deleted when there are live references to the memory block. The managed heap avoids these issues by enumerating all live objects and building a list of all referenced objects before collection. As a bonus, this method also handles circular memory reference issues.

If there is a live reference to an object, that object is said to be strongly rooted. .NET also introduces the notion of weakly rooted references. A weak reference provides a way for programmers to indicate to the GC that they want to be able to access an object, but they don't want to prevent the object from being collected. Such an object is available until it is collected by the GC. For example, you could allocate a large object, and rather than fully deleting and collecting the object, you could hold onto it for possible reuse, as long as there is no memory pressure to clean up the managed heap. Thus, weak references behave somewhat like a cache.

Large Object Heap

The .NET memory manager places all allocations of 85,000 bytes or larger into a separate heap called the large object heap. This heap consists of a series of virtual memory blocks that are independent from the main managed heap. Using a separate heap for larger objects makes garbage collection of the main managed heap more efficient because collection requires moving memory, and moving large blocks of memory is expensive. However, the large object heap is never compacted; this is something you must consider when you make large memory allocations in .NET.

For example, if you allocate 1 MB of memory to a single block, the large object heap expands to 1 MB in size. When you free this object, the large object heap does not decommit the virtual memory, so the heap stays at 1 MB in size. If you allocate another 500-KB block later, the new block is allocated within the 1 MB block of memory belonging to the large object heap. During the process lifetime, the large object heap always grows to hold all the large block allocations currently referenced, but never shrinks when objects are released, even if a garbage collection occurs. Figure 2.4 shows an example of a large object heap.

Figure 2.4. Large object heap

Scenario: Memory Consumption

Now that you have been introduced to .NET memory management and garbage collection mechanisms, let's see how these are used in ASP.NET applications. This scenario investigates how to debug memory consumption problems. As you might already know, memory leaks are caused by not deallocating memory that was dynamically allocated. A small memory leak might not be noticed and might cause only minimal damage, but large memory leaks could severely impact performance by draining available memory. In addition, there might be other memory-related problems that aren't "true" memory leaks, but which exhibit memory-leak symptoms. This scenario focuses on the latter category of memory-related problems.

Here are some typical customer scenarios that could indicate memory problems:

• Scenario 1: An e-commerce Web site sells products. Browser clients complain of losing data and seeing Server Application Unavailable errors. They have to log in again and don't understand why. In addition, server performance slows when memory usage increases.
• Scenario 2: A Web site provides a file upload facility for video files. When a browser client uploads a large file, the process recycles and the file upload fails.

Breaking Down the Thought Processes

There are many things to check when trying to debug memory-related problems. Figure 2.5 shows a flowchart you can follow when troubleshooting consumption problems.

Figure 2.5. Flowchart for troubleshooting memory consumption issues

Although the thought processes are described in the walkthrough that follows, let's look at the flowchart in more detail.

Do the Errors Point to Memory Leak Symptoms?

If you're using Microsoft Internet Information Services (IIS) 5.x, check to see if there is an application event log error that indicates that an ASP process was recycled. If this has occurred, you'll see an error message similar to the following: "aspnet_wp.exe (PID: 1234) was recycled because memory consumption exceeded n MB (n percent of available RAM)."

Where Did My Memory Go?

The next step is to determine which process is using too much memory. Tools such as System Monitor (known as Performance Monitor in Windows 2000) or Task Manager can isolate those processes. Once you know which process is responsible, you can run AutoDump+ (ADPlus) to create a full memory dump of the Aspnet_wp.exe process and then, using the WinDbg debugger and the SOS.dll debugger extension, examine the differences between managed and native memory. This is discussed later in this chapter.

If the project is being run in a controlled environment (for example, in development, quality assurance (QA), or test), reproduce the problem and run ADPlus in -hang mode to produce a full memory dump of the Aspnet_wp.exe process. To allow more time before the process becomes recycled, you can use the memorylimit attribute on the <processModel> element in machine.config to increase the limit from the default of 60 percent to a higher percentage.

You can also use .NET APIs or System Monitor to gather more information. For example, with System Monitor, you can look for patterns that indicate either a .NET managed memory leak or a native memory leak.

Native Memory Consumption

If the Private Bytes counter in System Monitor increases while the .NET # of Bytes in all Heaps counter stays flat, this is evidence of native memory consumption. These counters are discussed in more detail in "Memory Consumption Walkthrough" later in this chapter.

Managed Memory Consumption

When using System Monitor, if both the Private Bytes counter and the .NET # of Bytes in all Heaps counter increase at the same rate, this is evidence of managed memory consumption. Look at the size of allocated managed memory and consider what the GC is doing. When looking at the dump file, use SOS commands to list the managed heap size and compare it to the total size of the dump file. Consider what details you can learn about the large object heap and the generations. Look to see if large objects (85 KB or more) or smaller objects consume most of the memory. If it's the former, look at the details of the large object heap. If it's the latter, consider what generations 0, 1, and 2 contain. For large objects, determine if they are rooted, and whether or not they should be rooted. If they aren't rooted, they are candidates for collections. Determine if they are eventually collected properly.

With WinDbg and SOS.dll, it can be difficult to look at all of the small objects' details if there are many small objects. In such cases, it might be easier to use the Allocation Profiler to look at details for both large and small objects. "Memory Consumption Walkthrough" uses all of these tools to diagnose a memory consumption issue.

Memory Consumption Walkthrough

The following walkthrough presents a simplified, realistic scenario: An ASP.NET application allocates too much memory and ends up recycling because it breaches the memory limit allowed in the process model. The purpose of the walkthrough is to illustrate techniques that can be used to help you troubleshoot such a problem. Production environment scenarios might not be as clear cut as this simplified example, but you can apply similar techniques to help identify the causes of memory consumption.

This walkthrough deals with a large memory leak that could severely impact performance by draining available memory. An example ASP.NET page allocates large objects (20 MB each), and then caches them in the ASP.NET cache. By default, when memory consumption exceeds 60 percent of available RAM, the ASP.NET process is recycled. The walkthrough aims to determine which process is using too much memory, and why.

In this scenario, you will perform the following steps:

1. Browse to the ASP.NET page at https://localhost/debugging/memory.aspx and consider the values displayed in the process model table.
2. Follow the thought processes in the flowchart and find the errors presented both in the browser and the application event log.
3. Create a dump file and examine the dump data using WinDbg and SOS.dll.
4. Inspect the managed memory for the ASP.NET worker process and determine which objects are consuming the most memory.
5. After you've found evidence in the dumps for possible problem areas, look at the source code.

Baseline View

First, gather some baseline data for comparisons with the native and managed memory consumptions throughout the walkthrough. For instructions on how to download the sample pages, see Chapter 1, "Introduction to Production Debugging for .NET Framework Applications."

To view Memory.aspx

• Open a browser window, and then browse to https://localhost/Debugging/memory.aspx.

  Note the information displayed on the page, especially the fields and values in the table.

The following browser view provides a baseline from which to compare data after you create objects that consume memory.

Figure 2.6. Baseline browser data

The following table describes what these fields display:

Table 2.1: Description of fields in Memory.aspx

Field	Description
StartTime	The time at which this Aspnet_wp.exe process started
Age	The length of time the process has been running
ProcessID	The ID assigned to the process
RequestCount	Number of completed requests, initially zero
Status	The current status of the process: if Alive, the process is running; if Shutting Down, the process has begun to shut down; if ShutDown, the process has shut down normally after receiving a shut down message from the IIS process; if Terminated, the process was forced to terminate by the IIS process
ShutdownReason	The reason why a process shut down: if Unexpected, the process shut down unexpectedly; if Requests Limit, requests executed by the process exceeded the allowable limit; if Request Queue Limit, requests assigned to the process exceeded the allowable number in the queue; if Timeout, the process restarted because it was alive longer than allowed; if Idle Timeout, the process exceeded the allowable idle time; if Memory Limit Exceeded, the process exceeded the per-process memory limit
PeakMemoryUsed	The maximum amount of memory the process has used. This value maps onto the Private Bytes (maximum amount) count in System Monitor

You can explore the code and read through the comments by opening another instance of Memory.aspx.cs in either Microsoft Visual Studio® .NET or Notepad.

Each time the Allocate 20 MB Objects button is clicked, five unique cache keys and five 20-MB objects are created and stored in the System.Web cache. The Allocate 200 K Objects button works in a similar way, but it creates 200-KB objects that are stored in the System.Web cache. The code behind this button simulates a non-fatal leaking condition for experimenting with alternative testing scenarios.

Free Memory obtains a list of cache keys from Session scope and clears the cache. It also calls the System.GC.Collect() method to force a garbage collection.

Note

   

System.GC.Collect() is used for demonstration purposes, not as a recommended procedure. Explicitly calling GC.Collect() changes the GC's autotuning capabilities. Repeatedly calling GC.Collect() suspends all threads until the collection completes. This could greatly impede performance.

Finally, Refresh Stats refreshes the memory statistics.

You'll use Task Manager and System Monitor to confirm or discover more information about the ASP.NET process:

To use Task Manager

1. Open Task Manager.

2. On the Processes tab, click the View menu, click Select Columns, and then select the Virtual Memory Size check box.

   This adds virtual memory size to the default columns.

   Note

      

   If you are using Windows 2000 Server through a Terminal Server Client, you need to select the Show processes from all users check box.

What is the Aspnet_wp.exe process's virtual memory size? In this test, the virtual memory size of Aspnet_wp.exe is 9,820 KB. This value is a reasonable baseline.

Table 2.2: Baseline Task Manager data

Process	Virtual memory size
Aspnet_wp.exe	9,820 KB

Start the System Monitor.

To start the System Monitor

1. In Control Panel, click Administrative Tools, and then double-click Performance.

2. On the System Monitor toolbar, click '+', and then in the Add Counters dialog box, select .NET CLR Memory from the Performance object drop-down list.

3. In the Select instances from list box, click aspnet_wp.

4. Use CTRL+Click to select the entries shown in Figure 2.7 from the Counters list box, and then click Add.

   Repeat the process for the ASP.NET Applications and Process performance objects, and click Add after selecting the appropriate instances and counters.

5. Click Close, and then click the View Report button on the toolbar to switch to text view. You should see a screen similar to the one in Figure 2.7 below.

Figure 2.7. Baseline System Monitor data

Using the System Monitor report, pay particular attention to # Bytes in all Heaps, Large Object Heap Size, and Cache API Entries.

• # Bytes in all Heaps displays the sum of the Gen 0 Heap Size, Gen 1 Heap Size, Gen 2 Heap Size, and Large Object Heap Size counters. The # Bytes in all Heaps counter indicates the current memory allocated in bytes on the garbage collection heaps.
• Large Object Heap Size displays the current size, in bytes, of the large object heap. Note that this counter is updated at the end of a garbage collection, not at each allocation.
• Cache API Entries is the total number of entries in the application cache.

The baseline values before allocating any memory are:

• Large Object Heap Size: 22,608 bytes
• # Bytes in all Heaps: 2,589,252
• Cache API Entries: 0

Note

   

To learn more about the System Monitor .NET Performance Counters, see:

• ".NET Framework General Reference: Memory Performance Counters" on the MSDN Web site at https://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpgenref/html/gngrfmemoryperformancecounters.asp.
• ".NET Framework Developer's Guide: Performance Counters for ASP.NET" on the MSDN Web site at https://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpguide/html/cpconperformancecountersforaspnet.asp.

First Allocation Pass

Now that you've established the baseline statistics for the application and System Monitor counters with the ASP.NET process running, consider the amount of native and managed memory consumed. On the Memory.aspx page, click Allocate 20 MB Objects once. The data in your browser should resemble the following tables:

Table 2.3: Baseline and one allocation browser data

Field	Baseline value	New value
StartTime	07/08/2002 12:14:06 PM	07/08/2002 12:40:59 PM
Age	00:00:04.2961776	00:00:33:9585520
ProcessID	3,212	3,212
RequestCount	0	1
Status	Alive	Alive
ShutdownReason	N/A	N/A
PeakMemory Used	5,948	8,904

Updated Memory Stats	Baseline value	New value
GC.TotalMemory	780 KB	100,912 KB
Private Bytes	8,896 KB	117,616 KB

A new table appears on the page that contains two entries:

• GC.TotalMemory is the total amount of managed memory that the GC heap has allocated for managed objects. (This amount maps onto the System Monitor # Bytes in all Heaps counter for the .NET CLR Memory Performance object.)
• Private Bytes is the total size of native memory that a process has allocated that cannot be shared with other processes.

After clicking Allocate 20 MB Objects, the RequestCount field increments by one to count the completed request. You can also see that GC.TotalMemory and Private Bytes equal approximately 100 MB.

Look at the code for the ASP.NET page to see the source of the 100 MB:

for (int i = 0; i< 5; i++)
{
  long objSize = 20000000;
  string stime = DateTime.Now.Millisecond.ToString();
  string cachekey = "LOCache-" + i.ToString() + ":" + stime; 
  Cache[cachekey] = CreateLargeObject(objSize);
  StoreCacheListInSession(cachekey);
}
  

The Private Bytes figure contains both the managed and native allocations for the process.

The example scenario displays the Private Bytes and GC.TotalMemory memory statistics on the ASP.NET page itself. When production debugging, you might need to use other tools, such as Task Manager and System Monitor, to obtain this information. You can use Task Manager to display native data, while System Monitor shows both managed and native memory data.

On the Processes tab in Task Manager, locate the Aspnet_wp.exe process, which has the same process ID (PID). You can see that the virtual memory size has increased to almost 115,000 KB.

Table 2.4: Baseline and one allocation Task Manager data

Process	Baseline value	New value
Aspnet_wp.exe	9,820 KB	114,840 KB

The virtual memory size has increased by about 100,000 KB. The ASP.NET process is consuming most of the memory on the machine.

Note

   

The VM Size column in Task Manager roughly maps to the System Monitor Process: Private Bytes counter.

In the chart view of System Monitor, you'll see that the Private Bytes and # Bytes in all Heaps counter values are both increasing at the same rate. This implies an increase in the use of managed memory, rather than native memory.

Figure 2.8 shows the increase in memory usage when the button is clicked. To change the maximum value of the y-coordinate on a graph, right-click the graph, and then select Properties. Click the Graph tab, and then change the value in the Vertical scale Maximum box.

Figure 2.8. System Monitor graph

Second Allocation Pass

The next step consumes more memory. Click Allocate 20 MB Objects once more, making two total clicks. Note how the values reported on the ASP.NET page change, especially RequestCount, PeakMemoryUsed, and GC.TotalMemory.

Table 2.5: Baseline, one allocation, and two allocations browser data

Field	Baseline value	One allocation	Two allocations
StartTime	07/08/2002 12:14:06 PM	07/08/2002 12:40:59 PM	07/08/2002 13:25:51 PM
Age	00:00:04.2961776	00:00:23:9585520	00:47:33.1343328
ProcessID	3,212	3,212	3,212
RequestCount	0	1	2
Status	Alive	Alive	Alive
ShutdownReason	N/A	N/A	N/A
PeakMemory Used	5,948	8,904	113,752

Memory Stats	Baseline value	One allocation	Two allocations
GC.TotalMemory	780 KB	100,912 KB	200,916 KB
Private Bytes	8,896 KB	117,616 KB	221,450 KB

To see the results of this second button-click event on system memory, look at the details for the process in Task Manager.

Table 2.6: Baseline, one allocation, and two allocations Task Manager data

Process	Baseline value	One allocation	Two allocations
Aspnet_wp.exe	9,820 KB	114,840 KB	216,240 KB

The VM Size value has gone up another 100,000 KB, and the Aspnet_wp.exe process is still consuming most of the memory.

The System Monitor report and the chart views confirm these values. Now look at the Cache API Entries counter. As you saw in the code, each click event runs a loop that creates five cache items. The loop has been run twice, so there have been ten cache API entries.

The difference between the #Bytes in all Heaps and the Large Object Heap Size counters indicates that most of the managed memory is being used by the large object heap. This is shown in figure 2.9.

Figure 2.9. System Monitor data

Third Allocation Pass

The final step consumes even more memory. Click Allocate 20 MB Objects a third time, quickly switch to the Performance tab of the Task Manager, and examine the Page File Usage History.

Note

   

In Windows XP, the Task Manager displays Page File Usage on the Performance tab, whereas in Windows 2000, the Task Manager displays Memory Usage on the Performance tab.

Each time the Allocate 20 MB Objects button is clicked, the graph shows a steep incline, which levels off until the next time the button is clicked. If the process is recycled, you will see a sudden drop in Page File Usage. Figure 2.10 shows this happening in Task Manager.

Figure 2.10. Task Manager data

Task Manager can be used for a "guesstimate" of overall system performance. Committed memory is allocated to programs and the operating system, and the Page File Usage History graphs these values. For each click event, you can see the commit charge value increase proportionately to the graph.

You can also see the results of the Aspnet_wp.exe process being recycled. As noted, memory usage increases with each button click and is not released until the process is recycled. You can check that the process has been recycled by looking in the application event log.

Event Type:     Error
Event Source:   ASP.NET 1.0.3705.0
Event Category: None
Event ID:       1001
Date:           6/7/2002
Time:           1:04:14 AM
User:           N/A
Computer:       machinename
Description:     aspnet_wp.exe  (PID: 2740) was recycled because 
               memory consumption exceeded the 306 MB 
               (60 percent of available RAM).
  

Switch back to the ASP.NET page in the browser, and click the Refresh Stats button. You can now see a second table added to the page, which contains data from a new process. The Process Model health monitoring shut down the original process because the memory limit was reached, and then started a new process. Compare the two tables and look particularly at the Status and ShutdownReason fields, and the Updated Memory Stats table.

For the first process, the value of the Status field has changed from Alive to Terminated, while the second process has become Alive. The first process has been terminated because the memory limit was exceeded. Note that ASP.NET terminates the process after you allocate memory and exceed the maximum memory-peak memory limit.

Because the process was recycled, the Updated Memory Stats table shows the baseline values again.

Exploring Further

Refresh Stats only updates the statistics. To vary the results, you can experiment with the controls on the browser interface:

• Click Allocate 20 MB Objects again instead of clicking Refresh Stats. The RequestCount value in the first table increments, and the second table shows a Request Count of 0 (a new worker process was created). The Process Model uses intelligent routing when the memory limit is reached. It acknowledges the request, kills the old process, and creates a new process to accommodate requests.
• Click Allocate 20 MB Objects one more time. This increments the second table's RequestCount value because we're dealing with the new worker process and the memory limit has not been met for that process.
• Click Allocate 200 K Objects repeatedly. This creates multiple 200-KB objects that eventually accumulate in the large object heap. You might want to try this option, as it uses a similar code path with less memory usage.
• Let's review the conclusions that can be made so far:
• You're dealing with managed memory usage. The System Monitor chart view shows how the Private Bytes and # Bytes in all Heaps counters both increase at the same slope.
• The Aspnet_wp.exe process consumes most of the memory. Once the Aspnet_wp.exe process recycles, the committed memory count normalizes.
• Most of the managed heap memory is in the large object heap. The size of the large object heap is almost the same as the value of the Total # Bytes in all Heaps counter.

Debugging User-Mode Dump Files with WinDbg

To confirm these findings and learn more about what is happening in the ASP.NET process, use ADPlus to create a dump file, and then examine the dump file with WinDbg and SOS.dll. For more information on ADPlus, see article Q286350, "HOWTO: Use Autodump+ to Troubleshoot 'Hangs' and 'Crashes'" in the Microsoft Knowledge Base at https://support.microsoft.com/default.aspx?scid=kb;en-us;Q286350. Also, if you want to run ADPlus through Terminal Server, see article Q323478, "PRB: You Cannot Debug Through a Terminal Server Session" in the Microsoft Knowledge Base at https://support.microsoft.com/default.aspx?scid=kb;en-us;Q323478.

To start from the beginning

• Open a command prompt window, and then type iisreset at the prompt to restart IIS.

By default, when memory consumption exceeds 60 percent of available RAM, an Aspnet_wp.exe process is recycled. The time before the memory limit is reached could be short, and might allow only minimal time for debugging. To prevent the process from terminating before you can debug it, modify the registry to alter the default behavior of ASP.NET when a process should be recycled.

Adding Registry Keys

You need to add two registry values: DebugOnHighMem and UnderDebugger. These registry settings tell ASP.NET to call DebugBreak instead of terminating the process.

Note: The following exercise requires that you make changes to the registry. It is always recommended that you back up the registry before making changes. In addition, when you have finished debugging, it is recommended that you delete the DWORDs created below. These registry DWORDs are undocumented, and should only be used during specific debugging sessions.

To create the registry values

1. Start Regedit, and then locate the following key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ASP.NET.
2. Right-click ASP.NET, point to New, and then select DWORD Value.
3. Change the name of the new entry from New Value #1 to DebugOnHighMem.
4. Double-click the new name and change the HEX data value from 0 to 1.
5. Create another New DWORD Value and name it UnderDebugger. Leave the data value as HEX 0.

You should end up with the settings shown in figure 2.11.

Figure 2.11. Registry Editor

The memory limit monitoring mechanism is automatically disabled whenever a debugger is attached to an Aspnet_wp.exe process, unless the value of the UnderDebugger DWORD value is set to zero, in which case the mechanism is not disabled.

Configuring ADPlus and Creating a Dump File

ADPlus can be configured to create a full dump if CTRL+C is pressed while it is running in -crash mode. Open the ADPlus.vbs script and change the value of the Full_Dump_on_CONTRL_C from FALSE to TRUE. Here's the relevant code from ADPlus.vbs:

' Set Full_Dump_on_CONTRL_C to TRUE if you wish to have AD+ produce a full  
' memory dump (as opposed to a mini-memory dump) whenever CTRL-C is pressed to stop 
' debugging, when running in 'crash' mode.
' Note:  Depending upon the number of processes being monitored, this could 
' generate a significant amount of memory dumps.
 
Const Full_Dump_on_CONTRL_C = TRUE

  

Even though the process is not crashing, you can use ADPlus in -crash mode to gather process data. To capture the information, run the ASP.NET pages again.

To Run ADPlus in -crash mode

1. Browse to https://localhost/debugging/memory.aspx.
2. Open a command prompt and start ADPlus from the command line.
3. Specify the -crash switch and the process name, which in this case is Aspnet_wp.exe.

You can also use the -quiet option to suppress extra program information, as in this example:

c:\debuggers>adplus.vbs -pn aspnet_wp.exe  -crash -quiet  
  

In this case, the ADPlus output will be found in a uniquely named subdirectory of C:\debuggers. To specify an output directory, use the -o switch to specify a path, as shown in the following example:

c:\debuggers>adplus.vbs -pn aspnet_wp.exe -crash -quiet -o 
c:\labs\highMemory
  

Note

   

The -quiet switch ensures that you can still create a dump file if the symbol path for ADPlus is not set.

Output from the debugger will be similar to the following:

C:\debuggers>adplus.vbs -pn aspnet_wp.exe -crash -quiet
The '-crash' switch was used, Autodump+ is running in 'crash' mode.
The '-quiet' switch was used, Autodump+ will not display any modal dialog boxes.

Monitoring the processes that were specified on the command line
for unhandled exceptions.
----------------------------------------------------------------------
Attaching the CDB debugger to: Process - ASPNET_WP.EXE with PID–344
  

You should also see a minimized window running CDB.exe.

At this point, you create a dump file so that you can examine what is consuming memory.

Note

   

The dump files generated by ADPlus can be extremely large. Every time the Allocate 20 MB Objects button is clicked, 100 MB is allocated, so consider the amount of free disk space you have on your machine. If you have 512 MB of RAM and you click the button three times, you need 300+ MB of available disk space.

To create a dump file, use the ASP.NET page to create objects that consume memory.

To allocate memory

1. Switch to the Memory.aspx page in the browser.
2. Click Allocate 20 MB Objects three times. The memory limit is reached after the third button-click, at which point CDB.exe creates a dump.
3. Click Refresh Stats.

The debugger terminates the process, generates a log, and exits. Locate the files generated by ADPlus. If you didn't specify a path on the ADPlus command line, these files will be placed in a directory whose name is made up from the date and time at which the crash occurred, as shown in the following example:

c:\debuggers\Crash_Mode__Date_05-19-2002__Time_19-41-21PM
  

The following describes the files typically generated by ADPlus:

• Autodump+-report.txt is a report that ADPlus.vbs creates. It displays the list of constants that ADPlus knows about, and the constants' current settings. This report is used for diagnostic purposes if the ADPlus.vbs script doesn't function correctly after it has been modified from its original form.
• PID-1234__process.exe.cfg is passed to CDB.exe to script the commands that need to be run. This file can be useful for diagnosing problems when running ADPlus.vbs.
• PID-1234__PROCESS.EXE__Reason__full/mini_timestamp.dmp is the full dump file for the process. The number after the PID is the process ID for the dumped process. Process.exe will be replaced by the name of the process; mini_timestamp will be replaced by a timestamp; and Reason will be replaced by a string identifying the reason for the dump, for example "CTRL+C" for a crash dump. Therefore, a typical dump file name might be PID-344__ASPNET_WP.EXE__CTRL-C__full_2002-05-19_19-43-59-798_0158.dmp. This sample file was 358 MB in size.
• PID-1234__PROCESS.EXE__Reason__full/mini_timestamp.log is an output log of the debugger. This file can be useful if you are looking for some history of exceptions or events prior to the failure. Once again, parts of the file name will be replaced with the PID, the process name, and the reason.
• Process_List.txt is the output of Tlist.exe -v. This file is used by ADPlus.vbs to determine the PIDs and names of processes running on the machine.

Examining the Dump File

Examine the dump file that was created when Aspnet_wp.exe process was recycled.

To examine the dump file using WinDbg

1. On the Start menu, point to Debugging Tools for Windows, and then click WinDbg.
2. On the File menu, click Open Crash Dump.
3. Select the appropriate dump file, and then click Open.
4. If you are prompted to "Save Base Workspace Information," click No.
5. If a Disassembly window pops up, close the window, and then on the Window menu, click Automatically Open Disassembly.

Symbol paths must be entered for the modules that will be used by the debugger to analyze the dump file. The symbol file versions that are needed on the debugging computer should match the versions on the system that produced the dump file. In this scenario, you need to include symbols for the .NET Framework SDK, Visual C Runtime, and Windows.

To enter the symbol paths, do one of the following:

• From the command line, type:

  .sympath 
  SRV*c:\symbols\debugginglabs*https://msdl.microsoft.com/download/symbols;C:\
  symbols\debugginglabs;C:\Program Files\Microsoft Visual Studio 
  .NET\FrameworkSDK\symbols;C:\windows\system32
  

• Enter the symbol path for WinDbg through the _NT_SYMBOL_PATH environment variable.

• On the File menu in WinDbg, select Symbol File Path, and then type:

  SRV*c:\symbols\debugginglabs*https://msdl.microsoft.com/download/symbols;C:\
  symbols\debugginglabs;C:\Program Files\Microsoft Visual Studio 
  .NET\FrameworkSDK\symbols;C:\windows\system32
  
  

The "SRV" in the path tells WinDbg to go to an external symbol server and copy symbols into the local symbol cache.

To use these same symbol paths for other dumps

• On the File menu, select Save Workspace As, and then type a name for the saved paths, such as .NET Debugging Symbols.

WinDbg debugs native code. To examine managed code, you need to load the SOS.dll debugger extension.

Note

   

If you haven't already done so, install the SOS.dll debugger extension. For download instructions, see Chapter 1, "Introduction to Production Debugging for .NET Framework Applications."

To examine managed code

1. Press ALT+1 to open the WinDbg command window.
2. Type .load <SOS>\SOS.dll to load the extension, and then replace <SOS> with the SOS.dll location.
3. Use the !findtable command to initialize SOS with the table information for the runtime version you will be debugging.

The output should be similar to the following:

0:001> .load sos\sos.dll
0:001> !findtable
Args: ''
Attempting data table load from SOS.WKS.4.3215.11.BIN
Failed timestamp verification.
  Binary time: 05:30:57, 2002/1/5.
  Table time: 03:40:50, 2001/9/5
Attempting data table load from SOS.WKS.4.2916.16.BIN
Failed timestamp verification.
  Binary time: 05:30:57, 2002/1/5.
  Table time: 03:38:40, 2001/6/7
Attempting data table load from SOS.WKS.4.3705.BIN
Loaded Son of Strike data table version 4 from "SOS.WKS.4.3705.BIN"
  

Note

   

SOS.dll requires a .bin file that matches the .NET runtime version. In the example previous, two .bin files were examined before the correct version was found. Future versions of SOS will not require a .bin file.

Because you configured ADPlus to create a full dump, and this dump was generated with a reference to CTRL+C in the filename, you know that DebugBreak was called and excessive memory consumption was detected by the Process Model health monitoring.

ADPlus created the dump when the Aspnet_wp.exe process was recycled. As you saw in the application event log entry, the memory consumption exceeded 60 percent of the available RAM. To troubleshoot this memory consumption issue, you need to examine the dump file, and this involves following some steps that are not always intuitive. These steps are explained in the following sections.

Analyzing the Dump: Summary

Before diving into the details, here's a summary of the steps you'll follow to analyze the dump file.

When dealing with memory consumption issues, consider what the GC is doing. You can use the !eeheap WinDbg command to list the managed heap size, and then compare this size to the total size of the dump file.

Next discover what .NET managed objects are taking up space on the managed heap. In this example, use the !dumpheap command to discover how many System.Byte [] objects are in the large object heap.

You need to know how long these objects will exist and how large they are. The !gcroot command shows that the System.Byte [] objects have a strong reference, which means they are rooted. The System.Web.Caching.Cache class references the System.Byte [] objects, which is why the objects are rooted.

You have now found objects referenced by the ASP.NET cache, so attempt to discover how much memory is referenced by the cache.

To find the memory address of the System.Web.Caching.Cache, use the !name2ee command qualified with the assembly and class name. This returns a MethodTable address, and you can use the !dumpheap command to dump the managed heap contents for the objects with that MethodTable.

The !dumpheap command gives the actual System.Web.Caching.Cache object address, which can then be used with the !objsize command to find the number of bytes that the root keeps alive on the GC heap. Because this size is comparable to the size of the dump file, you can infer that most of the managed memory is being referenced by the System.Web.Caching.Cache object.

Analyzing the Dump: Details

First consider the size of the GC heap, and then compare it to the total size of the dump file.

Use the !eeheap command to list the GC Heap Size and starting addresses for generation 0, 1, 2, and the large object heap. You can then compare the GC Heap Size to the total size of the dump file.

Examine the GC Heap Size with !eeheap -gc:

0:001> !eeheap -gc
generation 0 starts at 0x0110be64
generation 1 starts at 0x01109cd8
generation 2 starts at 0x01021028
 segment    begin allocated     size
01020000 01021028  0110de70 000ece48(970312)
Total Size   0xece48(970312)
------------------------------
large block  0x11e1fc04(300022788)
large_np_objects start at 17b90008
large_p_objects start  at 02020008
------------------------------
GC Heap Size  0x11f0ca4c(300993100)
  

In this case, the GC Heap Size is over 300 MB, and the size of the entire dump was 358 MB. Use the !dumpheap -stat command to discover which .NET managed objects are taking up space on the GC Heap. This command produces output under four columns labeled MT, Count, TotalSize, and Class Name. The MT output represents the Method Table, which identifies the type of the object, and the MTs are listed in ascending order of total size. TotalSize represents the object size times the number of objects.

Note

   

Because of the amount of data that it has to collect and display, the !dumpheap–stat command can take a considerable amount of time to execute.

The following listing shows the first few of the 14,459 objects listed by the !dumpheap–stat command.

0:001> !dumpheap -stat
Bad MethodTable for Obj at 0110d2a4
Last good object: 0110d280
total 14459 objects
Statistics:
      MT    Count TotalSize Class Name
3c6185c        1        12 System.Web.UI.ValidatorCollection
3c2e110        1        12 System.Web.Configuration.MachineKeyConfigHandler
3c29778        1        12 System.Web.Configuration.HttpCapabilitiesSectionHandler
3c23240        1        12 System.Web.SafeStringResource
3c22484        1        12 System.Web.Configuration.HandlerFactoryCache
3c22028        1        12 System.Web.UI.PageHandlerFactory
3c21a48        1        12 System.Web.Configuration.HttpHandlersSectionHandler
3b5f730        1        12 System.Web.Configuration.AuthorizationConfigHandler
3b5f54c        1        12 System.Web.Configuration.AuthorizationConfig
3b5e458        1        12 System.Web.Configuration.AuthenticationConfigHandler
3b5ccb8        1        12 System.Web.SessionState.InProcStateClientManager
3b5c528        1        12 System.Web.SessionState.SessionStateSectionHandler
3b5c1c4        1        12 System.Web.SessionState.SessionOnEndTarget
3b5c108        1        12 System.Web.Caching.OutputCacheItemRemoved
3b5c078        1        12 System.Web.Security.FileAuthorizationModule
3b5bfa8        1        12 System.Web.Security.UrlAuthorizationModule
  

The end of the dumpheap -stat list is shown in the second listing below. The objects with the largest TotalSize entries are at the bottom of the MT list and include System.String, System.Byte [], and System.Object []:

D12f28      1133     46052  System.Object[]
153cb0        88     76216  Free
321b278       85    178972 System.Byte[]
d141b0      6612    416720 System.String
Total 14459 objects
  

System.String is the last object in the list because its TotalSize is largest. The Free entry represents objects that are not referenced and have been garbage collected, but whose memory hasn't been compacted and released to the operating system yet.

This data is immediately followed by a list of the objects in the large object heap, which includes all objects over 85 KB.

large objects
 Address       MT     Size
17b90018  321b278 20000012 System.Byte[]
16220018  321b278 20000012 System.Byte[]
14f00018  321b278 20000012 System.Byte[]
13650018  321b278 20000012 System.Byte[]
12330018  321b278 20000012 System.Byte[]
11010018  321b278 20000012 System.Byte[]
 ec50018  321b278 20000012 System.Byte[]
 d560018  321b278 20000012 System.Byte[]
 bed0018  321b278 20000012 System.Byte[]
 a8a0018  321b278 20000012 System.Byte[]
 92d0018  321b278 20000012 System.Byte[]
 7d60018  321b278 20000012 System.Byte[]
 6850018  321b278 20000012 System.Byte[]
 53a0018  321b278 20000012 System.Byte[]
 3f50018  321b278 20000012 System.Byte[]
 2020018   d12f28     2064 System.Object[]
 2020840   d12f28     4096 System.Object[]
  

You can learn more about objects in the large object heap by using the !gcroot command, which finds the roots for objects. You specify an object by using its address from the dumpheap -stat output. For example, !gcroot 17b90018 provides information for the first System.Byte [] object in the previous listing. Here's the output from !gcroot:

0:001> !gcroot 17b90018
Scan Thread 1 (4e8)
Scan Thread 5 (bb0)
Scan Thread 6 (d0)
Scan Thread 10 (43c)
Scan Thread 11 (308)
Scan Thread 12 (6e4)
Scan HandleTable 14e340
Scan HandleTable 150e40
Scan HandleTable 1a6fa8
HANDLE(Strong):37411d8:Root:020784d8(System.Object[])-
>0108b504(System.Web.HttpRuntime)->0108b9d0(System.Web.Caching.CacheSingle)-
>0108ca68(System.Web.Caching.CacheUsage)->0108ca78(System.Object[])-
>0108cb3c(System.Web.Caching.UsageBucket)-
>010f95fc(System.Web.Caching.UsageEntry[])-
>01109c8c (System.Web.Caching.CacheEntry)->00000000()
  

The Scan Thread lines show that the !gcroot command is scanning threads for root references to objects. The Scan HandleTable lines show that the command is also scanning the HandleTables for roots. Each application domain (or AppDomain), which is an isolated environment where applications execute, has a HandleTable, and the HandleTable is simply another source of root references. References found for a thread might be in registers, arguments, or local variables.

If the output contains HANDLE(Strong), a strong reference was found. This means that the object is rooted and cannot be garbage collected. Other reference types can be found in the Appendix.

This dump shows similar data for each System.Byte [] object scanned by !gcroot. Try passing another address from the output of dumpheap -stat to !gcroot. The output is the same as for the previous System.Byte [] object, except for the highlighted address for System.Web.Caching.CacheEntry. The other classes denote collections.

0:001> !gcroot 16220018
Scan Thread 1 (4e8)
Scan Thread 5 (bb0)
Scan Thread 6 (d0)
Scan Thread 10 (43c)
Scan Thread 11 (308)
Scan Thread 12 (6e4)
Scan HandleTable 14e340
Scan HandleTable 150e40
Scan HandleTable 1a6fa8
HANDLE(Strong):37411d8:Root:020784d8(System.Object[])-
>0108b504(System.Web.HttpRuntime)->0108b9d0(System.Web.Caching.CacheSingle)-
>0108ca68(System.Web.Caching.CacheUsage)->0108ca78(System.Object[])-
>0108cb3c(System.Web.Caching.UsageBucket)-
>010f95fc(System.Web.Caching.UsageEntry[])-
>01109be8 (System.Web.Caching.CacheEntry)->00000000()
  

It would be useful to find out how much memory is referenced in the cache. To find the memory address of System.Web.Caching.Cache, use the !name2ee command. This command takes two parameters—the assembly name and the fully qualified class name, System.Web.Caching.Cache. If you don't know the assembly name, you can find it in the .NET Framework SDK documentation. Note that you can obtain similar data by using System.Web.Caching.CacheSingle.

0:001> !name2ee System.Web.dll System.Web.Caching.Cache
--------------------------------------
MethodTable: 03887998
EEClass: 03768814
Name: System.Web.Caching.Cache
--------------------------------------
  

Note

   

EEClass is an internal structure used to represent a .NET class. To learn more about the output from SOS commands, consult the SOS Help.

To dump the managed heap contents for objects of this type, use the !dumpheap command, passing the MethodTable address. For example:

!dumpheap–mt 03887998 
  

Here's the output you'd expect from !dumpheap:

0:001> !dumpheap -mt 03887998
 Address       MT     Size
0108b8ac 03887998       12
Bad MethodTable for Obj at 0110d2a4
Last good object: 0110d280
total 1 objects
Statistics:
      MT    Count TotalSize Class Name
 3887998        1        12 System.Web.Caching.Cache
Total 1 objects
large objects
 Address       MT     Size
total 0 large objects
  

Using the -mt switch might seem confusing because you don't need specific information about the MethodTable; however, when the address of an object is passed to !dumpheap -mt, all running objects of this type in this AppDomain will be listed. If there were multiple AppDomains, each instance of the object would be listed here.

You can use this information to get the size of each object by passing the address to the !objsize command. In this case, there is only one address, so to dump the size of System.Web.Caching.Cache, use the command !objsize 0108b8ac. Here's the output you'd expect to see from !objsize:

0:001> !objsize 0108b8ac
sizeof(0108b8ac) = 300126128 (0x11e38fb0) bytes (System.Web.Caching.Cache)
  

Note

   

Execute the !objsize command without any parameters to list all the roots in the process.

For more information about caching in ASP.NET, see "ASP.NET Performance Tips and Best Practices" on the GotDotNet Web site at https://www.gotdotnet.com/team/asp/ASP.NET Performance Tips and Tricks.aspx.

Freeing Managed Memory

Free the managed memory and consider the effect on virtual memory.

To make sure that everything is being restarted from scratch

1. Open a command prompt window, and then type iisreset at the prompt to restart IIS.

2. Browse to https://localhost/debugging/memory.aspx.

3. Run System Monitor and make sure that the following counters are set: Private Bytes and # Bytes in all Heaps for Aspnet_wp.exe.

4. Click Allocate 20 MB Objects once. You'll see output similar to Figure 2.8.

   Note the changes that System Monitor displays. Both Aspnet_wp.exe Private Bytes and # Bytes in all Heaps counters increase proportionately.

5. Click Allocate 20 MB Objects again and note the changes in the System Monitor chart.

   Each time the button is clicked, the Private Bytes and #Bytes in All Heaps counters increase proportionately.

Figure 2.12 shows the results in System Monitor after Allocate 20 MB Objects has been clicked twice. If you switch back to the browser and click Free Memory, you can see that the System Monitor log shows Maximum private bytes at 222 MB even after the button has been clicked.

Figure 2.12. System Monitor data after allocating memory

Here's an extract from the code behind the Free Memory button:

ArrayList arr = (ArrayList)Session["CacheList"];
foreach (string s in arr)
{
   if (Cache[s] != null)
   {
      Cache.Remove(s);
      Label1.Text += "<TABLE BORDER>";
      Label1.Text += "<TR><TD>";
      Label1.Text += "Emptying out cachekey " + s;
      Label1.Text += "</TD></TR></TABLE>";
   }
}
arr.Clear();
Session["CacheList"] = arr;
  

The code also contains a call to GC.Collect(). This code clears the cache objects and causes a garbage collection, but it does not result in the virtual memory allocated by large object heap being freed. Figure 2.13 shows the cache being emptied and the reduction of the GC.TotalMemory usage, but that the Private Bytes remains high.

Figure 2.13. Browser data after clicking Free Memory

When the Free Memory button was clicked, all but 1 MB of memory was freed. The # Bytes in all Heaps counter decreases significantly, but the Private Bytes counter does not change because of the large object heap. This is shown in figure 2.14.

Figure 2.14. System Monitor graph after clicking Free Memory

Capturing a Dump

To learn more about the state of the process, run ADPlus in -hang mode to capture a dump of process state:

adplus.vbs–hang–pn aspnet_wp.exe–quiet
  

Start WinDbg and load the dump file, set the symbol path, and load and initialize the SOS debugger extension. Now use the !eeheap -gc command to find the size of the managed heap.

0:000> !eeheap -gc
generation 0 starts at 0x012cc0e4
generation 1 starts at 0x012afde8
generation 2 starts at 0x011c1028
 segment    begin allocated     size
011c0000 011c1028  012d6000 00114fd8(1134552)
Total Size  0x114fd8(1134552)
------------------------------
large block  0x8060(32864)
large_np_objects start at 00000000
large_p_objects start  at 021c0008
------------------------------
GC Heap Size  0x11d038(1167416)
  

The dump shows that the GC Heap Size is 1 MB, not 222 MB, which indicates that all but 1 MB was collected. Because the objects that were allocated were 20 MB, you can conclude that they have all been removed from the heap. The Private Bytes counter for Aspnet_wp is still 222 MB, so you need to investigate this further.

In the debugger, use the !dumpheap command with the "-stat" flag. The output is similar to the following:

  ...
  d19dd4       79     23608 System.Char[]
 32169e8      382     26304 System.Int32[]
 32174a0      179     39048 System.Collections.Hashtable/bucket[]
  d12f28     1136     46352 System.Object[]
  153bb0       91    146208      Free
 321b278       90    179136 System.Byte[]
  d141b0     6626    472128 System.String
Total 14525 objects
large objects
 Address       MT     Size
 21c0018   e82f28     2064 System.Object[]
 21c0840   e82f28     4096 System.Object[]
 21c1858   e82f28     4096 System.Object[]
 2217cb0   e82f28     2064 System.Object[]
 22184d8   e82f28     4096 System.Object[]
 22194f0   e82f28     4096 System.Object[]
 221a508   e82f28     2064 System.Object[]
 221ad30   e82f28     4096 System.Object[]
 221bd48   e82f28     2064 System.Object[]
 226e930   e82f28     2064 System.Object[]
 226f158   e82f28     2064 System.Object[]
total 11 large objects
  

In summary, the ASP.NET page allocates large objects that are 20 MB, and then caches them in the ASP.NET cache. Because the objects that are allocated are greater than 85 KB, they are allocated on the large object heap. When objects are removed from the cache in code, the managed memory (shown by the System Monitor # Bytes in all Heaps counter) is reduced from 200 MB to 1 MB. The private bytes for the process do not decrease, however, and stay at about 222 MB. Although the managed memory was freed, the amount of memory that was allocated by the VirtualAlloc API remains allocated to the process.

This is a known limitation of the version 1.0 large object heap. The large object heap grows to accommodate the amount of memory of all objects that are alive at one time. If the objects are freed, the memory is reused for new large allocations; however, the process private bytes does shrink when the objects are freed. This will be addressed in .NET Framework version 1.1.

So far, this chapter has illustrated:

• How managed code can consume excessive memory.
• How and when ASP.NET process recycling might occur.
• Techniques used to debug memory consumption problems.
• How to use tools during the debugging process, such as System Monitor, Task Manager, WinDbg, and SOS.dll.

Diagnosing Memory Leaks with the Allocation Profiler

The Allocation Profiler is another tool that can provide more insight into the causes of memory consumption. This tool examines the managed heap when an application runs and presents a graphical display of memory allocation on the heap. In this section, you use this tool to show what happens when objects greater than 85 KB are allocated on the large object heap. The Allocation Profiler shows where the largest objects are rooted and the corresponding amount of allocated memory.

Before using Allocation Profiler with ASP.NET code, you need to modify the machine.config file and change the username attribute in the <processModel> element from machine to system to enable privileges that allow you to run the tool. When you have finished using the tool, change the username attribute back to machine to avoid the security risk associated with running under the system account.

To run Allocation Profiler

1. Run Allocationprofiler.exe from C:\Debuggers\AllocProf.

2. Select Profile ASP.NET from the File menu.

   Multiple informational command windows pop up. One of these indicates that the tool is stopping the IIS Admin service, while another indicates that it is starting the World Wide Web server. The Allocation Profiler window title changes to "Stopping IIS." IIS then restarts and the window title changes to "Waiting for ASP.NET worker process to start up." This means that the tool is waiting for the first ASP.NET page to invoke Aspnet_wp.exe.

3. Browse to https://localhost/debugging/memory.aspx to start the ASP.NET worker process. The Allocation Profiler window title changes to "Profiling: ASP.NET."

4. Clear the Profiling active check box, close the Allocation Graph for: ASP.NET window, and then select the Profiling active check box.

   The Allocation Profiler now starts gathering allocation statistics about the process.

5. Switch to the browser window and click Allocate 20 MB Objects twice.

6. Switch to the Allocation Profiler, and then clear the Profiling active check box.

7. Slide the horizontal scroll bar to the right to scroll through the allocations.

Figure 2.15. Allocation graph

The screen shows that the Button::OnClick handler calls the Memory::btn20MB_Click method, which creates 191 MB of System.Byte array objects.

To view the heap graph

1. Close the Allocation Graph for: ASP.NET window. This is the Allocation Graph for ASP.NET.
2. Switch to the Allocation Profiler window, and then click Show Heap Now. A new Allocation Profiler window called "Heap Graph for ASP.NET" appears.
3. Scroll to the far right of this window, where you will see the allocation for the System.Byte [] object.

Figure 2.16. Heap graph

The second to last rectangle shows that the System.Byte [] object is rooted in the System.Web cache. This is shown by the references in the reference tree to Caching.CacheSingle and Caching.CacheEntry. Note that Caching.CacheEntry was called and contains 191 MB. This indicates that most of the memory is held by the cache.

To view the histogram

1. Close the graph window.

2. In the Allocation Profiler window, click Histogram by Age on the View menu.

   

   Figure 2.17. Histogram by age

3. In the left pane of the Histogram by Age for Live Objects window, scroll to the right until you see a big red bar.

   The System.Byte [] objects are at the top of the list of total memory taken up by Live Objects.

4. Position the mouse arrow over the red bar to obtain more information.

5. Right-click the red bar that represents System.Byte [], and then click Show Who Allocated on the Context menu.

   The original Allocation graph for ASP.NET is displayed.

Allocation Profiler can also trace the root tree, showing callers, callees, and which call allocated which memory block.

To display root tree information

1. In the Allocation Graph for Object window, right-click the Memory: btn20MB_Click bar, and then select callers & callees. This highlights the entire call tree.
2. Right-click the highlighted area, click Copy as text to clipboard, and then past the text in Notepad.

The following information is displayed:

<root>:   191 MB    (100.00%)
 System.Web.Hosting.ISAPIRuntime::ProcessRequest:   191 MB    (100.00%)
  System.Web.HttpRuntime::ProcessRequest:   191 MB    (100.00%)
   System.Web.HttpRuntime::ProcessRequestInternal:   191 MB    (100.00%)
    System.Web.HttpApplication::System.Web.IHttpAsyncHandler.BeginProcessRequest:   
191 MB    (99.96%)
     System.Web.HttpApplication::ResumeSteps:   191 MB    (99.96%)
      System.Web.HttpApplication::ExecuteStep:   191 MB    (99.96%)
       CallHandlerExecutionStep::Execute:   191 MB    (99.96%)
        System.Web.UI.Page::ProcessRequest:   191 MB    (99.96%)
         System.Web.UI.Page::ProcessRequest:   191 MB    (99.96%)
          System.Web.UI.Page::ProcessRequestMain:   191 MB    (99.96%)
           System.Web.UI.Page::RaisePostBackEvent:   191 MB    (99.94%)
            System.Web.UI.Page::RaisePostBackEvent:   191 MB    (99.94%)
             System.Web.UI.WebControls.Button::System.Web.UI.IPostBackEventHandler.RaisePostBac
kEvent:   191 MB    (99.94%)
       System.Web.UI.WebControls.Button::OnClick:   191 MB    (99.94%)
               Debugging.Memory:: btn20MB_Click:   191 MB    (99.94%)
                                           ...    
System.Byte []:   191 MB    (100.00%)
                                <bottom>:   191 MB    (100.00%)
  

This tree shows how the call from btn20MB_Click created the large object.

When you have finished running the Allocation Profiler, click Kill ASP.NET. Change the username attribute on the <processModel> element in machine.config from system back to machine to restrict privileges again, avoiding the security risk that running under system presents.

Allocation Profiler examined the managed heap and showed that the 20-MB large object was created by the btn20MB_Click event. The actual code calls from btn20MB_Click to CreateLargeObject, which creates System.Byte []. With the Release build, CreateLargeObject does not appear due to optimization or inlining. This is discussed in Chapter 3, "Debugging Contention Problems." A Debug build would show the call from btn20MB_Click to CreateLargeObject. With both builds, you can see that the large object was rooted (and not collectable by the GC) and that a considerable amount of memory was allocated for the large objects' existence.

Conclusion

In this chapter, you have:

• Seen how to use Microsoft tools to diagnose and debug memory allocation problems.
• Learned how objects are allocated on managed heaps.
• Learned how .NET garbage collection works.

Task Manager and the System Monitor let you gather evidence that helps in the diagnosis of memory leaks. WinDbg, in conjunction with the SOS extension, allows you to pinpoint where managed memory is allocated in an application. The Allocation Profiler can be used to show how managed memory has been allocated, and also can be used to find object roots.

Finally, you have seen that you need to think carefully before using objects that are larger than 85 KB in ASP.NET applications because the memory in .NET Framework version 1.0 the large object heap is not reclaimed by the GC.