Partager via

I am getting OutOfMemoryExceptions. How can I troubleshoot this?

  • Article

Problem:

You've written an ASP.NET application that is getting OutOfMemoryExceptions.

Cause:

Let's find out...

Resolution:

Use Windbg to take a look at the heap.

 

Is it a leak?

Take a look at the memory usage of you application using perfmon. If memory is slowly increasing and never released, then you have a leak. If it is going up and down like a rollercoaster, then you are most likely using huge amounts of memory for certain operations which is later garbage collected.

How do I troubleshoot this?

Below I'll try to give you a step by step description of how to troubleshoot this scenario

1. Get a memory dump

This is done using Windbg and Adplus. If you don't have Windbg installed, check out my previous post.

 

2. Open up the dump and load SOS

Open the dump in Windbg and load the SOS extension. You'll find it in the framework directory so if you're debugging an application for Framework 2.0 look in "C:\Windows\Microsoft.NET\Framework\v2.0.50727". Anyway, load SOS by typing:

.load [path]\sos

 

 

3. Run dumpheap

Execute the following command:

!dumpheap -stat

It will show statistics for the objects on the heap in a nice little summary divided in tho four columns.

  1. The method table of the object
  2. The number of objects of this type on the heap
  3. The total size of these objects in bytes
  4. The name of the object type

Be careful not to omit the -stat parameter. If you do then Windbg will dump the address of each object in the entire heap to your screen, which will be a lot of information to say the least.

Analyzing the information provided by !dumpheap

Here's a sample heap from one of my cases...

0:000> !dumpheap -stat
Statistics:
MT             Count       TotalSize Class Name
7a787cc4           1              12 System.IO.FileSystemWatcher+FSWAsyncResult
7a75904c           1              12 System.Diagnostics.OrdinalCaseInsensitiveComparer
7a7575cc           1              12 System.CodeDom.Compiler.CodeDomConfigurationHandler
7a7571a8           1              12 System.Net.DefaultCertPolicy
7a75661c           1              12 System.Diagnostics.TraceListenerCollection
7a755834           1              12 System.Diagnostics.PerformanceCounterCategoryType
........CONTINUED.........
68a66a88     227,559      12,743,304 System.Web.UI.WebControls.Literal
68a2f7fc     399,272      14,373,792 System.Web.UI.ControlCollection
68a92e2c     768,731      33,824,164 System.Web.UI.Control+OccasionalFields
68a884a0     641,952      38,517,120 System.Web.UI.LiteralControl
79124228     879,515      43,394,976 System.Object[]
790fa3e0   1,431,594     122,806,484 System.String

Total 10,389,625 objects, Total size: 463,313,540

So in this particular dump I have 1,431,594 strings whose total size is 122 MBytes, 879,515 Objects with a total size of 43 MBytes, etc.

Objects in heap may be larger than they appear

The TotalSize column isn't 100% true. Look at the LiteralControls that are on third place. They only use a total of 38 MBytes, or do they? The TotalSize refers to the object structure, but the member variables like strings, integers and other child objects are not included. It kind of makes sense, since otherwise the Total size would be completely off the scale. The LiteralControl objects contain a number of child objects and three of those are strings. Their respective size is listed with the System.String object.

Using !dumpobj

Let's take a closer look at one of the System.Web.UI.LiteralControls. We list all of the controls with the following command !dumpheap -type System.Web.UI.LiteralControl and quickly press Ctrl+Break before the screen fills with too many lines:

0:000> !dumpheap -type System.Web.UI.LiteralControl
 Address       MT Size Gen
023ea0a8 68a884a0   60   2 System.Web.UI.LiteralControl
023ea0e4 68a884a0   60   2 System.Web.UI.LiteralControl
023ea374 68a884a0   60   2 System.Web.UI.LiteralControl
023ea460 68a884a0   60   2 System.Web.UI.LiteralControl
023ea510 68a884a0   60   2 System.Web.UI.LiteralControl
023eab3c 68a884a0   60   2 System.Web.UI.LiteralControl
........CONTINUED........
023fe31c 68a884a0   60   2 System.Web.UI.LiteralControl
023fe414 68a884a0   60   2 System.Web.UI.LiteralControl
023fe4c4 68a884a0   60   2 System.Web.UI.LiteralControl
023fe500 68a884a0   60   2 System.Web.UI.LiteralControl

As you can see each LiteralControl is 60 bytes in size. Like I said before that's the size of the object structure alone, not it's referenced objects and properties. We now pick the address of one of the LiteralControls and execute the !dumpobj command (!do for short). This gives us the following result:

0:000> !do 023ea0a8
Name: System.Web.UI.LiteralControl
MethodTable: 68a884a0
EEClass: 68a88428
Size: 60(0x3c) bytes
GC Generation: 2
(C:\WINDOWS\assembly\GAC_32\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
      MT    Field Offset                   Type   VT     Attr    Value Name
790fa3e0  4001fe0      4          System.String    0 instance 00000000 _id
790fa3e0  4001fe1      8          System.String    0 instance 00000000 _cachedUniqueID
68a2af44  4001fe2      c   ...em.Web.UI.Control    0 instance 023e8864 _parent
68a91070  4001fe3     2c           System.Int32    0 instance        0 _controlState
68a85ea0  4001fe4     10   ...m.Web.UI.StateBag    0 instance 00000000 _viewState
68a2af44  4001fe5     14   ...em.Web.UI.Control    0 instance 023e8864 _namingContainer
68a273d0  4001fe6     18     System.Web.UI.Page    0 instance 01df4514 _page
68a92e2c  4001fe7     1c   ...+OccasionalFields    0 instance 00000000 _occasionalFields
68a2b378  4001fe8     20   ...I.TemplateControl    0 instance 00000000 _templateControl
68a14528  4001fe9     24   ...m.Web.VirtualPath    0 instance 00000000 _templateSourceVirtualDirectory
68a8bb48  4001fea     28   ...rs.ControlAdapter    0 instance 00000000 _adapter
68a3a8f8  4001feb     30   ...SimpleBitVector32    1 instance 023ea0d8 flags
790f9c18  4001fda    c70          System.Object    0   shared   static EventDataBinding
>> Domain:Value 000f0d00:NotInit 0011a720:01df0028 <<
790f9c18  4001fdb    c74          System.Object    0   shared   static EventInit
>> Domain:Value 000f0d00:NotInit 0011a720:01df0034 <<
790f9c18  4001fdc    c78          System.Object    0   shared   static EventLoad
>> Domain:Value 000f0d00:NotInit 0011a720:01df0040 <<
790f9c18  4001fdd    c7c          System.Object    0   shared   static EventUnload
>> Domain:Value 000f0d00:NotInit 0011a720:01df004c <<
790f9c18  4001fde    c80          System.Object    0   shared   static EventPreRender
>> Domain:Value 000f0d00:NotInit 0011a720:01df0058 <<
790f9c18  4001fdf    c84          System.Object    0   shared   static EventDisposed
>> Domain:Value 000f0d00:NotInit 0011a720:01df0064 <<
79124228  4001fec    c88        System.Object[]    0   shared   static automaticIDs
>> Domain:Value 000f0d00:NotInit 0011a720:01df0070 <<
790fa3e0  4002211     34          System.String    0 instance 02238664 _text

Cool, now we can take a look at the specifics. For example the value of the text-property, which is located in the string down at the bottom with address 02238664. To get it's value, simply perform a !do on the address:

0:000> !do 02238664
Name: System.String
MethodTable: 790fa3e0
EEClass: 790fa340
Size: 158(0x9e) bytes
GC Generation: 2
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String:      </td>
    </tr>
  </table>
<!-- end of content table -->
 
Fields:
MT Field Offset Type VT Attr Value Name
790fed1c 4000096 4 System.Int32 0 instance 71 m_arrayLength
790fed1c 4000097 8 System.Int32 0 instance 70 m_stringLength
790fbefc 4000098 c System.Char 0 instance 3c m_firstChar
790fa3e0 4000099 10 System.String 0 shared static Empty
>> Domain:Value 000f0d00:790d6584 0011a720:790d6584 <<
79124670 400009a 14 System.Char[] 0 shared static WhitespaceChars
>> Domain:Value 000f0d00:01d413b8 0011a720:01d44f80 <<

Okay, so we can see that the string contains some data to close off a table. We can also take a look at the other properties for the object and examine them if we wish. But there is another command that is really useful...

Using !objsize

Is there a way to get the total size of a specific System.Web.UI.LiteralControl? - Simple! We use the !objsize -command. !objsize looks at all the pointers within an object and calculate their total size. Below is the built in documentation for the !objsize -command:

0:000> !help objsize
-------------------------------------------------------------------------------
!ObjSize [<Object address>]
 
With no parameters, !ObjSize lists the size of all objects found on managed
threads. It also enumerates all GCHandles in the process, and totals the size
of any objects pointed to by those handles. In calculating object size,
!ObjSize includes the size of all child objects in addition to the parent.
 
For example, !DumpObj lists a size of 20 bytes for this Customer object:
 
0:000> !do a79d40
Name: Customer
MethodTable: 009038ec
EEClass: 03ee1b84
Size: 20(0x14) bytes
(C:\pub\unittest.exe)
Fields:
MT         Field Offset  Type              Attr    Value  Name
009038ec 4000008      4 CLASS          instance 00a79ce4  name
009038ec 4000009      8 CLASS          instance 00a79d2c  bank
009038ec 400000a      c System.Boolean instance        1 valid
 
but !ObjSize lists 152 bytes:
0:000> !ObjSize a79d40
sizeof(00a79d40) = 152 ( 0x98) bytes (Customer)
 
This is because a Customer points to a Bank, has a name, and the Bank points to
an Address string. You can use !ObjSize to identify any particularly large
objects, such as a managed cache in a web server.

 

Objects in heap may also be smaller than they appear

So what do we get if we run !objsize on our LiteralControl? This is really interesting, because what happens is; the debugger gets really busy for quite some time and eventually we get this:

0:000> !objsize 023ea0a8
sizeof(023ea0a8) = 456918136 ( 0x1b3c0478) bytes (System.Web.UI.LiteralControl)

456 MBytes! How is that possible? Well if you scroll up to where we ran the !do command on the LiteralControl, you'll see that the control holds a reference to the page. The page in turn has a reference to the cache, and before long we'll have referenced almost the entire heap.

Summary

Hopefully this is enough to give you a quick glimpse of what is possible with three relatively simple commands from the sos extension. The commands were:

  1. !dumpheap
  2. !dumpobj
  3. !objsize

Over and out

/ Johan

Comments

  • Anonymous
    January 11, 2007
    The comment has been removed
  • Anonymous
    January 11, 2007
    Hi Heather, Thank you for the follow-up.
  1. Calling GC.Collect() is rarely a good idea. Rico Mariani has a good post on this at the following address: http://blogs.msdn.com/ricom/archive/2003/12/02/40780.aspx What I would do instead is make absolutely sure I call Image.Dispose() as soon as I'm finished with it. This is also recommended in the documentation. http://msdn2.microsoft.com/en-us/library/8th8381z(VS.80).aspx Unless you do so the image will be put in the finalizer queue in the next GC and moved up a generation which is most likely the reason why you're eventually getting OutOfMemoryExceptions. Calling dispose on an image is just as important as calling close on a database connection.
  2. To my knowledge there is no way to calculate the total size of the image. You'd have to do it manually through code instead. Width * height * 4bytes (32bits) should give you a fair estimate. / Johan

  • Anonymous
    January 15, 2007
    Thanks for your reply Johan! In response to #1, the problem was that I wasn't ready to release the images yet. I was actually getting the exception in the middle of a routine that was still loading them all up. I made this a little tighter as I realized some of the time I was loading the same image multiple times, so I wrote a wrapper around my image creation routine that checked to see if a particular image was already loaded in memory and just returned the same object. This reduced my memory footprint significantly and avoided running into the OutOfMemoryException in most cases, but I still had to leave the GC.Collect() code in to catch the rare cases where it didn't. sigh As for #2, I was hoping in general there was a way to track down the unmanaged resources of any .NET object, as that could be very useful in many situations other than just images. Oh well. :)

  • Anonymous
    January 15, 2007
    Hi Heather, Well if you're not done with the images, then I can only assume there's something else on the heap that needs to be dealt with. I'd get a hangdump right before loading the images and analyze what's on the heap at that time. Do you have any data connections, streams, etc. that need to be closed? / Johan

  • Anonymous
    January 22, 2007
    The comment has been removed

  • Anonymous
    June 16, 2007
    Hello, thanks for the article. I'm trying to pinpoint OOM problem and looking at LOH dump I can't understand small objects in the LOH: 0:028> !EEHeap -gc ... Large object heap starts at 0x04f21000 segment    begin allocated     size 04f20000 04f21000  05ee4238 0x00fc3238(16527928) 31d60000 31d61000  32489e00 0x00728e00(7507456) Total Size  0x9aec2a0(162448032) ... 0:028> !dumpheap 04f21000  05ee4238 Address       MT     Size 04f21000 03d25350       16 Free 04f21010 79124228     4096     04f22010 03d25350       16 Free 04f22020 79124228     4096     04f23020 03d25350       16 Free 04f23030 79124228      528     04f23240 03d25350       16 Free 04f23250 79124228     4096     04f24250 03d25350       16 Free 04f24260 79124228     6952     04f25d88 03d25350       16 Free 04f25d98 79124228     4096     04f26d98 03d25350       16 Free 04f26da8 79124228     4096     04f27da8 79124228     4096     04f28da8 03d25350       16 Free 04f28db8 79124228     4096     04f29db8 79124228      528     04f29fc8 03d25350       16 Free 04f29fd8 79124228      528     04f2a1e8 03d25350       16 Free ... :028> !do 04f29fd8 Name: System.Object[] ... Size: 528(0x210) bytes Array: Rank 1, Number of elements 128, Type CLASS Element Type: System.Object Those objects of 528 bytes are object[128] filled with strings (those are stored in Gen2 heap). I have no idea, what are those 16 Free bytes in the LOH between arrays. Could you please shed some light on this dumps? Or may be I'm doing something wrong?

  • Anonymous
    June 17, 2007
    Hi Ilya, This is very common for Framework 1.1. You'll normally see these small free segments after static arrays. So pay no further attention to them. / Johan

  • Anonymous
    June 18, 2007
    Johan, Thanks for your answer! Actually, it is .NET 2.0. Is it normal for this version too?

  • Anonymous
    December 21, 2007
    Johan, Excellent Article! I was looking for this information for almost four years can you help me in building strategy for toubleshooting OOM in web applications with 10-15 pages it would be great help

  • Anonymous
    December 28, 2007
    Also I was wondering why does the "!dumpheap -stat" doesn't show me the type of objects and their individual sizes when I use this command on the dump that I collected whereas in your case it shows each type and its associated size.

  • Anonymous
    January 01, 2008
    Please check your version of sos. Try loading it using ".loadby sos mscorwks" / Johan

  • Anonymous
    August 05, 2009
    Great post!! It's really good! I almost envy you, Johan!!

  • Anonymous
    September 28, 2012
    Thanks! Very useful post! 5 years have passed - but it's still actual for debugging in the production environment :)