Debug Diag script for troubleshooting .NET 2.0 Memory leaks
Article
I have put together a quick and dirty debug diag script for troubleshooting .net memory leaks. (attached to this post)
The reason I put it together was mainly to show how you can create your own debug diag scripts but feel free to use it to troubleshoot memory leaks, knowing that it does string parsing on the output so it is a bit prone to errors if sos changes output formats.
NOTE: This script will only work on .NET 2.0 x86 memory dumps
Mourad recently published a whitepaper on debug diag that talks about how to use it for various scenarios and he also explains how to create some basic scripts and I used this as my starting point.
My script loads up sos from the framework directory and goes through the following steps.
1. Print .NET Framework Version
2. Print information on the GC (Garbage Collector) heaps and report the amount of memory used for .NET objects
3. Print and categorize the 40 most memory consuming object types (so that you can spot which object types you might be leaking or using a lot of)
4. Print the finalizequeue so that you can see what objects on the heap have finalizers to verify that they are neccessary. It also reports potential blocking of the finalizer including the finalizer stack so that you can see if/why your finalizer thread is blocked.
5. Print objects on the large object heap
6. Determine and report the size of the cache for each asp.net application in the process
7. Report the number of active sessions in the process
8. Check if the process was in a GC when the dump was collected as this may invalidate heap/object information
2. Unzip the attached file and put the DotNetMemoryAnalysis.asp file in the C:\Program Files (x86)\DebugDiag\Scripts directory (or your debugdiag/scripts directory if you installed it somewhere else)
This will add a new analysis script to the Advanced Analysis tab in debug diag:
2. Generate a memory dump when memory usage is high. With debug diag you can do this by going into the processes tab, right-click on the process and select “Create Full Userdump”
4. In the Advanced Analysis tab, select Add Data Files, add your memory dump, select the DotNetMemoryAnalysis.asp script and click Start Analysis (note that some of the steps may take a while depending on the size of the dump). If you get an InStr exception that means that you are either trying to debug a 64 bit dump or the sos version in your framework directory is not matching up to the framework version of the dump (i.e. the proper mscordacwks symbol could not be found)
Once the analysis is done you will be presented with a mhtm file containing the report.
Sample report:
Here is a sample report generated for a dump taken for Lab 3 where the finalizer is blocked because of some bad code in the destructor for the Link object.
For this example we can see already from the analysis summary that something is fishy with the finalizer, and in the finalizer section we can also see the finalizer stack.
Scrolling down to the 40 most memory consuming object types we find a lot of Link/Link_aspx and System.Web.UI… objects, so in this case the report is pretty much straight on. In other cases it might be a bit less obvious but hopefully the suggested articles help narrow it down.
Analysis Summary
Type
Description
Recommendation
Warning
Number of objects ready for finalization: 35765
This is an indication that your finalizer thread may be blocked. Look at finalizequeue info and finalizer stack to determine why/if the finalizer is blocked
Information
Cache Size: 1397928 Bytes This includes memory for objects stored in in-process sessions
Number of GC Heaps: 2------------------------------Heap 0 (001aa5c0) generation 0 starts at 0x39d0c6a8generation 1 starts at 0x39005c0cgeneration 2 starts at 0x02d10038ephemeral segment allocation context: none segment begin allocated size02d10000 02d10038 06cf1098 0x03fe1060(66981984)16fd0000 16fd0038 1afca200 0x03ffa1c8(67084744)1efd0000 1efd0038 22fc61f0 0x03ff61b8(67068344)26fd0000 26fd0038 2afc1338 0x03ff1300(67048192)2efd0000 2efd0038 32e7c290 0x03eac258(65716824)36fd0000 36fd0038 3a0e7734 0x031176fc(51476220)Large object heap starts at 0x0ad10038 segment begin allocated size0ad10000 0ad10038 0ad23b78 0x00013b40(80704)Heap Size 0x16f99b74(385457012) ------------------------------Heap 1 (001ab888) generation 0 starts at 0x3de2921cgeneration 1 starts at 0x3d17acc8generation 2 starts at 0x06d10038ephemeral segment allocation context: none segment begin allocated size06d10000 06d10038 0acfe46c 0x03fee434(67036212)1afd0000 1afd0038 1efaba04 0x03fdb9cc(66959820)22fd0000 22fd0038 26fa3d7c 0x03fd3d44(66927940)2afd0000 2afd0038 2efa58d8 0x03fd58a0(66934944)32fd0000 32fd0038 36c42aa8 0x03c72a70(63384176)3afd0000 3afd0038 3e0e39b4 0x0311397c(51460476)Large object heap starts at 0x0cd10038 segment begin allocated size0cd10000 0cd10038 0cd10048 0x00000010(16)Heap Size 0x16cf97e0(382703584) ------------------------------GC Heap Size 0x2dc93354(768160596)
More information: Compare the total GC Heap size to the number of private bytes in the process when the dump was taken (or the dump size on disk) to determine if most of your memory is on the .NET GC Heap In the !eeheap -gc output above you can also see if most of your .NET GC memory is on the small object heaps or on the large object heap (LOH) (objects over 85000 bytes). If you see that most of your memory is on the LOH, look at theLOH outputto see what those objects are. If the GC Heap is relatively small, run debug diag with leaktracking to track native leaks and analyze the dump with the MemoryAnalysis script instead. See debug diag help for more information about this. Related posts:I have a memory leak!!! What do i do? (defining the "where")Are you really leaking .NET Memory?
SyncBlocks to be cleaned up: 0MTA Interfaces to be released: 0STA Interfaces to be released: 0----------------------------------------------------------------Heap 0generation 0 has 370 finalizable objects (107a9a9c->107aa064)generation 1 has 12 finalizable objects (107a9a6c->107a9a9c)generation 2 has 47 finalizable objects (107a99b0->107a9a6c)Ready for finalization 17909 objects (107aa064->107bb838) ------------------------------Heap 1generation 0 has 247 finalizable objects (107bcb40->107bcf1c)generation 1 has 4 finalizable objects (107bcb30->107bcb40)generation 2 has 36 finalizable objects (107bcaa0->107bcb30)Ready for finalization 17856 objects (107bcf1c->107ce61c) Statistics: MT Count TotalSize Class Name6614bbc0 1 12 System.Web.Configuration.ImpersonateTokenRef79334808 1 20 Microsoft.Win32.SafeHandles.SafeFileMappingHandle793347b0 1 20 Microsoft.Win32.SafeHandles.SafeViewOfFileHandle79317fac 1 20 Microsoft.Win32.SafeHandles.SafeTokenHandle66147c6c 1 20 System.Web.PerfInstanceDataHandle6614b038 1 32 System.Web.Compilation.CompilationMutex7932335c 2 40 System.Security.Cryptography.SafeProvHandle6612f6d0 2 56 System.Web.Security.FileSecurityDescriptorWrapper79317928 3 60 Microsoft.Win32.SafeHandles.SafeWaitHandle79321cd0 2 120 System.Runtime.Remoting.Contexts.Context7932a0f4 3 132 System.Threading.ReaderWriterLock79316fb0 7 168 System.Threading.TimerBase7932b108 14 280 Microsoft.Win32.SafeHandles.SafeRegistryHandle661483d8 10 280 System.Web.DirMonCompletion7932a09c 31 496 System.WeakReference66151384 70 1680 System.Web.HttpResponseUnmanagedBufferElement66148130 96 1920 System.Web.ApplicationImpersonationContext66151404 100 2000 System.Web.ClientImpersonationContext66103cb4 68 2992 System.Web.UI.WebControls.TableStyle79330ec0 78 4368 System.Threading.Thread0fbf1edc 35989 575824 LinkTotal 36481 objects
More Information:!finalizequeue will show all the objects on the heap that have finalizer methods, and have yet not been disposed of. It is a good idea to look through this list and verify that all your "custom" objects on this list really need finalizers/destructors, as having unneccesary finalizers will lead to higher memory consumption and a potential for blocked finalizers. Related posts:Unblock my finalizerTo dispose or not to dispose, that's the 1 GB question
As the number of finalizable objects is more than 0, please check the finalizer thread to see if it is blocked or active Finalizer Thread
More information: A high amount of large objects (strings and arrays over 85000 bytes) can lead to GC Heap fragmentation and thus higher memory usage in your application. Look through the large objects, to dig deeper you can run !do on the object address in windbg, to see if these objects are expected and if you can minimize their usage in any way, by caching etc. Common reasons for high amounts of large objects arelarge viewstateandDataset serialization
More information: There is one System.Web.Caching.Cache object referencing all cached objects, per web application In-Proc session state is stored in the cache, so the size of all session vars is also included in the size of the cache for the specific application Related articlesHow much are you cachingUI objects in session scope
Anonymous
May 17, 2009
Web The Evolution of a Website Design Twitter from ASP.NET IIS 7 Tip # 10 You can generate machine keys
Anonymous
May 17, 2009
WebTheEvolutionofaWebsiteDesignTwitterfromASP.NETIIS7Tip#10Youcangenera...
Anonymous
May 22, 2009
When the .NET Framework was first released, many developers believed the introduction of the garbage
Anonymous
May 26, 2009
Tess,
Good Morning ...
Really Cool Stuff
Couple of suggestion:
Can you show the process size
Showing whether Debug is set to true
Lastly showing the no. in MB instead of bytes (bytes looks scary at first look)
i.e
Cache Size: 1397928 Bytes
Cache Size: 1.3 MB
Thanks
Jas
Anonymous
May 26, 2009
When the .NET Framework was first released, many developers believed the introduction of the garbage
Anonymous
May 26, 2009
When the .NET Framework was first released, many developers believed the introduction of the garbage
Anonymous
May 27, 2009
Hi Tess,
This is really great and very usefull.
I second jaskis thoughts on shosing few more details(Debug Status, Process Size and if possible showing the size in MB's)
Thanks,
Sojesh
Anonymous
May 27, 2009
Tess you are already my hero, but I would like to third the suggestion to add Debug flag information to the script. JITOptimizerDisabled; IsJITTrackingEnabled;web.config and machine.config text checks <3
Anonymous
June 17, 2009
The comment has been removed
Anonymous
June 17, 2009
Nitin,
It's hard to say if a given number is "ok" or not, it completely depends on what you do in in the app.
If you are looking to find out if you have a native memory leak, you should really look at if the native part is growing or not (processprivate bytes - .net clr memory#Bytes in all heaps in perfmon).
In your case here you have the same difference whether your process is 712 MB or 1.27 GB and you can clearly see that the increase in memory = increase in .net memory, so here i would definitely say, just concentrate on the .net stuff.
Anonymous
June 24, 2009
The comment has been removed
Anonymous
June 24, 2009
not hard at all once debug diag x64 goes public.
It's just a matter of tweaking a few things with the formatting of the output
Anonymous
August 31, 2009
Thanks Tess! This script looks really helpful. I'll give it a spin on some memory dumps soon. I've been using the built in tool for a couple years but this looks like it offers more details and insight.
Anonymous
September 07, 2009
Hi Tess.
This script worked great, but I am still puzzled about my memory problem. The Virtual memory size of my appPool is 1.3gb and the dump made from debugdiag is 840mb. Still your script reports:
GC Heap Usage:5422712
Heap 0 size: 1147572
Heap 1 size: 961668
Heap 2 size: 927148
Heap 3 size: 822328
Heap 4 size: 320712
Heap 5 size: 210356
Heap 6 size: 299548
Heap 7 size: 733380
Finalizer queue:
SyncBlocks to be cleaned up: 0
MTA Interfaces to be released: 0
STA Interfaces to be released: 0
and there are no objects on the LOH.
Do you have any idea what can cause this? I occationally get OutOfMemoryException on this appPool, so I do have a problem somewhere :)
Anonymous
September 07, 2009
If this is all you have in terms of managed memory your memory usage is either mostly native or in dynamic assemblies.
Try running debug diag with leaktracking and run it through the normal analysis. (you can even run it through the normal memory analysis without leaktracking just to get an idea of where your memory is going)
Anonymous
December 14, 2009
Hi Tess,
Now that the x64 version of debugdiag is released, have you created an x64 version of this?
Thanks!
Anonymous
December 14, 2009
I haven't, but I may do so later. If someone else wants to do it, please feel free to post a link to the 64 bit version here. Should be very similar to the 32-bit version with small changes for parsing 64bit values. Most of it should actually work as is.
Anonymous
March 12, 2010
The comment has been removed
Anonymous
March 27, 2010
Hi Tess,
Thanks so much for your blog, I've learned so much about debugging from your labs and videos. It's great how you start from scratch and walkthrough various scenarios. Regarding this debugdiag script, I'm trying to run it on my dump but it seems to be stuck on "Determining the size of cache, this might take awhile" for at least 20min so far. Is this normal?
Anonymous
March 28, 2010
D,
Depending on the size of the dump, yes, it might take a long while since it has to enumerate everything referenced from the cache...
Anonymous
March 29, 2010
I left it running over night and it never finished. I'll try it again I guess.
Anonymous
February 07, 2011
Hi Tess ,
Is there any script will work on .NET 1.0/1.1 x86 memory dumps?
Thanks
MS
