Procdump: How to PROPERLY gather dump (.dmp) files for crashes and hangs, CPU spikes, etc, including gathering PERF data, for Exchange issues
Is your application crashing or hanging? Need to know why?
Well, prodcump may be able to help you. Procdump is the newest thing around! You might remember adplus (eww!), well now its all about the Procdump! Procdump allows us to "dump" the contents of the memory at the time (hopefully) that the "issue" occurs.
https://technet.microsoft.com/en-us/sysinternals/dd996900.aspx - download procdump from this link
Here are some basic procdump usage note:
-ma full memory dump, always do this on 2003 as 4gb is not much and it is good to have the heap
-mp miniplus, 2007 – 2010 grabs the essential linked heap memory
-n 3 Typical hang dump to get 3 dumps - We typically want 3 dumps, so we can see what changes, or what DOESN'T change, in memory (i.e., is the same thread waiting in all 3 dumps?). A single dump file is typcially not that useful.
-accepteula self explanatory
-e create a dump file only when an UNHANDLED exception occurs. We're typically not interested in HANDLED exceptions, because those types of exceptions get HANDLED and your program will continue to run.
And last but not least, we need a place to store the dump file
So, an example of a dump of a typical Exchange store.exe hang would be something like this:
Procdump -e –mp store.exe –n 3 –accepteula d:\dumps\store_issue.dmp
This will set up procdump to monitor store.exe and when/if an unhandled exception occurs, it will create 3 dump files and store them in the specified location.
Here's another fine example, courtesy of Edwin Huber. This was used to catch an MSFTEFD crash
You can use a combination of procdump v5 with the filter option and https://toolbox/getmulticrash to catch the culprit: (This is an MSFT internal only)
ProcDump –ma –e 1 –f “”
or
ProcDump –ma –e 1 –f *FailFastException
Another neat trick is to trigger the dump off of a performance counter. Remember, you MUST get the RIGHT data if you want to help your customer resolve their issue!
In this Example, we will catch 3 hang dumps dumps when version bucket usage is 2480:
Procdump –mp store.exe -p “\Database(Information store)\Version buckets allocated” 2480 –n 3 –accepteula d:\dumps\store_issue.dmp
Often times it can be beneficial to get perf data when gathering a procdump. This will allow the EE reviewing your dump (or you, if you're that awesome!) to get a better idea of WHERE in the dump to look. Usually you will want to gather threads in this type of perf. The easiest way to do this is to use ExPerfWiz: https://experfwiz.codeplex.com/
You will want to be sure to grab threads with the -threads switch, so your command might look something like this:
Here's a sample command:
.\experfwiz.ps1 -circular –maxsize 1024 –StoreExtendedOn –EseExtendedOn -threads -interval 30 -filepath D:\Logs
To STOP the perfwiz, run .\experwiz.ps1 -stop. The files will be of type .blg
Zip the files and upload them to the workspace provided to you.
NOTE: you might need to adjust which switches you use based on your particular issue!!
ExPerfWiz can be downloaded from https://experfwiz.codeplex.com
Once you have gathered your dump files, you will need to submit them to CTS for review. If possible, please make sure the dumps are on a share LOCAL to the office where the EE reviewing is located. So, if the EE is in Charlotte, NC, and you have a desk/cube, etc, in that location, pull the files down locally and share them out (unzipped).
For IIS issues, or anything that involves w3wp or application pool crashes, procdump may not be your best bet. Below are detailed posts on using debugdiag to generate dump files in such circumstances:
Many thanks to Mark Russinovich, Mike Lagase, Edwin Huber, Theo Browning, Konstantin Papadakis and Jamie Tsottles for assistance with this blog post. And thank you to Jeff Stokes for making the suggestion :)