Crashing Scenario

A crashing scenario is when the application terminates unexpectedly.  This can happen a variety of ways, but usually a  result of an unhandled exception.  Most often if the error reporting system of Windows has not been changed, an error dialog similar to "Send Error Report to Microsoft" will be displayed and there will be an application event log telling the module that failed, but actual cause of the crash is usually in a different module.  I have seen a few times application just disappearing, no dialog, or anything mostly in managed code.  This is usually a result of a manged heap corruption causing the crash.  If the app is handling the exception and gracefully closing the application out or logging it, this is not a crash.  The tools listed below will help in taking crash dumps for postmortem debugging.

Hanging Scenario

A hang is when the application is not responsive due to a dead hang, 0% CPU usage, or a busy hang, 100% CPU usage, where it needs to be killed.  The dead hang can happen if the application is blocked from execution due resource contention, blocked or orphaned critical sections, or an indefinite wait for a resource or a piece of data to be processed.  The busy hang mostly from unintended infinite loops/recursions.  The question to ask here is how long the hang happens, if it's an indefinite hang there is an obvious problem, but usually the hang can happen for 30 seconds, 5 -10 minutes, or any random amount of time.  With a hang scenario, a few hang dumps needs to be captured in succession to get a good picture of what's going on.  Depending on how long the application hangs, divide that time up by 3,4,5 and at each interval of the process take a hang dump using adplus -hang or use debugdiag and it will give a good snapshot of the process over time.  This can help determine which thread is hung when doing postmortem debugging. 

Other special scenario, to be covered later.

Tools to tackle the little bugger

If the problem is happening during development then I am sure most will try and debug it as normal and resolve the problem.  The following information might help those seeking help when stuck.  Say the application is crashing on a production environment or on a customer's customer site and there is limited control/access to collect the right debug information to resolve the issue or the issue happens once in a blue moon.

The first thing that all developers need to be aware of when building their application is to always have symbol files, *.PDB, generated.  These files should be saved for each build so that it can be used for debugging in the future if there are problems.  A symbol store can be setup or it can be saved anywhere and pulled in later using a debugging tool like Visual Studio or windbg.  By default VS 2002 and later build symbols in release unlike VS6.0 which was not the default.  Symbols for native/mix-mode modules are required and managed module is not really needed when debugging.  It's always painful asking a customer to rebuild the application with symbols, re-deploy, and repro the issue again as not having symbols makes it very hard to debug the issue.  Old symbols used for newer builds will not work.  There's always the option to force load the symbol, but the call stack maybe incorrect and unreliable, so always make sure the symbols are from the same build as the application having the problem.

Once the symbol issue is resolved, now to start collecting the data to debug the problem with some tools used to live debug or collect memory dumps for postmortem debugging.

DebugDiag.exe:  This tool is mainly targeting IIS process, but can be used for all processes.  This tool is very user friendly and flexible.  Rules can be setup to monitor a process and when it crashes a memory dump can be generated or manual dumps can be taken of the process while it's hung.  It has added features to inject leaktracker.dll to monitor for memory leaks.  The best feature is Advanced Analysis.  This gives a pretty good overview of what is going on with the process or what is causing the problem, many times it will not point exactly to the cause of the problem.  Can be used for both managed and unmanaged process, needs to be installed on the client machine.

adplus.vbs:  This is a script to attach the CDB debugger to the process to monitor it.  This is a robust tool, but complex as config files need to be created to handle custom events.  This tool comes with the installation Debugging Tools For Windows and can be used for simple crash scenarios to to take quick hang dumps.  The documentation for the tool after install is very helpful, this works for both managed and unmanaged processes and does not need an install.  Can xcopy the Debugging Tools for Windows directory to client, no install:     adplus -crash -pn <ProcessName.EXE> -quiet or adplus -hang -pn <ProcessName.EXE>  -quiet

windbg.exe:  Many articles on using this debugger.  It's pretty powerful and can be used as a live debugger or a postmortem debugger.  Works for both managed and native processes and can be xcopied over without having to install/register any components on client machine.

drwtsn32.exe:  Comes standard on all Windows OS, maybe embedded Windows is optional.  So no install is needed.  Only constraint is that it cannot collect dumps for managed processes.  Use this tool in most restrictive environment, tool defaults to taking minidumps, need to configure it to take full dumps.

Visual Studio:  Can be used to live debug and do postmortem debugging with some limitation on postmortem managed memory dumps.  Not the best tool to use in a limiting environment.

If stuck, dumps and symbols are good data to have ready to pass onto support to get the help needed.  I listed some links below those willing to read on.  I'll continue this discussion on a later post talking more about debugdiag, adplus, windbg and how to use it to collect information to resolve the issue.  There are lots of articles out there already on how to resolve, but collecting the correct information is the most important.

Root Out Elusive Production Bugs with These Effective Techniques

Managed debugging extensions

Production Debugging for .NET Framework Applications

Windows Debuggers: Part 1: A WinDbg Tutorial (There are 7 parts)

1.  Keep all assemblies in the local application directory, private assemblies

Basically the vendor would have to deploy 3rd party SDK DLL's with their app.  This would make it very difficult to service the DLL's once it's in the field as there could be countless private assemblies sitting on the same machine in different folders, remind anyone of CRT redist issues?  This would also put the burden on vendors to keep the private 3rd party DLL's up-to-date by continuously updating their application when the update needs to be done by the SDK vendor.

2.  GAC all your DLL's

This is a good alternative.  This is how the .Net framework works; each .Net assembly used in the CLR is GAC'ed so no one have to re-deploy the runtime DLL's once the client has the framework installed.  Servicing is easy, they can have publisher policies to redirect binding to newer versions of the DLL's, this would be the ideal situation, or the app vendor can use app.config files to redirect binding to newer versions.  But what about native dll dependencies?

GAC Phobia 
https://blogs.msdn.com/alanshi/archive/2004/03/28/100765.aspx

3.  What if you have native dependencies?

In this scenario, you can create a multifile assembly, meaning .Net netmodules.  This way your native assembly will be linked into the managed code so when you install the assembly into the GAC all the modules and the native DLL's will be copied into the GAC under it's own directory so minimizing probing issues.  Servicing of native DLL's would be easy as you can rebuild and redeploy the new .Net assembly and redirect with publisher policy or app.config files.  The other alternative is to wrap the native DLL's into it's own .Net assembly.  This way you can keep the managed assembly separate from the native code and servicing can be more modular.

There are a couple of issues with this, first the native DLL is platform specific.  Meaning when you build a native DLL it's either x86 or x64 and .Net builds with "AnyCPU" configuration as default which is processor agnostic.  So when you link in a native DLL, you will get this al.exe error AL1070, so building with /platform:x86 for all netmodules is required.  The other problem is DLL bloat.  This would load all the code modules into the process even if it's not being used.  So some perf hit here if perf is relevant.  Also when servicing this assembly a rebuild of the modules are required.  This is different from GAC'ing all DLL's above is because the probing path for native dependency will be the standard probing path of the application so the native DLL would have to sit in the app's directory or maybe system32 which is not a good location.  This method resolves the issue with servicing native DLL's that has .Net dependencies. 

csc.exe /target:module GACDLL.cs /addmodule:nongac.netmodule /platform:x86     :building the netmodule with x86 as targeted platform

al.exe /keyf:key.snk /link:nativedll.dll gacdll.netmodule /target:lib /out:test.dll /platform:x86     :linking in native dll targeting the whole dll as x86

When loading assemblies into Powershell, you should load the assemblies in the right binding context, use Load or LoadFrom and not LoadFile.  The reason for this .Net is a strong typed framework, so loading assemblies with different load methods will treat the same types in each assembly as different types, type mismatch.  Here's a good blog on this:

Choosing a Binding Context
https://blogs.msdn.com/suzcook/archive/2003/05/29/57143.aspx

This is the error you would get if you were load using LoadFile and trying to instantiate the .Net object with dependencies using new-object, no TypeLoadException or any obvious exceptions.

New-Object : Exception calling ".ctor" with "0" argument(s): "Could not load file or assembly "..., Version=x.x.x.x, Culture=neutral, PublicKey
Token=null' or one of its dependencies. The system cannot find the file specified."

To resolve this you can use the following with some caveats:

1.  [Reflection.Assembly]::LoadFrom

LoadFrom takes a path and will load assemblies in the same context as Load.  The advantage is that the assemblies can be located anywhere.  You just need to load the main assembly and it will load all dependent assemblies in that same path.  This is the best, except when you do serialization you will get an error concerning type-mismatch because LoadFrom is not entirely the same as Load, explained below.  So serialization assemblies show be GAC'ed or placed in local app's directory.

Run-time Serialization 
https://msdn.microsoft.com/msdnmag/issues/02/04/net/default.aspx

2.  [Reflection.Assembly]::Load

You only have to load the main assembly and all dependent assemblies will be loaded when needed.  The problem with this is that the DLL's needs to be in the application's local directory, in this case it's Powershell's:  C:\WINDOWS\system32\windowspowershell\v1.0.  This is actually messy, might use it temporarily since you will need to dump all dependent assemblies into the directory.

3.  GAC everything which is not the best thing or use a script to create a new AppDomain and load everything into it and you can also have a config file to set assemblies private path, I have not tried yet but seems possible with Powershell :)

Use fuslogvw.exe tool part of the SDK to troubleshoot binding and probing issues and this article is a good read as well.

How the Runtime Locates Assemblies
https://msdn2.microsoft.com/en-us/library/yx7xezcf(vs.71).aspx