Really an Invalid Image Format?

Or was it just meant for another architecture?

Diagnosis Tactics

Dependency Walker

Process Monitor

WinDbg with Loader Snaps

Still Didn't Figure it Out?

Validating Certificates

• Verifies the presence of the Basic Constraints extension and its value, which must be either Subject Type=End Entity or unspecified.
• Verifies the value of the Enhanced Key Usage property, which must contain Code Signing and may also contain Lifetime Signing. Any other EKUs are prohibited.
• Verifies the value of the KeyUsage (KU) property, which must be either Unset or DigitalSignature.
• Verifies the existence of a private key exists.
• Verifies whether the certificate is active, hasn’t expired, and hasn't been revoked

At that point, we just enabled first-chance exception dumping – ‘procdump -n 100 -ma -e 1 Test.exe’. This generated many, many dump files. Did I mention this was a mixed-mode managed app that contains both managed and native code? Managed code can potentially throw a lot of first-chance exceptions – even a lot of AV first chance exceptions as the managed NullReferenceException starts out a first-chance AV.

After looking at a lot of dump files I was able to notice a pattern of native code FCE.

The Windbg callstack looks like:

This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(c3c.c88): Access violation - code c0000005 (first/second chance not available)
eax=0bd3f000 ebx=7ffffff6 ecx=00000073 edx=00000007 esi=01282138 edi=0bd3eff0
eip=6ad71891 esp=0019eae4 ebp=0019ef68 iopl=0         nv up ei pl nz na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010206
msvcr100!_woutput_l+0x93b:
6ad71891 66833800        cmp     word ptr [eax],0         ds:0023:0bd3f000=????

0:000> k
ChildEBP RetAddr 
0019ef68 6ad71af1 msvcr100!_woutput_l+0x93b [output.c @ 1625]
0019efac 6ad71b38 msvcr100!_vsnwprintf_l+0x8e [vswprint.c @ 155]
0019efc8 01281043 msvcr100!_vsnwprintf+0x18 [vswprint.c @ 227]
0019f000 012810d0 Test!EvilFunc+0x43 [c:\test.cpp @ 19]
0019f834 0128125a Test!main+0x60 [c:\test.cpp @ 38]
0019f878 7726ee6c Test!__tmainCRTStartup+0x122 [crtexe.c @ 555]

This is my after the fact repro code of course, developers are never kind enough to identify bad code as EvilFunc. This little sample shows the two bugs they had interacting to produce this scenario:

#define _CRT_SECURE_NO_WARNINGS
#include <stdio.h>
#include <stdarg.h>
#include <tchar.h>
#include <string.h>

// Build /Eha to catch SEH Exceptions
void EvilFunc(wchar_t * pWChar, ...)
{
    wchar_t sz[1024] = { 0 };
    int nBufLen = (sizeof(sz) / sizeof(sz[0])) - 1;
    va_list args;

    va_start(args, pWChar);
    int nBuf = 0;

    try
    {
        nBuf = _vsnwprintf(sz, nBufLen, pWChar, args);
        // Do something with sz...
    }
    catch(...)
    {
    }

    va_end(args);
}

int main()
{
    long count = 1;

    while(true)
    {
        char* szUserName = _strdup("Steve Horne");
        EvilFunc(_T("The UserName is : %s"), szUserName));
        printf("loop count %d\n", count++);

        free(szUserName);
    }

    return 0;
}

EvilFunc is using the ‘…’ vararg syntax in C/C++. The first parameter they were passing was a wide-char – they didn’t explicitly declare it wchar_t*, they were using one of the OLE macros which compiled to wchar_t* – I’ve simplified things down to the minimum repro here.

The first bug is passing in a char* as the … variable params. The compiler doesn’t type check the … so it was happily calling  _vsnwprintf with a wide-char format string and char string, just like the developer told it to.

The page-heap enabled crash dump shows the problem with that. They walked off the end of the char* string in CRT code (it was looking for a wchar_t two-byte NULL string termination) and started accessing  memory unintentionally:

0:000> r
eax=0bd3f000 ebx=7ffffff6 ecx=00000073 edx=00000007 esi=01282138 edi=0bd3eff0
eip=6ad71891 esp=0019eae4 ebp=0019ef68 iopl=0         nv up ei pl nz na pe nc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010206
msvcr100!_woutput_l+0x93b:
6ad71891 66833800        cmp     word ptr [eax],0         ds:0023:0bd3f000=????

0:000> db @eax
0bd3f000  ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ??  ????????????????
0bd3f010  ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ??  ????????????????
0bd3f020  ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ??  ????????????????

0:000> db @eax-10
0bd3eff0  53 74 65 76 65 20 48 6f-72 6e 65 00 d0 d0 d0 d0  Steve Horne.....
0bd3f000  ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ??  ????????????????
0bd3f010  ?? ?? ?? ?? ?? ?? ?? ??-?? ?? ?? ?? ?? ?? ?? ??  ????????????????

This was corrupting the heap after the _strdup allocation – unless you turn on the wonderful page-heap which book-ends the allocation with non-writable memory so the overwrite faults.

The second bug, or issue is the catch(…) filter used with the /Eha compiler option. This allows the catch(…) to catch all SEH exceptions, completely hiding the AV from procdump or any other tool trying to trap unhandled exceptions. I’m not positive why this was done, but my suspicion is that they couldn’t figure out why this code kept crashing and just added the catch for SEH exceptions to keep the process up. Not a good idea. It’s far, far, far better to have you app crash and fix that crash than to mask a heap corrupting bug.

Heap corruption is an insidious evil. When you finally do get a crash or hang due to a heap corruption, it’s impossible to look back in time and find the cause of the heap corruption – even if you are able to figure out that your issue is actually caused by heap corruption.  Whatever code corrupted the heap is long past executed and not in any current stack frames. Page-heap can easily help you track down these types of buffer over-run issues – but not if you hide the exceptions from the debugging tools!

This repro requires an older CRT –  VS 2010 was the version being used here. Later versions of the CRT do not reproduce this buffer overwrite due to changes in the CRT implementation.

I recently helped a developer that needed to enumerate all of the modules in an arbitrary process. They are given a Process ID (PID) and need to enumerate all of the modules, both native and managed that have been loaded by the process.

They started by creating a System.Diagnostics.Process object for the given PID with Process.GetProcessById. With the managed process object they were then able to get a collection of loaded modules with the Modules property. All well and good but they discovered that not all managed assemblies that are loaded in the process are included in the collection returned by the Modules property.

This is actually expected behavior for framework versions 4 and above. The .NET Framework 4 Migration Issues article in the section 'Assembly loading' describes this. It was an optimization created such that LoadLibrary is not used for managed assembly loading -- The MapViewOfFile Win32 api is used instead. An effect of this is that the windows loader does not see these module loads and Process.Modules, or native api that enumerate loaded modules via the Process Environment Block (PEB) are not aware of the assemblies.

We had looked at various resolutions to this problem -- attaching a managed debugger was considered. This would allow the enumeration of AppDomains and their assemblies which would give a list of all managed assemblies. This was rejected primarily because it would prevent debugging of the application after the managed debugger was attached.

The developer had also started investigating some undocumented ways of finding all file handles in the process and matching those with file names. SysInternals seems to use this method for some of their tools if you are interested in searching for that resolution. CSS cannot recommend or investigate solutions that use undocumented api -- these are subject to change at any time so it's a dangerous path.

A managed profiler would also be a way to capture managed module loads -- but that would require the profiler to be loaded in every managed process which was not a viable solution for this developer either.

Lee Culver suggested his excellent Microsoft Diagnostics Runtime (ClrMD) package as a way to accomplish this goal.

ClrMD is an open source project on GitHub:

     https://github.com/Microsoft/clrmd

It's also available as a NuGet package:

    www.nuget.org/packages/Microsoft.Diagnostics.Runtime/0.8.31-beta

This is still not officially released so when you search for it in the Visual Studio package manager, make sure you select the 'Include prerelease' checkbox so it can be found.

Here is a simple console app which uses ClrMD to enumerate the assemblies in an instance of Visual Studio. Add this code to a Wizard generated C# console app that has the ClrMD NuGet package added to the solution (Project menu, Manage NuGet Packages...).

using System;
using System.Linq;
using System.Diagnostics;
using Microsoft.Diagnostics.Runtime;

/*
 This requires the Microsoft.Diagnostics.Runtime NuGet package (which 
 is marked pre-release, so it won't show up unless you check that box
 in NuGet).
*/

class Program
{
    static void Main(string[] args)
    {
        string processName = "devenv";
        int pid = Process.GetProcessesByName(processName).First().Id;

        /*
         The key here is to use the Passive attach flag.This will "attach" to
         the process by using OpenProcess / ReadProcessMemory but it does not 
         attach a debugger to it or pause the process in any way.
        */
        DataTarget dt;
        using (dt = DataTarget.AttachToProcess(pid, 5000, AttachFlag.Passive))
        {
            /*
             To see native module list, use dt.EnumerateModules.However, this
             is NOT what the customer was looking for, since it has the same 
             problem they report. Instead we will request the managed module 
             list from the runtime object.
            */

            // First, loop through each Clr in the process (there may be 
            // multiple in the side-by-side scenario).
            foreach (ClrInfo clrVersion in dt.ClrVersions)
            {
                ClrRuntime runtime = clrVersion.CreateRuntime();
                foreach (ClrModule module in runtime.Modules)
                {
                    if (module.IsFile)
                        Console.WriteLine(module.FileName);
                }

                /*
                 Note that ClrMD builds state in caches with every API call.
                 Since the process is live and constantly changing this means 
                 subsequent calls to ClrRuntime.Modules will get the same  
                 data back over and over. Instead you need to call 
                 ‘ClrRuntime.Flush’ if your reuse the runtime object to check 
                 the module list repeatedly:
                */

                /*
                 Uncomment this if you ever loop through ClrRuntime.Modules
                 again, expecting updated results.
                */
                // runtime.Flush();
            }
        }
    }
}

 While work on an issue; I came across a requirement where customer is migrating his project from VS2012 IDE to VS2015 IDE. Currently they are migrating only IDE of the project where they are still using the old compiler. There was a reason behind this. Let's don't go into that part.This brings a problem. Whenever a developer opens a the solution file, it throws a  dialog warning "Update VC++ Compiler and Libraries" about the old compiler settings and ask you to upgrade. Screen Shot below. Looks familiar?

Its fine till the time you are choosing the right option of not upgrading to newer compiler but there are many developers working on the same project, there are chances that a developer migrate the compiler settings and checking in the source safe as well. This cause a bigger problem when all the developers will do get latest and get a copy where compiler setting is also updated and developers start getting different errors.

 Now the solution part. To make sure that developers doesn’t get the warning every time open the solution, make the following changes in the project file.

  <PropertyGroup Label="Globals">

   
<ProjectGuid>{75254B93-8568-4DE7-81F0-A67FBD62FDA7}</ProjectGuid>

   
<Keyword>Win32Proj</Keyword>

   
<RootNamespace>UserGPOInfo</RootNamespace>

       <VCProjectUpgraderObjectName>NoUpgrade</VCProjectUpgraderObjectName>

 
</PropertyGroup>

Note: Please remove the above property once you are ready to migrate to new compiler.

Alternative approach ( Not recommended)

There is no project bases setting saved in the registry key and shouldn’t be. We need to make sure that any settings or change should be on solution/project bases. Having said that, I noticed that warning only comes when you first open the project in VS2015. Once you press OK or cancel, it saves it and don’t prompt warning again when you reopen the
project in VS2015. Internally, it creates a .VS folder which is hidden. This folder does have a project corresponding .suo file which contains this information. To avoid repetitive work of cancelling the warning, you can save the .VS folder for solution/project. It will be a one time job and keeping this folder in the solution file folder and save it to your source safe, will not throw you compiler warning again.

Hope the above post will be helpful.

Written and approved by: Irfan Ahmed, Senior Support Escalation Engineer

Exception: System.ComponentModel.Win32Exception
Message: Error creating window handle. 

It was pretty evident from the error message that we are unable to create handles and one other aspect of the issue was that we really had high no of objects in the Task manager. An important point to note here is that we do not see a crash in the
application. To make sure we have the right data, I had to capture a first chance exception dump on Win32Exception. For more information on First chance exception please read the Blog. There are loads of tools you could use to capture this dump, DebugDiag is an user friendly UI tool which helps you fiddle around with exception names. Considering the issue to be intermittent Debugdiag is an ideal tool in this scenario as all we have to do is to set up a rule and DebugDiag will take  care of the rest. While the DebugDiag UI is pretty straight forward , Once you open DebugDiag , Click on Add Rule->Crash->Click Next-> A Specific Process(This feature to be used for exes)->Now Select the process (from the process list or write the Application name if it is not running)-> Click Next . On this windows I would add specific actions considering I want a first chance dump for a particular exception. Now click on Exceptions->Add Exception. Further Select E0434352 and add the corresponding details as shown below.

After all the above steps , it is important to RESTART YOUR APPLICATION. 

Well after the above steps we did receive the dump file and here is how we nailed down the issue. Excerpts from analysis below. I believe windbg is a powerful debugger and I prefer to use it, any other debugger can be used to debug the dumps. Visual Studio has many inbuilt feature for memory analysis , a very handy tool to be used in High memory scenarios.

0:000> !pe
Exception object: 024c96bc
Exception type: System.ComponentModel.Win32Exception
Message: Error creating window handle.
InnerException: <none>
StackTrace (generated):
 SP IP Function
 001DED08 6692CF75 System_Windows_Forms_ni!System.Windows.Forms.NativeWindow.CreateHandle(System.Windows.Forms.CreateParams)+0x2c5
 001DED80 6692CBA5 System_Windows_Forms_ni!System.Windows.Forms.Control.CreateHandle()+0x115
 001DEDDC 6692E9FC System_Windows_Forms_ni!System.Windows.Forms.Control.get_Handle()+0x4c
 001DEDEC 668DB724 System_Windows_Forms_ni!System.Windows.Forms.Control.CreateGraphicsInternal()+0x8
 001DEDF4 67090D38 System_Windows_Forms_ni!System.Windows.Forms.ThreadExceptionDialog..ctor(System.Exception)+0x800
 001DEEF8 66E8EA1E System_Windows_Forms_ni!System.Windows.Forms.Application+ThreadContext.OnThreadException(System.Exception)+0xb6
 001DEF34 66E7A782 System_Windows_Forms_ni!System.Windows.Forms.Control.WndProcException(System.Exception)+0x16
 001DEF40 66E7BA6A System_Windows_Forms_ni!System.Windows.Forms.Control+ControlNativeWindow.OnThreadException(System.Exception)+0xa
 001DEF44 6692E18B System_Windows_Forms_ni!System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)+0x9b
 00000000 00000001 System_Windows_Forms_ni!System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG ByRef)+0x2
 001DF1E0 6693DCA1 System_Windows_Forms_ni!System.Windows.Forms.Application+ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(IntPtr, Int32, Int32)+0x24d
 001DF26C 6693D921 System_Windows_Forms_ni!System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(Int32, System.Windows.Forms.ApplicationContext)+0x155
 001DF2BC 6693D792 System_Windows_Forms_ni!System.Windows.Forms.Application+ThreadContext.RunMessageLoop(Int32, System.Windows.Forms.ApplicationContext)+0x4a
 001DF2E8 668E14BD System_Windows_Forms_ni!System.Windows.Forms.Application.Run(System.Windows.Forms.Form)+0x31
 001DF2FC 00414203 Example1!Example1.Program.Main(System.String[])+0x143
 
The above exception is triggered, we have to find our which user objects are being used that lead to this. 
Although the nature of the objects occupying memory may vary , we have to understand that we should 
only be worried about UI objects precisely which would make the application throw such an error, namely 
Dialogs, Windows, timers, Ui controls etc. We will use a command !address -summary to find out which 
objects occupy what amount of memory.

MT Count Size Class Name
66974944 619 9904 System.Windows.Forms.PropertyStore+SizeWrapper
68097098 226 9944 System.Reflection.Emit.OpCode
66975558 242 10344 System.Windows.Forms.PropertyStore+IntegerEntry[]
680ccfc8 6 10952 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[System.String, mscorlib]][]
6ab68ea0 463 20372 System.Drawing.Font
004298e4 588 21168 QSYS.MSG.MessageText
680d8b40 898 21552 System.Collections.ArrayList
67589644 229 28396 System.ComponentModel.ReflectPropertyDescriptor
6808ef70 521 29176 System.Reflection.RuntimePropertyInfo
6758749c 1525 30500 System.ComponentModel.EventHandlerList+ListEntry
669749f8 591 30732 System.Windows.Forms.CreateParams
680dd288 598 31096 System.Collections.Hashtable
680db970 82 32140 System.Char[]
6a975f6c 584 32704 System.Configuration.ConfigurationProperty
669743e0 592 33152 System.Windows.Forms.Control+ControlNativeWindow
08e31cc4 355 39760 com.ibsag.qsys.svr.system.XDQSFormItem
680dc738 335 42496 System.Int32[]
6697497c 602 51124 System.Windows.Forms.PropertyStore+ObjectEntry[]
08e32b80 5 56100 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[QSYS.WindowsControls.BL.LayoutTranslationEngine+LayoutItem, QSYS_WindowsControls]][]
680cee64 838 56984 System.Reflection.RuntimeParameterInfo
680d7618 1338 58872 System.Signature
00b02918 348 64032 QSYS.WindowsControls.IBSControls.Button
680d8ce0 5899 70788 System.WeakReference
680db350 6080 72960 System.Object
680dae80 599 76224 System.Collections.Hashtable+bucket[]
680d7940 1529 91740 System.Reflection.RuntimeMethodInfo
6697476c 1 101040 System.Windows.Forms.NativeWindow+HandleBucket[]
680dc168 5157 144396 System.RuntimeType
66975130 5110 163520 System.Windows.Forms.NativeMethods+WndProc
680d08a0 5535 177120 System.EventHandler
66961308  4976 238848 System.Windows.Forms.Timer
6696306c 4970 258440 System.Windows.Forms.Timer+TimerNativeWindow
6808ae88 5854 369200 System.Object[]
004bbc98 15 565718 Free
680dafb0 14211 911970 System.String
680dd244 501 958102 System.Byte[]
6808e948 94719 3031008 System.Globalization.DaylightTime
Total 192318 objects

Based on the above summary it is very clear that for now we will have to ignore System.String objects and 
concentrate on these UI objects. Having known the highlighted objects occupy a fair amount of memory 
we can look into our code to see where we use these objects and see if we are leaking memory somewhere. 
We can further debug in the memory dumps to find out the exact roots of these objects and track them
accordingly, So I set out to find out the roots for alacrity to our customer as to where exactly does he have
to look into. I would dump the objects using !dumpheap -mt Methodtable to dump the below info.

02423764 66961308  48 
02423a54 66961308 48 

.

.
02425dac 66961308 48 

02426144 66961308 48 
024261e0 66961308 48 
0242627c 66961308 48 
.
.
02426f44 66961308 48 
02426fe0 66961308 48 
0242707c 66961308 48 
02427118 66961308 48 
02428938 66961308 48 
.

.

0242baa0 66961308 48 
0242bb3c 66961308 48 
0242bbf8 66961308 48 
0242bc94 66961308 48 
0242bd50 66961308 48 
0242c118 66961308 48 
0242c224 66961308 48 
0242c2c0 66961308 48 
0242c35c 66961308 48 
0242c3f8 66961308 48 
0242c4b4 66961308 48 
0242c550 66961308 48 
0242c764 66961308 48 
0242c82c 66961308 48 
0242c9c4 66961308 48 
0242cab4 66961308 48 
02434f28 66961308 48 
02435010 66961308 48 
 
Statistics:
 MT Count TotalSize Class Name
66961308 4976 238848 System.Windows.Forms.Timer

0:000> !gcroot -nostacks 02435010
HandleTable:
 001f1130 (strong handle)
 -> 02353720 System.Windows.Forms.Timer
 -> 0235375c System.EventHandler
 -> 0205d524 SPCInsp.MainFormSPCInsp
 -> 0217de18 SPCInsp.LoginCenter
 -> 02434fe4 SPCInsp.AutoPnrHandler
 -> 02435010 System.Windows.Forms.Timer

You can observe that we have found the roots at 02435010. I can confirm that most of the other roots were very 
synonymous. Further we should look at the other type of objects from what we had yellow marked from 
!address -Summary, let’s look at the other object in a similar way. 

024388b4 66975130 32 
.
.
02448874 66975130 32 
02448a3c 66975130 32 
02448a5c 66975130 32 
02448a7c 66975130 32 
02448ad8 66975130 32 
02448b34 66975130 32 
02448b90 66975130 32 
02448be4 66975130 32 
02448c40 66975130 32 
02448c60 66975130 32 
.
.
024c6500 66975130 32 
024c6520 66975130 32 
024c6540 66975130 32 
024c6560 66975130 32 
024c6580 66975130 32 
024c65dc 66975130 32 
024c6610 66975130 32 
024c6630 66975130 32 
024c6650 66975130 32 
024c66ac 66975130 32 
024c66cc 66975130 32 
024c66ec 66975130 32 
024c670c 66975130 32 
024c672c 66975130 32 
024c674c 66975130 32 
024c67a8 66975130 32 
024c70f0 66975130 32 
024c7110 66975130 32 
024c7130 66975130 32 
024c7150 66975130 32 
024c7170 66975130 32 
024c7288 66975130 32 
024c72a8 66975130 32 
024c72c8 66975130 32 
024c72e8 66975130 32 
024c7308 66975130 32 
024c7328 66975130 32 
024c7348 66975130 32 
024c7368 66975130 32 

Randomly dumping the objects with roots,

0:000> !gcroot -nostacks 024c7368 
HandleTable:
 001f1130 (strong handle)
 -> 02353720 System.Windows.Forms.Timer
 -> 0235375c System.EventHandler
 -> 0205d524 Example1.MainFormExample1
 -> 02435378 Example2.WindowsControls.Controls.Capturing.Inspections.Inspections
 -> 024356dc System.Collections.ArrayList
 -> 0248c8ec System.Object[]
 -> 02446224 Example2.WindowsControls.Example3.ComboBox
 -> 024463d4 System.ComponentModel.EventHandlerList
 -> 02487ab8 System.ComponentModel.EventHandlerList+ListEntry
 -> 02487aa4 System.ComponentModel.EventHandlerList+ListEntry
 -> 02487a90 System.ComponentModel.EventHandlerList+ListEntry
 -> 02487a7c System.ComponentModel.EventHandlerList+ListEntry
 -> 02487a68 System.ComponentModel.EventHandlerList+ListEntry
 -> 02487a54 System.ComponentModel.EventHandlerList+ListEntry
 -> 024484d4 System.ComponentModel.EventHandlerList+ListEntry
 -> 02448124 System.ComponentModel.EventHandlerList+ListEntry
 -> 0248abbc System.EventHandler
 -> 0248ab8c System.Object[]
 -> 02448104 System.EventHandler
 -> 02444bf4 Example2.WindowsControls.Controls.Capturing.AcquisitionControl.SearchPage
 -> 02445008 System.ComponentModel.EventHandlerList
 -> 02487a40 System.ComponentModel.EventHandlerList+ListEntry
 -> 02487a2c System.ComponentModel.EventHandlerList+ListEntry
 -> 02487a18 System.ComponentModel.EventHandlerList+ListEntry
 -> 02487a04 System.ComponentModel.EventHandlerList+ListEntry
 -> 024879f0 System.ComponentModel.EventHandlerList+ListEntry
 -> 024491d8 System.ComponentModel.EventHandlerList+ListEntry
 -> 024cb554 System.Windows.Forms.MouseEventHandler
 -> 024cb534 System.Object[]
 -> 024491b8 System.Windows.Forms.MouseEventHandler
 -> 0244463c Example2.WindowsControls.Controls.Capturing.AcquisitionControl.Acquisition
 -> 024448ec System.Windows.Forms.Control+ControlNativeWindow
 -> 024c7368 System.Windows.Forms.NativeMethods+WndProc
 
Found 1 unique roots (run '!GCRoot -all' to see all roots).
0:000> !gcroot -nostacks 024c5de0 
HandleTable:
 001f1130 (strong handle)
 -> 02353720 System.Windows.Forms.Timer
 -> 0235375c System.EventHandler
 -> 0205d524 Example1.MainFormExample1
 -> 02435378 Example2.WindowsControls.Controls.Capturing.Inspections.Inspections
 -> 024c5c28 System.Windows.Forms.LayoutEventArgs
 -> 02482048 Infragistics.Win.UltraWinTabControl.UltraTabControl
 -> 02489f2c Infragistics.Win.UltraWinTabControl.UltraTabsCollection
 -> 02489fac System.Collections.ArrayList
 -> 02489fc4 System.Object[]
 -> 0243570c Infragistics.Win.UltraWinTabControl.UltraTab
 -> 024358e0 Infragistics.Win.UltraWinTabControl.UltraTabPageControl
 -> 024c5da4 Infragistics.Win.UIAutomation.UiaProviderControlNativeWindow
 -> 024c5de0 System.Windows.Forms.NativeMethods+WndProc
 
Found 1 unique roots (run '!GCRoot -all' to see all roots).
0:000> !gcroot -nostacks 02483e7c 
HandleTable:
 001f1130 (strong handle)
 -> 02353720 System.Windows.Forms.Timer
 -> 0235375c System.EventHandler
 -> 0205d524 Example1.MainFormExample1
 -> 02435378 Example2.WindowsControls.Controls.Capturing.Inspections.Inspections
 -> 024356dc System.Collections.ArrayList
 -> 0248c8ec System.Object[]
 -> 02477260 Example2.WindowsControls.Example3.ComboBox
 -> 02477410 System.ComponentModel.EventHandlerList
 -> 02489638 System.ComponentModel.EventHandlerList+ListEntry
 -> 02489624 System.ComponentModel.EventHandlerList+ListEntry
 -> 02489610 System.ComponentModel.EventHandlerList+ListEntry
 -> 024895fc System.ComponentModel.EventHandlerList+ListEntry
 -> 024895e8 System.ComponentModel.EventHandlerList+ListEntry
 -> 024895d4 System.ComponentModel.EventHandlerList+ListEntry
 -> 02479510 System.ComponentModel.EventHandlerList+ListEntry
 -> 02479160 System.ComponentModel.EventHandlerList+ListEntry
 -> 0248c2bc System.EventHandler
 -> 0248c28c System.Object[]
 -> 02479140 System.EventHandler
 -> 02475c50 Example2.WindowsControls.Controls.Capturing.AcquisitionControl.SearchPage
 -> 02476064 System.ComponentModel.EventHandlerList
 -> 024895c0 System.ComponentModel.EventHandlerList+ListEntry
 -> 024895ac System.ComponentModel.EventHandlerList+ListEntry
 -> 02489598 System.ComponentModel.EventHandlerList+ListEntry
 -> 02489584 System.ComponentModel.EventHandlerList+ListEntry
 -> 02489570 System.ComponentModel.EventHandlerList+ListEntry
 -> 0247a178 System.ComponentModel.EventHandlerList+ListEntry
 -> 024cc60c System.Windows.Forms.MouseEventHandler
 -> 024cc5ec System.Object[]
 -> 0247a158 System.Windows.Forms.MouseEventHandler
 -> 02475698 Example2.WindowsControls.Controls.Capturing.AcquisitionControl.Acquisition
 -> 02475a4c System.ComponentModel.EventHandlerList
 -> 024894e4 System.ComponentModel.EventHandlerList+ListEntry
 -> 024894d0 System.ComponentModel.EventHandlerList+ListEntry
 -> 024894bc System.ComponentModel.EventHandlerList+ListEntry
 -> 024894a8 System.ComponentModel.EventHandlerList+ListEntry
 -> 02489494 System.ComponentModel.EventHandlerList+ListEntry
 -> 02481638 System.ComponentModel.EventHandlerList+ListEntry
 -> 024cc58c System.Windows.Forms.MouseEventHandler
 -> 024cc574 System.Object[]
 -> 02481618 System.Windows.Forms.MouseEventHandler
 -> 02471e64 Example2.WindowsControls.Controls.Capturing.MontageCapturing.MontageCapturing
 -> 024721b4 System.Collections.ArrayList
 -> 02481848 System.Object[]
 -> 0247af90 Example2.WindowsControls.Example3.InfraTextEditor
 -> 0247b924 System.ComponentModel.EventHandlerList
 -> 02489b9c System.ComponentModel.EventHandlerList+ListEntry
 -> 02489b88 System.ComponentModel.EventHandlerList+ListEntry
 -> 02489b74 System.ComponentModel.EventHandlerList+ListEntry
 -> 02483d6c System.ComponentModel.EventHandlerList+ListEntry
 -> 0247fbb0 System.ComponentModel.EventHandlerList+ListEntry
 -> 0247fb7c System.ComponentModel.EventHandlerList+ListEntry
 -> 0247fb48 System.ComponentModel.EventHandlerList+ListEntry
 -> 0247f480 System.ComponentModel.EventHandlerList+ListEntry
 -> 0247f40c System.ComponentModel.EventHandlerList+ListEntry
 -> 0247bb60 System.ComponentModel.EventHandlerList+ListEntry
 -> 0247b95c System.ComponentModel.EventHandlerList+ListEntry
 -> 0247b948 System.ComponentModel.EventHandlerList+ListEntry
 -> 0247f440 System.Windows.Forms.ControlEventHandler
 -> 0247a8bc Example2.WindowsControls.Controls.Capturing.MontageCapturing.MontageEntry
 -> 02486000 System.Windows.Forms.LayoutEventArgs
 -> 0247aca8 Example2.WindowsControls.Example3.GroupBox
 -> 02485ff0 System.Windows.Forms.LayoutEventArgs
 -> 0247adc8 Example2.WindowsControls.Example3.TableLayoutPanel
 -> 0247ae84 System.Windows.Forms.PropertyStore
 -> 02481170 System.Windows.Forms.PropertyStore+ObjectEntry[]
 -> 0247ee28 System.Windows.Forms.Layout.TableLayout+ContainerInfo
 -> 0247ff68 System.Object[]
 -> 0247f044 System.Windows.Forms.Layout.TableLayout+LayoutInfo
 -> 0247c1dc Example2.WindowsControls.Example3.InfraTextEditor
 -> 02483e40 Infragistics.Win.UltraWinEditors.TextEditorControlBase+TextEditorControlBaseNativeWindow
 -> 02483e7c System.Windows.Forms.NativeMethods+WndProc
 
Found 1 unique roots (run '!GCRoot -all' to see all roots).
0:000> !gcroot -nostacks 02479c60 
HandleTable:
 001f1130 (strong handle)
 -> 02353720 System.Windows.Forms.Timer
 -> 0235375c System.EventHandler
 -> 0205d524 Example1.MainFormExample1
 -> 02435378 Example2.WindowsControls.Controls.Capturing.Inspections.Inspections
 -> 024356dc System.Collections.ArrayList
 -> 0248c8ec System.Object[]
 -> 02477260 Example2.WindowsControls.Example3.ComboBox
 -> 02477410 System.ComponentModel.EventHandlerList
 -> 02489638 System.ComponentModel.EventHandlerList+ListEntry
 -> 02489624 System.ComponentModel.EventHandlerList+ListEntry
 -> 02489610 System.ComponentModel.EventHandlerList+ListEntry
 -> 024895fc System.ComponentModel.EventHandlerList+ListEntry
 -> 024895e8 System.ComponentModel.EventHandlerList+ListEntry
 -> 024895d4 System.ComponentModel.EventHandlerList+ListEntry
 -> 02479510 System.ComponentModel.EventHandlerList+ListEntry
 -> 02479160 System.ComponentModel.EventHandlerList+ListEntry
 -> 0248c2bc System.EventHandler
 -> 0248c28c System.Object[]
 -> 02479140 System.EventHandler
 -> 02475c50 Example2.WindowsControls.Controls.Capturing.AcquisitionControl.SearchPage
 -> 02476abc Infragistics.Win.UltraWinEditors.UltraOptionSet
 -> 02476bd4 System.Windows.Forms.Control+ControlNativeWindow
 -> 02479c60 System.Windows.Forms.NativeMethods+WndProc
 
Found 1 unique roots (run '!GCRoot -all' to see all roots).
0:000> !gcroot 0242d0d0 
HandleTable:
 001f1130 (strong handle)
 -> 02353720 System.Windows.Forms.Timer
 -> 0235375c System.EventHandler
 -> 0205d524 Example1.MainFormExample1
 -> 0242cbd0 Infragistics.Win.UltraWinTabControl.UltraTab
 -> 0242ce2c Infragistics.Win.UltraWinTabControl.UltraTabPageControl
 -> 0242d094 Infragistics.Win.UIAutomation.UiaProviderControlNativeWindow
 -> 0242d0d0 System.Windows.Forms.NativeMethods+WndProc
 
Found 1 unique roots (run '!GCRoot -all' to see all roots).

Recently I was troubleshooting an intermittent, hard to reproduce application crash in the production environment. We had a restriction to install debuggers. In such situation, Windows Error Reporting (WER) comes as a handy tool. Here are the settings that we need:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps\ManagedCorruption.exe]

"DumpFolder"= REG_EXPAND_SZ: C:\WERDumps

"DumpCount"=dword:00000005

"DumpType"=dword:00000002

Here you need to replace the ‘ManagedCorruption’ with your executable. You can also change the path C:\WERDumps as per your preference.

We made the necessary entries on customer’s machine, but the dumps were not created when the application crashed. The same settings were working fine on other machines, but not on the production machine. What do we do now?

On further research we found that “Windows Error Reporting Service” was disabled on the machine.

Here is how you can verify:

1. Start->Run->services.msc
2. Select “Windows Error Reporting Service”.
3. Set its startup type to “Manual (Trigger Start)” : 

Once we enabled the service, the dumps were collected successfully.

This is particularly very useful in situations like Fatal Execution Engine error where the application doesn’t throw exceptions. So make sure that “Windows Error Reporting Service” is enabled on the machine.

Here is the MSDN article which speaks about Collecting User-Mode Dumps

I have come across this issue where customer has migrated his website from .NET Framework 3.5 to .NET Framework 4.0 and the Web server has .NET Framework 4.5. 1 installed. The web application faces intermittently hangs. During our troubleshooting and different dump analysis we found that the process is hung due to a deadlock scenario between two threads. 

Dump analysis on the Hang dump taken

 0:046> kL 

# ChildEBP RetAddr 

00 01d1ef70 77928e44 ntdll!ZwWaitForSingleObject

01 01d1efd4 77928d28 ntdll!RtlpWaitOnCriticalSection

02 01d1effc 7405bf8f ntdll!RtlEnterCriticalSection

03 (Inline) -------- clr!UnsafeEnterCriticalSection

04 01d1f010 73fb3b2f clr!CrstBase::SpinEnter

05 01d1f030 740a7337 clr!CrstBase::Enter

06 (Inline) -------- clr!CrstBase::AcquireLock

07 (Inline) -------- clr!CrstBase::CrstHolder::{ctor

08 01d1f0d0 73fd4a5b clr!ClassLoader::LoadTypeHandleForTypeKey_Body

09 01d1f114 73fd56ca clr!ClassLoader::LoadTypeHandleForTypeKey

0a 01d1f19c 73fd1ca7 clr!ClassLoader::LoadTypeDefThrowing

0b 01d1f210 73fc23a2 clr!ClassLoader::LoadTypeDefOrRefThrowing

0c 01d1f2c8 73fc599b clr!SigPointer::GetTypeHandleThrowing

0d 01d1f3a4 73fd2f90 clr!SigPointer::GetTypeHandleThrowing

      0e 01d1f3e4 73fffd83 clr!ClassLoader::LoadTypeDefOrRefOrSpecThrowing

 0:046> .frame 0n4;dv /t /v

04 01d1f010 73fb3b2f clr!CrstBase::SpinEnter  

@esi              class CrstBase * this = 0x1ebff438

@ebx              unsigned long iter = 0xa

@edi              unsigned long i = 0

<unavailable>     int delayCount = <value unavailable>

0:046> dt _RTL_CRITICAL_SECTION  0x1ebff438

w3wp!_RTL_CRITICAL_SECTION

   +0x000 DebugInfo        : 0x1ec36c38 _RTL_CRITICAL_SECTION_DEBUG

   +0x004 LockCount        : 0n-14

   +0x008 RecursionCount   : 0n1

   +0x00c OwningThread     : 0x000023d0 Void     >> thread 47

   +0x010 LockSemaphore    : 0x00000f20 Void

               +0x014 SpinCount        : 0

 This is how 47 looks like

0:047> kL

# ChildEBP RetAddr 

00 1dddd3f0 77928e44 ntdll!ZwWaitForSingleObject

01 1dddd454 77928d28 ntdll!RtlpWaitOnCriticalSection

02 1dddd47c 7405bf8f ntdll!RtlEnterCriticalSection

03 (Inline) -------- clr!UnsafeEnterCriticalSection

04 1dddd490 73fb3b2f clr!CrstBase::SpinEnter

05 1dddd4b0 740a7337 clr!CrstBase::Enter

06 (Inline) -------- clr!CrstBase::AcquireLock

07 (Inline) -------- clr!CrstBase::CrstHolder::{ctor

08 1dddd550 73fd4a5b clr!ClassLoader::LoadTypeHandleForTypeKey_Body

09 1dddd594 73fd56ca clr!ClassLoader::LoadTypeHandleForTypeKey

0a 1dddd61c 73fd1ca7 clr!ClassLoader::LoadTypeDefThrowing

0b 1dddd690 73febfc9 clr!ClassLoader::LoadTypeDefOrRefThrowing

0c 1dddd6e0 73feab99 clr!ClassLoader::LoadApproxTypeThrowing

0d 1dddd714 73fe7ae7 clr!ClassLoader::LoadApproxParentThrowing

0e 1dddd8c4 73fd8330 clr!ClassLoader::CreateTypeHandleForTypeDefThrowing

 0:047> dt this  0x1ecd5878

Local var @ esi Type CrstBase*

  +0x000 m_criticalsection : _RTL_CRITICAL_SECTION

   +0x000 m_pHostCrst      : 0x028714c8 IHostCrst

   +0x018 m_dwFlags        : 0xc0000023

0:047> dt _RTL_CRITICAL_SECTION  0x1ecd5878

w3wp!_RTL_CRITICAL_SECTION

   +0x000 DebugInfo        : 0x028714c8 _RTL_CRITICAL_SECTION_DEBUG

   +0x004 LockCount        : 0n-10

   +0x008 RecursionCount   : 0n1

   +0x00c OwningThread     : 0x00001e20 Void   >> thread 46

   +0x010 LockSemaphore    : 0x00000f1c Void

   +0x014 SpinCount        : 0

 One of the waiting thread

 0:320> kM10

# ChildEBP RetAddr 

00 3fe5d690 77928e44 ntdll!ZwWaitForSingleObject

01 3fe5d6f4 77928d28 ntdll!RtlpWaitOnCriticalSection

02 3fe5d71c 7405bf8f ntdll!RtlEnterCriticalSection

03 (Inline) -------- clr!UnsafeEnterCriticalSection+0x7

04 3fe5d730 73fb3b2f clr!CrstBase::SpinEnter

05 3fe5d754 73fc7e99 clr!CrstBase::Enter

06 (Inline) -------- clr!ListLockEntry::FinishDeadlockAwareEnter

07 3fe5d784 73fe38a5 clr!ListLockEntry::LockHolder::DeadlockAwareAcquire

08 3fe5d85c 73ff1ebe clr!MethodDesc::MakeJitWorker

09 3fe5d8d0 73fc50f6 clr!MethodDesc::DoPrestub

0a 3fe5d938 73fb42b9 clr!PreStubWorker

0b 3fe5d968 23b36012 clr!ThePreStub

This has been found as bug and has been fixed in the latest release of CLR. The other work around for .NET 4.0 , 4.5.1 and 4.5.2 is to disable profileGuidedOptimizations. This is a new feature added in .NET 4.5. this workaround may add fractional increase in the startup time.

Written by: Irfan Ahmed, Senior Support Escalation Engineer, EMEA/INDIA Team

Reviewed by: Rachan Rosario, Escalation Engineer