Resource cleanup and Completion of Thread Pool Callbacks

In order for the callback function to perform user defined tasks or works, it often has to use some custom resources, and those resources normally requires proper cleanup when the callback function is no longer needed, things like memories to be released, handles to be closed or DLL to be unloaded. Because of the concurrent nature of thread pool API, the resource cleanup and completion of thread pool callbacks must have some sort of synchronization, otherwise there would be a race condition between the thread which is cleaning up the resources and the callback threads which are still using those resources.

In particular, SetThreadpoolWait function allows a thread pool worker thread to call the wait object’s callback function after the handle becomes signaled (or timeout happens). If the program wants to listen to the event notification continuously, it must re-register (by calling SetThreadpoolWait again) the event with the wait object before signaling it each time to trigger the wait callback . Usually, the re-register logic is put inside the callback function itself.

VOID MyWaitCallback (…)
{
    //user codes to perform necessary tasks or works;
    …;

     //using some resources;
    …;

    //re-register another wait
    SetThreadpoolWait(Wait, myWaitHandle, …);                                             (1.1)
};

To overcome the race condition between any potential in-flight callbacks and the resource cleanup, the key is to make sure that the code waits for all callbacks to complete before starts freeing any resources. [CPW08] chapter 7 gives a three-step process to achieve this for wait callbacks: (1) cancel the waits, (2) wait for callbacks to finish, and finally (3) close the wait object

VOID StopTheWait (…)
{

     //Step 1: cancel the wait.
    SetThreadpoolWait(myWait, NULL, NULL);                                                 (1.2)
    //Step 2: wait. m_pNduByteLimitWait(NULL),
    WaitForThreadpoolWaitCallbacks(myWait, FALSE);                                  (1.3)
    //Step 3: close the wait object.
    CloseThreadpoolWait(myWait);                                                                  (1.4)

    //now it’s safe to clean up resources, freeing the resource
    …;
};

However, this solution is not completely correct and it does not solve all race conditions. Here it is why:

Imaging there is a in-flight callback, which is about to execute line (1.1), and it gets preempted. At the same time, the program decides to stop the thread pool wait and start the cleaning up, so line (1.2) gets executed. After that the OS does context switch back to the callback thread, and executes line (1.1). The net effect of these interleaving operations is that line (1.1) happens after line (1.2), and the wait is in fact not being canceled. Therefore, additional callbacks could be queued after line (1.3), and start executing before line (1.4). In this case, the code moves on to free the resources concurrently as the callback executes.

Graceful completion of thread pool wait callback

Here are two approaches for stopping thread pool wait in a truly graceful manner:

If the program have control over the source of the wait notification

In this case, the code can just simply turn off the source of the notification first, before waiting for any in-flight/queued callbacks to be finished. For example, it might call the matching UnregisterXXX function to stop the wait event from signaling, or stop calling SetEvent function if the signal is generated by the code itself. After than, it can call WaitForThreadpoolWaitCallbacks and CloseThreadpoolWait to stop thread pool wait safely.

If the program does NOT have control over the source of the wait notification

Neeraj Singh proposed the following pattern:

VOID MyWaitCallback (…)
{
    …;

    if (dontReregister == FALSE)
        SetThreadpoolWait(Wait, myWaitHandle, …);                                                                                       (2.1)
};

VOID StopTheWait (…)
{

    //Step 1: set “don’t re-register” flag.
    dontReregister = TRUE;                                                                                                                            (2.2)
    //Step 2: wait for any in-flight callbacks which have NOT observed the flag to complete
    WaitForThreadpoolWaitCallbacks(myWait, TRUE);                                                                                   (2.3)
    //Step 3: call the wait.
    SetThreadpoolWait(myWait, NULL, NULL);                                                                                               (2.4)
    //Step 4: wait for any additional callbacks which should have observed the flag
    WaitForThreadpoolWaitCallbacks(myWait, TRUE);                                                                                   (2,5)
    //Step 3: close the wait object.
    CloseThreadpoolWait(myWait);                                                                                                                 (2.6)

    //now it’s safe to clean up resources, freeing the resource
    …;
};

The reason this pattern works is as following:

The if statement before line (2.1) makes sure that dontReregister flag is observed by any running callbacks before it calls SetThreadpoolWait. When the code decides to stop the thread pool wait, it first sets the dontReregister flag at line (2.2). The first WaitForThreadpoolWaitCallbacks at line (2.3) ensures that no callbacks are still running which have NOT observed that flag. After that, line (2.4) prevents any further callbacks from being queued from this point onwards. The second WaitForThreadpoolWaitCallbacks at line (2.5) is added to wait for/cancel any in-flight/queued callbacks which may sneak in between line (2.3 and line(2.4). Finally, the wait object is closed at line (2.6).

Windows Thread Pool API already has necessary memory barriers inside, so the re-ordering of memory access has been constrained at line (2.1) and line (2.3) even if nothing special is done for dontReregister flag.

Take Away:

On the surface, multithreading seems a straightforward and simple programming exercise and Windows Thread Pool API makes doing it even more accessible, but the truth is that it is extremely hard to get it done right. The bugs resulted from it are tricky and it’s difficult for the programmers to locate the cause because of the timing involved, even with substantial test effect. Furthermore, the consequences of this kind of bugs are usually undefined (which often means devastating).

There are many other deeper aspects of multithreading, such like lock performance, memory/cache synchronization, power consumption and more, to be understood by the developers before they can really enjoy the benefit of multithreading. In short, there is a huge risk involved in adopting multithreading. Hence, before starting, carefully judge the cost/benefit and use it responsibly

References:

1. [CPW08] Concurrent Programming On Windows

2. [WVCC07] Windows via C/C++

3. [MSDN07/10] Improve Scalability With New Thread Pool APIs

4. [MSDN11/08] The Windows Thread Pool and Work

5. [MSDN11/09] The Thread Pool Environment

6. [MSDN11/10] Thread Pool Cancellation and Cleanup

7. [MSDN11/11] Thread Pool Synchronization

Thanks to Neeraj Singh and Alex Bendetov for answering my questions on this topic

The Bug:

The following C# code segment is to obtain the hardware plug-and-play information from a remote machine:

ConnectionOptions options = new ConnectionOptions(

    null,

    "administrator",

    "!!123abc",

    "NTLMDOMAIN:10.177.17.6",

    ImpersonationLevel.Impersonate,

    AuthenticationLevel.PacketPrivacy,

    true,

    null,

TimeSpan.MaxValue);

 

ManagementScope scope = new ManagementScope(

    @"\\10.177.17.6\root\cimv2",

options);

 

scope.Connect();

 

ManagementClass targetClass = new ManagementClass(

    scope,

    new ManagementPath("Win32_PnPEntity"),

    null);

 

ManagementObjectCollection collection = targetClass.GetInstances(); 

foreach (ManagementBaseObject wmiObject in collection)

{

   ...

}

The above codes use WMI .NET with 3.5 SP1 runtime. It connect to the target machine (10.177.17.6) using that machine’s local administrator account, and it also specifies NTLM as the authentication protocol (NTLMDOMAIN).  After the connection is established, it remotely enumerates all the instances from WMI Win32_PnPEntity class.

This seemingly simple code works fine in most of time; however, in one particular situation: if the target machine is running under Windows Preinstallation Environment 3.0, the code throws an COMException at some random time during the foreach loop. And the exception code can be either RPC_S_CALL_FAILED_DNE (0x800706BF The remote procedure call failed) or RPC_S_CALL_FAILED (0x800706BE The remote procedure call failed and did not execute).

The Debugging:

Firstly, some knowledge about how Microsoft technologies are layered can help to understand why the exception talks about RPC failure and why it is the type of COMException:

The WMI .NET is just a wrapper above native WMI framework, and the remote communication feature of WMI is built on top of DCOM, and then again, DCOM utilize MSRPC as its inter-process communication infrastructure. So this exception is basically trying to tell the developers that, there is an error on the RPC level detected by DCOM stack, and then that error is reported to the .NET framework layer, and lastly .NET framework raises that error as COMException back to the user code.

If Windbg is attached to the codes, the output shows that error 0x80070721 and 0x80070005 always happened before that COMException

(dd8.a6c): Unknown exception - code 80070721 (first chance)
(dd8.a6c): Unknown exception - code 80070005 (first chance)
(dd8.a6c): Unknown exception - code 800706bf (first chance)

0x8007005 is access denied, and 0x80070721 is less known, use !error command

0:000> !error 0x80070721
Error code: (HRESULT) 0x80070721 (2147944225) - A security package specific error occurred.

It looks like some security related problem has happened right before the client codes failed. That’s all the information there, and nothing furthermore can be investigated in the debugger (unless some serious reverse engineering are done, which is beyond this case study). It seems the debugging is getting into a dead-end, and now it is the time to start the bottom-up approach.

Note: Another possible approach is to setup the debugger to use symbols from Microsoft Reference Source Server, which would provide source codes access for .NET framework.

Use any packet analyzers (such as WireShark or Microsoft Network Monitor) and start a capture session, and repro this bug again, then save the capture file. The content of the capture file is shown below

No.     Time        Source                Destination           Protocol Info

    406 5.078086    10.177.17.15          10.177.17.6           DCERPC   Bind: call_id: 3 ISystemActivator V0.0, NTLMSSP_NEGOTIATE
    407 5.078721    10.177.17.6           10.177.17.15          DCERPC   Bind_ack: call_id: 3, NTLMSSP_CHALLENGE accept max_xmit: 5840 max_recv: 5840
    408 5.078987    10.177.17.15          10.177.17.6           DCERPC   AUTH3: call_id: 3, NTLMSSP_AUTH, User: 10.177.17.6\administrator

At the beginning of our capture file, the RPC layer established TCP connection and started to authenticate with the remote machine (10.177.17.6). The first 3 frames (highlighted in gray) are the standard 3-leg NTLM authentication protocol, NEGOTIATE –> CHALLENGE –> AUTH, see [PWS00] chapter 7 for details. As recorded in frame 408 (AUTH), the credential used for authentication is the local administrator account on the target machine. Everything is as expected.

    409 5.079045    10.177.17.15          10.177.17.6           ISystemActivator RemoteCreateInstance request[Malformed Packet] 
    411 5.081677    10.177.17.6           10.177.17.15          ISystemActivator RemoteCreateInstance response[Malformed Packet]

    ...

    517 5.334018    10.177.17.15          10.177.17.6           DCERPC   Request: call_id: 7 opnum: 6 ctx_id: 0 IEnumWbemClassObject V0
    518 5.334248    10.177.17.6           10.177.17.15          TCP      49157 > 59916 [ACK] Seq=221 Ack=789 Win=130560 Len=0
    519 5.334574    10.177.17.6           10.177.17.15          DCERPC   Response: call_id: 7 ctx_id: 0 IEnumWbemClassObject V0

    ...

After the initial authentication, the RPC conversation is established, and the DCOM starts to do its own stuff, such as remote activation (frame 409 - 411), reference count keeping, object lifecycle management etc., and then it is followed by WMI communication, such as IEnumWbemClassObject::Clone (frame 517 – 519, opnum 6 is Clone) and etc.. Most of those packages are about standard operations and have no interests to this bug, so they are not shown in this blog.

Note: see [EC98] and [MS-WMI] for more information about DCOM and WMI

Then, what happened next is most interesting

    520 5.334778    10.177.17.15          10.177.17.6           DCERPC   Alter_context: call_id: 8, 2 context items, 1st IRemUnknown2 V0.0, NTLMSSP_NEGOTIATE
    521 5.334891    10.177.17.6           10.177.17.15          DCERPC   Alter_context_resp: call_id: 8, NTLMSSP_CHALLENGE accept max_xmit: 5840 max_recv: 5840
    522 5.335016    10.177.17.15          10.177.17.6           DCERPC   AUTH3: call_id: 8, NTLMSSP_AUTH, User: REDMOND\jiangyue
    523 5.335047    10.177.17.15          10.177.17.6           IRemUnknown2 RemQueryInterface request IID[1]=IManagedObject
    524 5.335231    10.177.17.6           10.177.17.15          TCP      49157 > 59916 [ACK] Seq=821 Ack=1543 Win=129792 Len=0
    525 5.335313    10.177.17.6           10.177.17.15          DCERPC   Fault: call_id: 8 ctx_id: 1 status: Unknown (0x00000721)
    526 5.335379    10.177.17.15          10.177.17.6           IRemUnknown2 RemQueryInterface request IID[1]=IProvideClassInfo
    527 5.335574    10.177.17.6           10.177.17.15          DCERPC   Fault: call_id: 9 ctx_id: 1 status: nca_s_fault_access_denied
    528 5.335574    10.177.17.6           10.177.17.15          TCP      49157 > 59916 [FIN, ACK] Seq=885 Ack=1687 Win=131328 Len=0
    529 5.335600    10.177.17.15          10.177.17.6           TCP      59916 > 49157 [ACK] Seq=1687 Ack=886 Win=130304 Len=0
    530 5.336227    10.177.17.15          10.177.17.6           TCP      59916 > 49157 [FIN, ACK] Seq=1687 Ack=886 Win=130304 Len=0
    531 5.336356    10.177.17.6           10.177.17.15          TCP      49157 > 59916 [ACK] Seq=886 Ack=1688 Win=131328 Len=0

As highlighted by yellow color, RPC decided to re-authenticate with target machine using a different set of credential redmond\jiangyue (frame 520 - 522), and this credential is belong to the security token of current calling thread. However, redmond\jiangyue is not a valid credential on the target machine.  Therefore, when the client machine called QueryInterface on the remote object for IManagedObject, the RPC layer on the remote machine responded with Fault message containing error code 0x00000721 (frame 525). Then again, the client tried to make another request for IProviderClassInfo to remote PRC layer, this time it responded with another Fault message containing error code 0x00000005 (access_denied). Those two errors matched the two exceptions seen by Windbg earlier on. After received two consecutive failed attempt, the RPC layer on the remote machine decided to send out the FIN message and close the current TCP connection, as indicated in frame 528 – 531.

    532 5.336862    10.177.17.15          10.177.17.6           TCP      59917 > 49157 [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=8 SACK_PERM=1
    533 5.336943    10.177.17.6           10.177.17.15          TCP      49157 > 59917 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1460 WS=8 SACK_PERM=1
    534 5.337055    10.177.17.15          10.177.17.6           TCP      59917 > 49157 [ACK] Seq=1 Ack=1 Win=131328 Len=0
    535 5.337389    10.177.17.15          10.177.17.6           DCERPC   Bind: call_id: 11, 2 context items, 1st IEnumWbemClassObject V0.0, NTLMSSP_NEGOTIATE
    536 5.337595    10.177.17.6           10.177.17.15          DCERPC   Bind_ack: call_id: 11, NTLMSSP_CHALLENGE accept max_xmit: 5840 max_recv: 5840
    537 5.337729    10.177.17.15          10.177.17.6           DCERPC   AUTH3: call_id: 11, NTLMSSP_AUTH, User: 10.177.17.6\administrator
    538 5.337755    10.177.17.15          10.177.17.6           DCERPC   Request: call_id: 11 opnum: 3 ctx_id: 0 IEnumWbemClassObject V0
    539 5.337971    10.177.17.6           10.177.17.15          TCP      49157 > 59917 [ACK] Seq=221 Ack=789 Win=130560 Len=0
    540 5.338106    10.177.17.6           10.177.17.15          DCERPC   Response: call_id: 11 ctx_id: 0 IEnumWbemClassObject V0
    541 5.338385    10.177.17.15          10.177.17.6           DCERPC   Alter_context: call_id: 12, 2 context items, 1st IRemUnknown2 V0.0
    542 5.338450    10.177.17.6           10.177.17.15          DCERPC   Alter_context_resp: call_id: 12 Provider rejection, reason: Proposed transfer syntaxes not supported
    543 5.338491    10.177.17.15          10.177.17.6           IRemUnknown2 RemQueryInterface request
    544 5.338605    10.177.17.6           10.177.17.15          IRemUnknown2 RemQueryInterface response 

The client seemed ignore all the security errors and continue its execution. The client made a new TCP connection to the remote machine by sending a new SYN message, as indicated in frame 532 – 534. And this time, it uses the right credential 10.177.17.6\administrator so it was enable to make some progress, until again the RPC layer decided to re-authenticate with a wrong credential redmond\jiangyue to query IManagedObject interface. This pattern repeated itself again and again in the captured file.

The client was struggling to move forward until it reached the foreach loop, where it needed to call MoveNext() on the enumerator repeatedly.  

    745 5.338889    10.177.17.15          10.177.17.6           DCERPC   Alter_context: call_id: 13 IWbemFetchSmartEnum V0.0, NTLMSSP_NEGOTIATE 
    746 5.338996    10.177.17.6           10.177.17.15          DCERPC   Alter_context_resp: call_id: 13, NTLMSSP_CHALLENGE accept max_xmit: 5840 max_recv: 5840 
    747 5.339116    10.177.17.15          10.177.17.6           DCERPC   AUTH3: call_id: 13, NTLMSSP_AUTH, User: REDMOND\jiangyue 
    748 5.339146    10.177.17.15          10.177.17.6           DCERPC   Request: call_id: 13 opnum: 3 ctx_id: 3 IWbemFetchSmartEnum V0 
    749 5.339337    10.177.17.6           10.177.17.15          TCP      49157 > 59917 [ACK] Seq=685 Ack=1759 Win=131328 Len=0 
    750 5.339415    10.177.17.6           10.177.17.15          DCERPC   Fault: call_id: 13 ctx_id: 3 status: Unknown (0x00000721) 
    756 5.339866    10.177.17.6           10.177.17.15          DCERPC   Response: call_id: 14 ctx_id: 0 IEnumWbemClassObject V0 
    757 5.339887    10.177.17.15          10.177.17.6           TCP      59917 > 49157 [ACK] Seq=1871 Ack=5101 Win=131328 Len=0 
    758 5.340308    10.177.17.15          10.177.17.6           DCERPC   Request: call_id: 15 opnum: 3 ctx_id: 3 IWbemFetchSmartEnum V0 
    759 5.340479    10.177.17.6           10.177.17.15          DCERPC   Fault: call_id: 15 ctx_id: 3 status: nca_s_fault_access_denied 
    760 5.340500    10.177.17.6           10.177.17.15          TCP      49157 > 59917 [FIN, ACK] Seq=5133 Ack=1967 Win=131072 Len=0 
    761 5.340515    10.177.17.15          10.177.17.6           TCP      59917 > 49157 [ACK] Seq=1967 Ack=5134 Win=131328 Len=0 
    762 5.340568    10.177.17.15          10.177.17.6           TCP      59917 > 49157 [FIN, ACK] Seq=1967 Ack=5134 Win=131328 Len=0 
    763 5.340651    10.177.17.6           10.177.17.15          TCP      49157 > 59917 [ACK] Seq=5134 Ack=1968 Win=131072 Len=0  

As highlighted above in yellow, after the client enter the foreach loop,  it sent out the IWbemFetchSmartEnum::GetSmartEnum request (in frame 747, opnum 3) using the wrong credential once again, and it resulted the remote RPC to drop the TCP connection. This was the last user actions. After that, DCOM started garbage collections (frame 842-845) and then the communication over the wire was completely stopped.

    836 6.612612    10.177.17.15          10.177.17.6           TCP      59918 > 49157 [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=8 SACK_PERM=1  
    837 6.612861    10.177.17.6           10.177.17.15          TCP      49157 > 59918 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1460 WS=8 SACK_PERM=1  
    838 6.613020    10.177.17.15          10.177.17.6           TCP      59918 > 49157 [ACK] Seq=1 Ack=1 Win=131328 Len=0 
    839 6.613592    10.177.17.15          10.177.17.6           DCERPC   Bind: call_id: 17, 3 context items, 1st IRemUnknown2 V0.0, NTLMSSP_NEGOTIATE  
    840 6.614019    10.177.17.6           10.177.17.15          DCERPC   Bind_ack: call_id: 17, NTLMSSP_CHALLENGE accept max_xmit: 5840 max_recv: 5840  
    841 6.614199    10.177.17.15          10.177.17.6           DCERPC   AUTH3: call_id: 17, NTLMSSP_AUTH, User: 10.177.17.6\administrator  
    842 6.614236    10.177.17.15          10.177.17.6           IRemUnknown2 RemRelease request  
    843 6.614429    10.177.17.6           10.177.17.15          TCP      49157 > 59918 [ACK] Seq=245 Ack=881 Win=130560 Len=0  
    844 6.614761    10.177.17.6           10.177.17.15          IRemUnknown2 RemRelease response  
    845 6.616079    10.177.17.15          10.177.17.6           TCP      59918 > 49157 [RST, ACK] Seq=881 Ack=309 Win=0 Len=0

Frame 845 is the last record in the capture file.

After walking through the capture file, the root cause of the mysterious COMException should be clear now: Some code in WMI .NET layer was connect to remote RPC with a wrong credential, which makes the RPC server (on the remote machine) to drop the TCP connection. While the foreach loop was not aware of this, and kept reusing the dead TCP connection to make requests. When that happens, RPC reports generic error 0x6BE or 0x6BF

Now the next question is that why this issue only happens if the remote machine is running under WinPE environment, not in other full OS version. To answer that, another capture is made for the network traffic with the same codes running against a remote machine under full OS. By analyzing the new capture, it turns out that in full OS scenario, the RCP server behaves differently: It does NOT terminate the TCP connection when wrong credential is used.

With the knowledge obtained so far, do a search on Bing.com about “WMI.NET IManagedobject Current Security Token”, and it would reveal two KB articles from Microsoft (KB967622 and KB974893). They talked about exact the same issue faced here: When WMI.NET was using DCOM wrapper layer provided by .NET framework, the .NET framework is always trying to query IManagedObject interface for any COM object returned. However, .NET framework does not set the security blanket properly before issuing such a request. Without the security blanket being set, the request would use the default security context, i.e. the current security token of that process. This problem is annoying at least, because it would generate bunch of “Failure Audit” events on the remote machine, which causes security concerns, or in this WinPE case, it would actually disrupt the normal code execution.

Note: see [PWS00] chapter 9 for more information about SetSecurityBlanket

The same KB articles provides hotfix from Microsoft to fix this bug for earlier version of .NET. From .NET 4.0 onwards, this fix has already been integrated. After installing the hotfix for .NET 3.5 SP1, the same codes run smoothly against a WinPE machine. By observing the network traffic using package analyzer, it can be confirmed that the remote QueryInterface for IManagedObject requests are not sent over the wire anymore.  

Take Away:

Coding in managed environment has greatly simplified the developer’s life. While enjoying the mighty of modern programming techniques, it is also very helpful to understand the history of Microsoft technologies and the internal working of such technologies. To understand what happens under the hood, it is not only giving the developers the confidence when facing bugs, but also giving out examples how ones could design/implement to solve similar issues. Similarly, to know the path of how technologies evolve, it gives the developers the insight to those technologies’ rising and decline along with paradigm shift in this industry.

Package analyzers are efficient tools not only for traditional RPC based intranet applications, it is useful as well for newly Internet/Cloud based program. Having basic understanding of some popular network protocol, like TCP and HTTP, can greatly help to solve bugs. Therefore, It is well worthy to invest time to learn them. More inter-process communication debugging tips and expert advises can be found at [AWD07] Chapter 8.

References:

1. [PWS00] Programming Windows Security

2. [EC98] Essential COM

3. [MS-WMI] Windows Management Instrumentation Remote Protocol Specification

4. [AWD07] Advanced Windows Debugging

Special thanks to Gabriel and Pradeep’s help in solving this bug in the first place

For a small distributed project team, for example, a two-person’s team moonlighting for Windows Phone 7 applications/games, it is not necessarily a simply job to find a free or low-cost private/close source project hosting service: either they are too unreliable, or they come with too many limitations, and of course, the ads. Thus, besides getting a higher priced premiere hosting service, it seems the only alternative is to build a revision control system yourselves. This post introduces an easy way to setup such a versioning service, which can be used by every developers from the project team who doesn’t necessarily have direct connections together: The MercurialMesh (Mercurial + Windows Live Mesh).

Mercurial is a cross-platform, distributed revision control software released under GNU GPL [MERC]. It is adopted as the core part of this versioning service because of Mercurial’s distributed architecture.  For developers to share changes,  Windows Live Mesh Service is used to synchronize different distributed repositories among each others. Windows Live Mesh Service is a free Internet-based file synchronization service [WLM11], and it eliminates the needs of exposing an webserver on the Internet yourself for sharing changes, or manually sending changes to each others by emails.

How to Setup

1. Install Windows Live Mesh Service [WLM11]

2. Install Mercurial [MERC]

After both software have been stored, the only thing you need to do is to setup the master copy of your Mercurial repository and mark it as sync-able by Windows Live Mesh

3. To create master copy of your Mercurial repository, use command “hg init path_to_your_project_folder”. This command will initialize a new repository in that given path. If the given path doesn’t  exist, it creates one.  For example:

C:\Projects>hg init myproject_master

C:\Projects>cd myproject_master
C:\Projects\myproject_master>dir
Volume in drive C has no label.
Volume Serial Number is 486B-D984 Directory of C:\Projects\myproject_master

26/10/2010  03:30 Ale    <DIR>          .
26/10/2010  03:30 Ale    <DIR>          ..
26/10/2010  03:30 Ale    <DIR>          .hg
               0 File(s)              0 bytes
               3 Dir(s)  131,480,449,024 bytes free

4. Create a new Mercurial configuration file hgrc inside the .hg folder under your newly created repository. Open hgrc and specify a username you like for master repository by adding the following line:

Note: If you are using an existing folder with files inside, you can now add those files into the master repository by using command: “hg add” and then followed by “hg commit”.

Now the master repository is ready, and then the next step is to mark the master repository to be synchronized by Windows Live Mesh. After initial setup, the master repository must not be worked on directly, so this user account is used only for maintenance and recovery purposes.

5. Open Windows Live Mesh window, use “Sync a folder” option to add the folder where the master repository is to Windows Live Mesh. And then use “Shared with” option to give other developers the permissions to sync the same folder.

When you get here, the setup of your MercurialMesh versioing service is finished, You can start to use it

How to Use

First, every developer should accept the sharing invitation sent by Windows Live Mesh, so that every one will have a master repository synced on his/her machine.

Create enlistment:

This is to create a local working copy of the master repository. Use command: “hg clone master_copy local_copy”, for example,

C:\Projects>hg clone c:\Projects\myproject_master c:\Projects\myproject_local
updating to branch default
0 files updated, 0 files merged, 0 files removed, 0 files unresolved

Open hgrc inside the .hg folder of your newly created local repository, and specify a username for yourself adding the following line:

Note: Every developer must work on their own local repository, NOT on the master repository, otherwise, the revision history could be altered in an unwanted way

Getting the latest changes from others:

This is to update your local repository with the latest changes which have been submit to master repository by other developers 

1. (Optional) Use “hg incoming” to see what new changes are in the master repository, for example:

C:\Projects\myproject_local>hg incoming ..\myproject_master
comparing with ..\myproject_master
changeset:   0:40aa6d24fb24
tag:         tip
user:        maintainer <maintainer@greatteam.com>
date:        Tue Oct 26 15:59:31 2010 -0700
summary:     Init master repro with helloworld

2. Once you confirm that you need to get those changes, use “hg pull” to get those new changes, for example:

C:\Projects\myproject_local>hg pull ..\myproject_master
pulling from ..\myproject_master
requesting all changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files
(run 'hg update' to get a working copy)

3. Don’t forget to apply those changes using “hg update” to your local repository. for example:

C:\Projects\myproject_local>hg update
1 files updated, 0 files merged, 0 files removed, 0 files unresolved

Note: “hg update” is the command actually modify your local files with those changes. Without “hg update” those new changes are just kept inside the Mercurial’s cache files, and they are not visible to you directly

Submit your changes to others:

This is to submit your local changes into master repository, so other developer can received your changes later on if they uses “hg pull”

1. (Optional) Use “hg status” to confirm the list of files you intend to change, for example:

C:\Projects\myproject_local>hg status
M helloworld.cpp

2. Submit the change to your local repository first by using “hg commit”, for example,

C:\Projects\myproject_local>hg commit

3. (Optional) See how is your change looks like to master repro by using “hg outgoing” command, for example:

C:\Projects\myproject_local>hg outgoing ..\myproject_master
comparing with ..\myproject_master
searching for changes
changeset:   1:0cd5548b8bbe
tag:         tip
user:        Alice <alice@greatteam.com>
date:        Tue Oct 26 16:06:19 2010 -0700
summary:     Fix alignment

4. Make your local change into master repository by using “hg push” command, for example:

C:\Projects\myproject_local>hg push ..\myproject_master
pushing to ..\myproject_master
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files

Now your change is in the repository, and available to others who choose to “hg pull”

View the history:

See the revision history of your project, use “hg log” command, for example:

C:\Projects\myproject_local>hg log
changeset:   1:0cd5548b8bbe
tag:         tip
user:        Alice <alice@greatteam.com>
date:        Tue Oct 26 16:06:19 2010 -0700
summary:     Fix alignment

changeset:   0:40aa6d24fb24
user:        maintainer <maintainer@greatteam.com>
date:        Tue Oct 26 15:59:31 2010 -0700
summary:     Init master repro with helloworld

And more:

There are more much things you can do using Mercurial as you expected from any other distributed revision control system, such like merging and conflict resolving etc. See more usage at Mercurial site [MERC]. If you prefer GUI to command line window, TortoiseHg is good choice. And if you want even more integration with Visual Studio, then install VisualHg on top of TortoiseHg.

Also, you can enable Windows Live Mesh to sync your master repository with your Windows Live SkyDrive. That will keep a copy of your master repository in Microsoft’s cloud storage space. For that, you will give your repository the same availability as Microsoft Live SkyDrive promised 

Summary

MercurialMesh (Mercurial + Windows Live Mesh) is very straightforward to setup. Basically, Mercurial is used for revision control, and Windows Live Mesh is used to share changes among developers. It is easy for daily use too: “hg pull” + “hg update” to receive changes from others; “hg commit” + “hg push” to submit your changes. This system costs your nothing, and it doesn’t requires any additional infrastructure setup, such like DNS Name, firewall and web server etc. For a small size team with infrequent changes exchange, it just simply works.

However, the scalability of this setup yet to be tested. There is a considerable delay between when a new change being made to master repository and when Windows Live Mesh finishes the sync. 

References

1. [MERC] Mercurial

2. [WLM11] Windows Live Mesh 2011

Note: See more information about pack and alignment in [MSDN-PACK]. For more on false sharing see [WVC07] Chapter 8.2 and [PPP09] Chapter 1

In this case study, a heap bug is encountered due to incorrect use of the packing. The debugging technique demonstrated here includes some basic knowledge of Windows heap as described in last post and some understanding of data alignment in 64 bits Windows.

The Bug:

MyProgram is a 64-bit native application running under Windows 7, and it crashes consistently when it is launched. From the Event Viewer (Windows Logs/Application), the following error is logged:

Faulting application name: MyProgram.exe, version: 0.0.0.0, time stamp: 0x4b9d52d7
Faulting module name: ntdll.dll, version: 6.1.7600.16385, time stamp: 0x4a5be02b
Exception code: 0xc0000374
Fault offset: 0x00000000000c6cd2
Faulting process id: 0x1074
Faulting application start time: 0x01cac3bc0628969b
Faulting application path: d:\MyProgram\x64\Release\MyProgram.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 4485473b-2faf-11df-8d9f-00151736c5a7

The exception code 0xc0000374 means STATUS_HEAP_CORRUPTION. At this point, it is clear that this is a heap corruption bug, but it still requires some more debugging to find out the what actually corrupted the heap.

The Debugging:

As mentioned in last post, debugger could change the behavior of a heap related bug. Therefore, the –hd option is used during debugging so that the heap layout remains exactly the same as if the MyProgram.exe is running without debugger. In addition, the symbol path is set to Microsoft Public Symbol Server.

Note: –hd prevents the debugger to use the special heap which has a different layout than the default heap used by Windows.

The debugger outputs the following information when Windows Heap Manager detected the heap problem:

Critical error detected c0000374
(1364.dd4): Break instruction exception - code 80000003 (first chance)
ntdll!RtlUnhandledExceptionFilter+0x29f:
00000000`77bc6c9f cc              int     3

With symbols loaded, uses command !heap to show more information, only the relevant outputs are shown

0:003> !heap
Details:

Error address: 00000000048b8410
Heap handle: 00000000048b0000
Error type heap_failure_buffer_overrun (6)
Last known valid blocks: before - 00000000048b8320, after - 00000000048b8fc0
Stack trace:
                0000000077bca4a8: ntdll!RtlpAnalyzeHeapFailure+0x00000000000003a8
                0000000077b5ea71: ntdll!RtlpAllocateHeap+0x0000000000002176
                0000000077b529ac: ntdll!RtlAllocateHeap+0x000000000000016c
                0000000071a6c887: msvcr90!malloc+0x000000000000005b
                0000000071a6c987: msvcr90!operator new+0x000000000000001f
                000007fef2fdbf55: MyProvider!CGenericThreadPool::Initialize+0x0000000000000135
                000007fef2fd74c5: MyProvider!SpecialThreadPool::SpecialThreadPool+0x0000000000000075
                000007fef2fcb5cc: MyProvider!CPolicyProvider::CPolicyProvider+0x00000000000000ac
                000007fef2fd3e84: MyProvider!MyProviderInitialize+0x0000000000000034
                000007fef308e043: MyProgram!Initialize+0x00000000000002a3

As the output indicates, the heap issue is a buffer overrun, and the address where this happens is at 00000000048b8410. From the stack trace, it is obvious that overrun happens inside some constructor calls and Windows Heap Manager detected it when RtlAllocateHeap is called.

To get a close look at the error address and the heap block it belongs to:

0:003> !heap -i 00000000048b8410
Detailed information for block entry 00000000048b8410
Assumed heap       : 0x00000000048b0000 (Use !heap -i NewHeapHandle to change)
Header content     : 0x65010001 0x0000284A
Owning segment     : 0x00000000048b0000 (offset 0)
Block flags        : 0x0 (free )
Total block size   : 0x1 units (0x10 bytes)
Previous block size: 0xf units (0xf0 bytes)
Block CRC          : ERROR - current 65, expected 0
Free list entry    : OK
Previous block     : 0x00000000048b8320
Next block         : 0x00000000048b8420

It reports checksum doesn’t match the expected value, which means this heap block’s header has been damaged. As explained in in last post, Windows Heap Manager cannot detect the overrun right at the moment when the damage happens, instead, it is only able to tell certain overrun has happened when the damaged block is used later on. In other word, if an operation X overrun its heap block a, which damages the next heap block a+1, Windows Heap Manager cannot detect X’s misconduct, However, if later on, the heap block a+1 is used by other operation Y, then Windows Heap Manager will find out a+1 has been damaged, but of course operation Y has nothing to do with this damage.

Therefore, what happened in this case is that, the heap block 00000000048b8410 is the a+1 block,  the stack track shown above is the innocent operation Y.  The real culprit is the one who operate on the previous heap block 0x00000000048b8320. So, to get a better look at 0x00000000048b8320:

0:003> !heap -i 0x00000000048b8320
Detailed information for block entry 00000000048b8320
Assumed heap       : 0x00000000048b0000 (Use !heap -i NewHeapHandle to change)
Header content     : 0x7717D4D1 0x0B0028C5 (decoded : 0x0E01000F 0x0B000080)
Owning segment     : 0x00000000048b0000 (offset 0)
Block flags        : 0x1 (busy )
Total block size   : 0xf units (0xf0 bytes)
Requested size     : 0xe5 bytes (unused 0xb bytes)
Previous block size: 0x80 units (0x800 bytes)
Block CRC          : OK - 0xe 
Previous block     : 0x00000000048b7b20
Next block         : 0x00000000048b8410

0:003> !heap -x 0x00000000048b8320
HEAP 00000000048b0000 (Seg 00000000048b0000) At 00000000048b8410 Error: invalid block size

Entry             User              Heap              Segment               Size  PrevSize  Unused    Flags
-------------------------------------------------------------------------------------------------------------
00000000048b8320  00000000048b8330  00000000048b0000  00000000048b0000        f0       800         b  busy

It tells this block was assigned to an allocation request for size of 0xe5 bytes memory. And look back at the stack track from the first !heap command, there are quite a few constructors calls. It is reasonable to guess that this heap block was allocated for an object. To show the content of this heap block, just do d command at the user portion of this heap block at address 00000000048b8330, i.e. heap block header address + header size(0x10):

0:003> dq 00000000048b8330 
00000000`048b8330  000007fe`f2fc1848 00000000`0478ded0
00000000`048b8340  00000000`00000000 00000000`00000000
00000000`048b8350  00000000`00000000 00000000`00000000
00000000`048b8360  00000000`00000000 00000000`00000000
00000000`048b8370  00000000`00000000 00000000`00000000
00000000`048b8380  00000000`00000000 00000000`00000000
00000000`048b8390  00000000`00000000 00000000`00000000
00000000`048b83a0  000007fe`f2fc1880 0000000c`00000000

If the assumption(this block is allocated for an object) is right, then the first 8 bytes should be pointing to the virtual function table of that object. To verify this, do ln command at the address pointed by the first 8 bytes

0:003> ln 000007fe`f2fc1848
(000007fe`f2fc1848)   MyProvider!CPolicyProvider::`vftable' Exact matches:

It is indeed an object, and this object is an instance of type CPolicyProvider. It is clear now that the heap block 0x00000000048b8320 was allocated as a response to  the constructor call of class CPolicyProvider.

Note: For more information about C++ Language Memory Model, see [ICOM96]

Note: Another easy way to find out what functions has operated on certain heap block is to use the stack trace. By specifying +ust flag with gflags.exe tool, it enables the user mode stack trace. After it is enabled, !heap –i command will automatically includes stack trace whenever available. This is a very useful feature.

To know the size of this type, do

0:003> ?? sizeof (OSDPxeProvider!OSD::CPolicyProvider)
unsigned int64 0xe5

It returns 0xe5 bytes, and it seems consistent with the size from heap block information. However, when trying to display the detailed type information using dt command, some inconsistency has been noticed (only relevant output has been shown below) :

0:003> dt -r MyProvider!CPolicyProvider
   +0x000 __VFN_table : Ptr64
   =000007fe`f2fec5d0 CPolicyProvider::s_theInstance : Ptr64 CPolicyProvider
      +0x000 __VFN_table : Ptr64
      =000007fe`f2fec5d0 CPolicyProvider::s_theInstance : Ptr64 CPolicyProvider
         +0x000 __VFN_table : Ptr64
         =000007fe`f2fec5d0 CPolicyProvider::s_theInstance : Ptr64 CPolicyProvider
         +0x008 m_hProvider      : Ptr64 Void
         +0x010 configPtr        : shared_any<Config *,close_delete,null_t,enum Unique>
         +0x020 m_bootCounter    : shared_ptr<IPerformanceCounter>
         +0x030 m_abortCounter   : shared_ptr<IPerformanceCounter>
         +0x040 m_passCounter    : shared_ptr<IPerformanceCounter>
         +0x050 m_requestTotalCounter : shared_ptr<IPerformanceCounter>
         +0x060 m_requestTimedOutCounter : shared_ptr<IPerformanceCounter>
         +0x070 m_threadPool     : SpecialThreadPool
      +0x008 m_hProvider      : Ptr64 Void

   +0x070 m_threadPool     : SpecialThreadPool
      +0x000 __VFN_table : Ptr64
      +0x008 m_bIsInitialized : Bool
      +0x009 m_bInitializeCOM : Bool
      +0x00c m_dwCurrentNumberOfThreads : Uint4B

      ... Other fields are not shown here

      +0x078 m_bIsCsThreadListInitialized : Bool
      +0x079 m_bIsShuttingDown : Bool
      +0x07a m_bAutoCreateNewThreads : Bool
      =000007fe`f2fc0000 THREAD_COUNT     : Uint4B

The last field of CPolicyProvider is type of SpecialThreadPool, size of SpecialThreadPool is at least 0x7a bytes(depends on padding). So, the entire size of CPolicyProvider should be at least 0x70 + 0x7a = 0xea. In other words, the instance memory layout has more bytes than the actual memory allocated (0xea > 0xe5). This definitely causes heap buffer overrun if there is a write operation to the last field of CPolicyProvider.

Looking into the source code, type SpecialThreadPool is inherited from type CGenericThreadPool, and it just has some more additional static field and functions than the parent class. Hence, the memory layout of SpecialThreadPool should be exact the same as CGenericThreadPool. However, the size of CGenericThreadPool reported by debugger is different from the size of SpecialThreadPool reported :

0:003> ?? sizeof (SpecialThreadPool)
unsigned int64 0x75

To understand where has gone wrong, the memory layout of SpecialThreadPool needs to be examined more closely:

0:003> dt OSD::PXERequestThreadPool
OSDPxeProvider!OSD::PXERequestThreadPool
   +0x000 __VFN_table : Ptr64
   +0x008 m_bIsInitialized : Bool
   +0x009 m_bInitializeCOM : Bool
   +0x00c m_dwCurrentNumberOfThreads : Uint4B
   +0x010 m_dwMaximumNumberOfThreads : Uint4B
   +0x018 m_hCompletionPort : Ptr64 Void
   +0x020 m_hMonitoringThread : Ptr64 Void
   +0x028 m_lastActivationTime : Int8B
   +0x030 m_lNumberOfThreadsWaiting : Int4B
   +0x034 m_dwCompletionThreadTimeout : Uint4B
   +0x038 m_spThreadContextArray : Ptr64 CThreadContext
   +0x040 m_listActivationTime : _LIST_ENTRY
   +0x050 m_csThreadList   : _RTL_CRITICAL_SECTION
   +0x078 m_bIsCsThreadListInitialized : Bool
   +0x079 m_bIsShuttingDown : Bool
   +0x07a m_bAutoCreateNewThreads : Bool

where the layout of _LIST_ENTRY and _RTL_CRITICAL_SECTION are:

And more intuitively, the memory layout of SpecialThreadPool could be represented as following block diagram (the unit in the brackets is byte):

As the diagram shows, the entire size of SpecialThreadPool should be 0x80 (0x78 + 0x8) bytes. 0xb (11) bytes out of 0x80 are just padding. Furthermore, it is clear that the packing size of SpecialThreadPool is 8 because every data field of SpecialThreadPool is aligned at multiple of 8 bytes.

Compare this value with the size 0x75 reported by debugger, the difference is just 0xb (0x80 - 0x75). It is the same as the number of  padding bytes. So it seems that the packing reported by debugger is just 1 (no padding), not 8.

At this point, with all the information uncovered by using debugger, it is easy to locate the root cause by just doing a simple code review. It turns out that type SpecialThreadPool and type CGenericThreadPool were using different packing size in spite of having an inheritance relationship.  In one of the header files that SpecialThreadPool includes, there is a line “#pragma pack(1)” which tells the complier to compile SpecialThreadPool using 1 as pack size. All the other classes are compiled with default pack size 8. This mistake in code cause the compiler to generate inconsistent code regarding the size of SpecialThreadPool type, which in turn results the heap buffer overrun in runtime.

Take Away:

The techniques shown in this case study is based on some fundamental understanding of how the memory is used in Windows. As a software developer, it is important to understand data structure memory layout and language object model, although most time the complier has abstracted this information away. These knowledge can comes handy while debugging memory relative debugs.

Besides packing, there are many other aspects of compiler can be configured and they would affect the final shape of the generated codes. When making setting changes in the code, it is recommended to localize its effect by saving and restore the original value. For example, instead of just throw “#pragma pack(1)” into some header file, it is better to use the following code:

#pragma pack(push)  //to save current packing size 

#pragma pack(1)     //enable special packing, no padding

...                 //your data structure here

#pragma pack(pop)   //restore the packing size to original value

This way, other files will not accidentally inherit this special packing setting when including this header file. On the other hand, if the setting changes are meant to be global, it is better to make those changes in complier level (like complier switchers), not in the code itself.

References:

1. [MSDN-PACK] Pack

2. [WVC07] Windows via C/C++, Fifth Edition Chapter 8.2

3. [PPP09] Principles of Parallel Programming Chapter 1

4. [ICOM96] Inside the C++ Object Model

Special thanks to Gabriel and Daniel’s help in finding this bug in the first place

Note: A major revamp has done to Windows Heap Manager in Windows Vista and continued in Windows 7. The purpose of those changes is to improve Heap Manager’s performance, security and error resilience and monitoring. There are many great books and articles online about the details of latest Windows Heap Manager, such like [AWD07] and [WININT09]. 

Standard Heap Monitoring by Windows Heap Manager

Windows Heap Manager provides some basic heap monitoring functionalities for every programs using heap.  It performs limited heap validation in many heap operations. The normal heap block layout managed by Windows Heap Manager looks like this:

As you can imagine, if the memory-write overruns the user allocated portion, then the next heap block’s header will be overwritten. Windows Heap Manager validates the heap header every time when a heap block is used or freed. If it finds the header has been damaged, then it announces a heap overrun has happened. Because of this detection mechanism used by Windows Heap Manager, it is not able to catch the overrun at the time it happens, but only detect it afterwards when the corrupted block is involved in some other operations.

Another important aspect here is that the size of user allocation space is always a multiple of 16 bytes. This means, even if the size of memory requested by user is just 1 byte, the Windows Heap manager will still allocate 16 bytes for that request, in which 15 bytes are marked as unused. In this case, if the overrun happens within the unused user allocation, then obviously the next heap block’s header will not be damaged. Thus, the Windows Heap Manager is not able to detect this kind of overrun. 

Example 1:

The following code request a 1 byte memory using malloc function of Microsoft C Runtime Library (Release Build). malloc internally calls the Windows Heap Manager APIs to full fill the request. Once the memory is obtained, it initializes the 1 byte value to ‘a’, and after that, it overruns the heap with a wild assignment. 

Debugger was attached once the program starts, and the following debugger output shown is after the malloc call:

Note: Do not use the debugger to launch the example program, the reason will be explained shortly. Also, the debugger needs to able to find the correct symbols for some of system DLLs, and the easiest way is to add Microsoft Public Symbol Server to the sympath

example1!wmain+0x91:
00000001`3ff010c1 ff1571100000    call    qword ptr [example1!_imp_malloc (00000001`3ff02138)] ds:00000001`3ff02138={MSVCR100!malloc (00000000`510c89ec)}
0:000> p
rax=00000000003897b0 rbx=00000000003ae590 rcx=0000000000000003
rdx=00000000003897b0 rsi=00000000003ae570 rdi=00000000003ae5b0
rip=000000013fb410c7 rsp=000000000030fa40 rbp=0000000000000000
r8=0000000000000000  r9=0000000000000000 r10=0000000000000080
r11=0000000000000000 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl nz na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206
example1!wmain+0x97:
00000001`3ff010c7 b910000000      mov     ecx,18h

In Windows 64 bit calling convention, the rax stores the return address of malloc function, i.e. address to the memory requested.

Note: For the completed guide of calling conversion, please see [FOGA10] 

To find more information about the heap block the returned address belongs to, use !heap –x command on the address in rax. it will find the heap block contains that particular address:

0:000> !heap -x 00000000003897b0
Entry             User              Heap              Segment               Size  PrevSize  Unused    Flags
-------------------------------------------------------------------------------------------------------------
00000000003897a0  00000000003897b0  0000000000380000  0000000000380000        20      1010        1f  busy

So we can see now, this heap block starting at 00000000003897a0 with its header, then followed by the user allocation portion from 00000000003897b0, which is also the result address in rax. And the size of this block is 20 bytes and only 1 bytes is used so there are 1f bytes unused. We will see in a bit how those numbers come around. And the block is marked as busy because it is allocated.

We can even get much more information about the heap block and its neighboring heap blocks by using !heap –i command on the block entry address:

0:000> !heap -i 00000000003897a0 
Detailed information for block entry 00000000003897a0
Assumed heap       : 0x0000000000380000 (Use !heap -i NewHeapHandle to change)
Header content     : 0x791E4849 0x1F00F699 (decoded : 0x03010002 0x1F000101)
Owning segment     : 0x0000000000380000 (offset 0)
Block flags        : 0x1 (busy )
Total block size   : 0x2 units (0x20 bytes)
Requested size     : 0x1 bytes (unused 0x1f bytes)
Previous block size: 0x101 units (0x1010 bytes)
Block CRC          : OK - 0x3 
Previous block     : 0x0000000000388790
Next block         : 0x00000000003897c0

It shows additional stuff like header content and next heap block (at address 0x00000000003897c0).  In order to get a better view of the heap block’s memory layout, let’s do a direct memory dump at the beginning of the heap block:

0:000> dc 00000000003897a0 
00000000`003897a0  00000000 00000000 791e4849 1f00f699  ........IH.y....
00000000`003897b0  003ae5d0 00000000 00380158 00000000  ..:.....X.8.....
00000000`003897c0  00000000 00000000 fa1f48cb 0000f79a  .........H......
00000000`003897d0  003ae5d0 00000000 00380158 00000000  ..:.....X.8.....

16 bytes value start from address 00000000003897a0 is the head header. You can see the same memory content 791e4849 1f00f699 is shown in the result of !heap –i command above as header content. After the header, it follows with 16 bytes user allocation space (starting at 00000000`003897b0). Although only 1 byte is requested by the user, the Windows Heap Manager still reserves 16 bytes memory space for that request. After the user allocation space, the next memory address 00000000`003897c0 is actually the address of next heap block (the 16 bytes value from 00000000`003897c0 contains the next block’s header). So, we can do the following math:

• Total block size = size of header + size of user allocation = 0x20 (32) bytes
• User allocation size = 0x10 (16) bytes, in which, 1 bytes is used (requested), 0x20 – 1 = 0x1F (31) bytes are unused. The unused bytes actually include the unused part of user allocation plus the block header which is not accessible to the user.

Let’s continue with the code execution,  a[0] = 'a'; this is a valid heap usage because it matches the 1 byte requested space. After this assignment, the memory looks like this:

So far so good. However, the upcoming assignment, a[24] = 'a';, is totally out of line. It is way beyond the requested memory size. After executing this wild code, the memory looks like this now:

As you can see,  the last assignment overrides the header value of next heap block. It broke the coherence of heap structures, and later on if the damaged heap block is involved in some other memory operation, the Windows Heap Manager will detect this error. The following error occurs once the program is allowed to continue execution:

0:000> g
Critical error detected c0000374
(c78.19c8): Break instruction exception - code 80000003 (first chance)
ntdll!RtlReportCriticalFailure+0x2f:
00000000`77a26c9f cc              int     3

0:000> !heap
Details:

Error address: 00000000003897c0
Heap handle: 0000000000380000
Error type heap_failure_buffer_overrun (6)
Parameter 1: 000000000000000a
Last known valid blocks: before - 00000000003897a0, after - 0000000000389fc0
Stack trace:
                0000000077a2a4a8: ntdll!RtlpAnalyzeHeapFailure+0x00000000000003a8
                00000000779bea71: ntdll!RtlpAllocateHeap+0x0000000000002176
                00000000779b29ac: ntdll!RtlAllocateHeap+0x000000000000016c
                00000000779936ba: ntdll!RtlpAllocateUserBlock+0x0000000000000145
                0000000077992b5b: ntdll!RtlpLowFragHeapAllocFromContext+0x00000000000004e7
                00000000779b1c58: ntdll!RtlAllocateHeap+0x00000000000000e4
                0000000078d48a47: MSVCR100!malloc+0x000000000000005b
                000000013fb410dc: example1!wmain+0x00000000000000ac
                000000013fb4139e: example1!__tmainCRTStartup+0x000000000000011a
                000000007785f56d: kernel32!BaseThreadInitThunk+0x000000000000000d
                0000000077993281: ntdll!RtlUserThreadStart+0x000000000000001d
Index   Address  Name      Debugging options enabled
  1:   00080000               
  2:   00010000               
  3:   00020000               
  4:   00380000

Therefore, the heap manager found this heap overrun error later on down the road, while it was trying to use the damaged heap block (00000000003897c0) to full fill another memory allocation request.

Note: error code 0xc0000374 means  STATUS_HEAP_CORRUPTION. It is a new error code added in Windows Vista. It is defined in header file ntstatus.h included in Windows SDK. The completed guild of Windows Error Code can be found at [MS-ERREF].

However, if we change the last assignment from  a[24] = 'a'; to a[1] = 'a';, What then? It is still a heap overrun error because a[1] exceeds the 1 bytes limit as well, although it is not as wild as a[24].  Can the Windows Heap Manager still catch this error?

0:000> dc 00000000003897a0 
00000000`003897a0  00000000 00000000 791e4849 1f00f699  ........IH.y....
00000000`003897b0  003a6161 00000000 00380158 00000000 aa:.....X.8.....
00000000`003897c0  00000000 00000000 fa1f48cb 0000f79a  .........H......
00000000`003897d0  003ae5d0 00000000 00380158 00000000  ..:.....X.8.....

The answer is no. Windows Heap Manager cannot detect this variant of overrun error because the overrun is within the unused user allocation of heap block. As shown above, the next heap block’s header is still intact and there is no way for Windows Heap Manager to notice any problems. For this simple example, it got lucky and it can continue execution to exit without any more problems, but for many other applications, this kind of heap overrun just like a time bomb waiting there to cause bigger problems.

Is there a way to catch this kind of just off by 1 overrun?

Special Debug Heap Monitoring

Instead of using the standard heap, Windows Heap Manager uses a special debug heap for the processes that launched by debugger. This debug heap has a different heap block layout:

Note: You can force a debugger launched process to use the standard heap instead of the special debug heap by setting _NO_DEBUG_HEAP environment variable to 1, or the -hd debugger command-line option. By just attaching the debugger to an existing process, it will NOT change from the standard heap to the debugger heap. That’s why the debugger must be attached after the example program started in Example 1.

Comparing to the basic heap block created by standard heap, the debug heap added more data after user allocation portion of each heap block. First data after user allocation is the Tail Checking Pattern. The pattern is a 16 bytes of ab with potential padding which aligns on 16-byte boundary. After that, it is followed by heap extra data (it is 16  bytes or more depends on padding). The Heap Manager will validate the Tail Checking Pattern on release of heap block to check if it has been changed by any heap overruns.

In the last example, the heap overrun within unused user heap space has slipped through the detection of Windows Heap Manager, but it will be caught by this special debugger heap because the Tail Checking Pattern covers unused portion of user space as well. The detection is still not in real time: The overrun can only be found when that heap block is released, not at the time it actually happened.

This detection mechanism works well if the overrun is a continuous one from user allocated space. However, if the memory-write jumps over the Tail Checking Pattern and lands before next head header (i.e. writes in the padding area or heap extra area), then the Tail Checking Pattern is left unchanged by this overrun. Therefore, for this kind of overrun, it cannot be detected by using the special debug heap.

Example 2:

The following code request a 24 byte memory using malloc function. After that, it tries to perform some heap operation to cause a heap overrun error. This time, we use debugger to launch this program.

The memory layout of the heap block returned by malloc looks like this:

0:000> !heap -i 0000000000234bf0 
Detailed information for block entry 0000000000234bf0
Assumed heap       : 0x0000000000230000 (Use !heap -i NewHeapHandle to change)
Header content     : 0x60B63EB2 0x380068BC (decoded : 0x02070005 0x38000103)
Owning segment     : 0x0000000000230000 (offset 0)
Block flags        : 0x7 (busy extra fill )
Total block size   : 0x5 units (0x50 bytes)
Requested size     : 0x18 bytes (unused 0x38 bytes)
Previous block size: 0x103 units (0x1030 bytes)
Block CRC          : OK - 0x2 
Previous block     : 0x0000000000233bc0
Next block         : 0x0000000000234c40

0:000> dc 0000000000234bf0 
00000000`00234bf0  feeefeee feeefeee 60b63eb2 380068bc  .........>.`.h.8
00000000`00234c00  baadf00d baadf00d baadf00d baadf00d  ................
00000000`00234c10  baadf00d baadf00d abababab abababab  ................
00000000`00234c20  abababab abababab feeefeee feeefeee  ................
00000000`00234c30  00000000 00000000 00000000 00000000  ................
00000000`00234c40  feeefeee feeefeee b3b53e62 000069ba  ........b>...i..
00000000`00234c50  00237ee0 00000000 00230158 00000000  .~#.....X.#.....
00000000`00234c60  feeefeee feeefeee feeefeee feeefeee  ................

The heap header is still 16 bytes starting from address 00000000`00234bf0. Then it is followed by 32 bytes user allocation starting from 00000000`00234c00. The heap manager always allocated in multiple of 16 bytes, even if the user only requested 24 bytes. Next, the debug heap put 16-byte Tail Checking Pattern immediately after the last byte of used portion of user allocation space, which is from 00000000`00234c18 to 00000000`00234c27. In order to get aligned at 16 bytes boundary, 8 bytes of padding is added after the Tail Checking Pattern. Lastly, there is another 16 bytes of heap extra starting from 00000000`00234c30. The next heap block starts next 16 bytes boundary (at address 00000000`00234c40).

• Total block size = size of header + size of user allocation + Tail Checking Pattern + Padding + Heap Extra  = 0x50 (80) bytes
• User allocation size = requested size + extra filling = 0x40 (64) bytes, in which, 0x18 (24) bytes is used (requested), 0x50 – 0x18 = 0x38 (56) bytes are unused.

Let’s continue executing the next assignment statement a[24] = 'a';. This assignment overruns the allocated heap by 1 byte, hence, it modified the first byte of Tail Checking Pattern. The memory layout after the assignment looks like this:

We have created a heap overrun just by 1 byte, just like we did in the last part of example 1. It is well within the unused user allocation space. Let the program to continue:

0:000> g
HEAP: Heap block at 0000000000234BF0 modified at 0000000000234C18 past requested size of 18
(2a54.1af0): Break instruction exception - code 80000003 (first chance)
ntdll!RtlpBreakPointHeap+0x21:
00000000`77a13741 cc              int     3
0:000> k
Child-SP          RetAddr           Call Site
00000000`001cf678 00000000`77a225a1 ntdll!RtlpBreakPointHeap+0x21
00000000`001cf680 00000000`779c887e ntdll!RtlpCheckBusyBlockTail+0x211
00000000`001cf6c0 00000000`77a2c499 ntdll!RtlpValidateHeapEntry+0x45f0e
00000000`001cf700 00000000`779cd0fe ntdll!RtlDebugFreeHeap+0xb9
00000000`001cf760 00000000`779b2075 ntdll!RtlpFreeHeap+0x1b07e
00000000`001cfaa0 00000000`77862c7a ntdll!RtlFreeHeap+0x1a2
00000000`001cfb20 00000000`78d68ac4 kernel32!HeapFree+0xa
00000000`001cfb50 00000001`3fd31152 MSVCR100!free+0x1c
00000000`001cfb80 00000001`3fd3138e example2!wmain+0x122
00000000`001cfbb0 00000000`7785f56d example2!__tmainCRTStartup+0x11a
00000000`001cfbe0 00000000`77993281 kernel32!BaseThreadInitThunk+0xd
00000000`001cfc10 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

As you see, The debug heap has detected this off-by-1 overrun. The call stack shows that the error is found when heap is being released. Hence, the debug heap is able to catch overruns which are within the unused user allocation area. By the way, this detection is also postmortem in nature as you might already notice.

Let’s try something else now. If we change the assignment from a[24] = 'a'; to a[40] = 'a';, then the memory-write skips over the Tail Checking Pattern and lands on the padding area, as shown on the memory layout below:

In this case, the Tail Checking Pattern is unchanged by the overrun, Thus, it cannot be detected by the debug heap. If the program is allowed to continuous, it will finish without rising any problems.

More interestingly, if the debug heap is not used (i.e. the program is not spawned by the debugger), this overrun, a[40] = 'a';, would have landed on the heap header of next heap block. And for that, the Windows Heap Manager could detect it. This should give you a basic idea on why having the debugger could make some of the heap corruption bug to disappear.

PageHeap Monitoring

So far we have seen the default Windows Heap Manager monitoring and detection ability, as well as the the ability of the special debug heap. However, both of them have certain limitations on detecting heap overruns.

To overcome those limitations, Microsoft has a tool called PageHeap which provides better heap monitoring and detection features than last two. As time of this writing, the PageHeap tool has been integrated into Gflags tool as well as AppVerifier tool. Its usage are well documented in MSDN and widely discussed in the community. Both books [AWD07] and [WININT09] have chapters on it too.

Using either Gflags or AppVerifier, a Full Page Heap layout can be enabled for each heap block as shown below

Diagram source: [MSDN-TSPHB]

And the actually memory layout for such a heap block looks like this (the debug output is showing a busy heap block): 

0:000> dc 3e50ff0-40
00000000`03e50fb0  abcdbbbb 00000000 03d21000 00000000  ................
00000000`03e50fc0  00000001 00000000 00001000 00000000  ................
00000000`03e50fd0  00000000 00000000 00000000 00000000  ................
00000000`03e50fe0  00227e70 00000000 00000000 dcbabbbb  p~".............
00000000`03e50ff0  d0d0d0c0 d0d0d0d0 d0d0d0d0 d0d0d0d0  ................
00000000`03e51000  ???????? ???????? ???????? ????????  ????????????????
00000000`03e51010  ???????? ???????? ???????? ????????  ????????????????
00000000`03e51020  ???????? ???????? ???????? ????????  ????????????????

The green part is the Prefix Pattern for that heap block starting with abcdbbbb and ending with dcbabbbb. Next, the purple part is the user allocation. There is 1 byte of c0 and 15 bytes of d0 in the user allocation, and that means the user only requested 1 byte of memory. The rest 15 bytes of d0 are the suffix pattern. Lastly, it is the non-accessible page (the red part) after the last d0 .

Let’s assume char* a is pointed to the valid address of that 1 byte allocation. A heap overrun, such as a memory write to the address at a[24], will immediately cause a access violation exception:

(297c.62c): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
example1!wmain+0x19:
00000001`3f9a1019 c6401862        mov     byte ptr [rax+18h],62h ds:00000000`03e51008=??

You can see the access violation happens at exact the same time the mov instruction trying to access an address in the red part of the memory layout. So first time ever, the heap overrun is caught right at when it happens.

But what if the heap overrun is within the suffix pattern, such as a off-by-1 overrun (a memory write to address at a[1])? In that case, the heap overrun will be detected only when that block is released. That’s when the suffix pattern validation is done, just like the debug heap. If the program is under debugger, the following error is shown when the program is trying to free(a)

===========================================================
VERIFIER STOP 000000000000000F: pid 0x2898: corrupted suffix pattern

    0000000003F31000 : Heap handle
    0000000004060FF0 : Heap block
    0000000000000001 : Block size
    0000000004060FF1 : corruption address

Note: When PageHeap is enabled, the Heap Stack Trace is also enable for every heap block. The Stack Trace records the entire call stack of last heap operation done to that heap block. For example, the !heap –p –a memory_address command finds the heap block contains the specific memory address, and it also displays the stack trace:

0:000> !heap -p -a 0000000003E50FF0
    address 0000000003e50ff0 found in
    _DPH_HEAP_ROOT @ 3d21000
    in busy allocation (  DPH_HEAP_BLOCK:         UserAddr         UserSize -         VirtAddr         VirtSize)
                                 3d24888:          3e50ff0                1 -          3e50000             2000
    000007fef3198513 verifier!AVrfDebugPageHeapAllocate+0x000000000000026f
    0000000077a39901 ntdll!RtlDebugAllocateHeap+0x0000000000000031
    00000000779fe6ea ntdll!RtlpAllocateHeap+0x000000000004bd2a
    00000000779b29ac ntdll!RtlAllocateHeap+0x000000000000016c
    000000006e5f8a47 MSVCR100!malloc+0x000000000000005b
    000000013fff1011 testprogram!wmain+0x0000000000000011
    000000013fff1262 testprogram!__tmainCRTStartup+0x000000000000011a
    000000007785f56d kernel32!BaseThreadInitThunk+0x000000000000000d
    0000000077993281 ntdll!RtlUserThreadStart+0x000000000000001d

From the stack trace, you can see the last operation is the allocation of that heap block. 

Gflags and AppVerifier also provide a slimmed down version of Full Page Heap called Normal Page Heap. The difference is that Normal Page Heap doesn’t have the non-accessible page appended after the suffix pattern. Therefore, using Normal Page Heap, it is not possible to catch heap overrun instantly.

Microsoft Visual Studio C Runtime (Debug Version)

The debug version of Microsoft Visual Studio C Runtime library (msvcrtXXd) also has its own Heap monitoring feature. The mechanism it uses is similar to PageHeap.

Takeaways

Windows Heap Manager provides different levels of heap monitoring in junction with debugger, gflags or AppVerifier. With PageHeap, a few overruns can be caught at the exact moment when it happens, but in most of time, the detection is only postmortem (i.e. the overrun is detected after it has happened). This nature of heap overrun make it become one of most difficult bug to solve. [AWD07] has more information on how to deal with it and how to avoid the overrun at the first place.

Furthermore, having debugger might trigger Windows heap Manager to use debug heap, which has different heap block layout. This is a side effect of debugger. As demonstrated above, if the debugger is in the picture, this side effect could actually delays the exposure of certain heap corruption bugs and even prevents them from happening at all. Therefore, be careful, and always be aware of the implication of the environment the program is running under.

References:

1. [AWD07] Advanced Windows Debugging Chapter 6, Chapter 15

2. [WININT09] Windows Internals, Fifth Edition Chapter 9

3. [MS-ERREF] Windows Error Codes

4. [FOGA10] Calling conventions for different C++ compilers and operating systems

5. [MSDN-DTFW] Debugging Tools For Windows - !heap

6. [MSDN-TSPHB] The Structure of a Page Heap Block

1. Services Control Manager must start the service with the debugger attached.
2. Starting from Windows Vista/2008, all windows services are running under a separated session 0 (which has a different WindowStations/Desktop than the normal user interactive desktop).
3. The visual studio debugger cannot be attached in mixed mode (both native and managed) if the code is written prior to .NET 4.0

One solution for this problem is to use WinDBG debugger with SOS extension, and configure the “Image File Execution” options to start the service under debugger server automatically. The following case study demonstrates this technique.

The Bug:

ManagedService is a winidows service, its binary managedservice.exe is implemented using C#. The following error occurs when ManagedService is trying to start:

C:\Users\administrator>net start ManagedService
The ManagedService service is starting...
The ManagedService service could not be started.

A system error has occurred.

System error 1067 has occurred.

The process terminated unexpectedly.

In Event Viewer (Windows Logs/Application), the actual CLR exception and call stack are recorded:

System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
   at System.Reflection.Module._GetTypesInternal(StackCrawlMark& stackMark)
   at System.Reflection.Assembly.GetTypes()
   at MyCompany.MyProduct.IndigoSerializableObject.InitializeKnownTypesCache(List`1 assembliesToExamine)
   at MyCompany.MyProduct.ManagedService.OnStart(String[] args)

This gives a better insight to the problem: This service was probably going to host some WCF services and maybe it was preloading some managed types to improve performance, and the CLR loader encountered an error while loading them. However, assuming there are quite few assemblies and each has a few types, this information alone is not enough to tell which exact type causes the error. What is really needed is the content of LoaderExceptions property so that we can pinpoint the guilty type.

Note: Fusion Log won’t help in this case, because it is a type loading exception not an assembly loading exception.

The Debugging:

First, We need to configure the service so that when it starts the debugger has already been attached. For that, we can use the “Image File Execution” option by specifying the “Debugger” entry. There are numerous blogs and technical articles detailed the usage of this. Basically what it does is to allow the debugger specified in the “Debugger” entry to launch first, and then the service process is in turn spawned by that debugger.

So which debugger we should use for “Image File Execution” option? Because all windows services are running under session 0 (Windows Vista/2008 above), whatever debugger we choose will also be spawned in session 0. If that debugger put out any GUIs, they will not be directly visible to the users (the “Interactive Services Detection” dialog box would pop up).  In other words, any interactive debugger won’t be a good choice. Thus, what we actually need is to start a window-less debugger server instead, and then from the current user session, we can use an interactive debugger to connect to that server.

To setup the debugger server on service startup, we can use either gflags.exe or regedit.exe or other tools to configure the “Debugger” entry of  “Image file Execution” option for managedservice.exe in the registry as:

%windbginstallpath%\cdb.exe –server tcp:port=1234 –c “g;”

%windbginstallpath% is where the installation location of Debugging Tools for Windows, and the cdb.exe is the debugger which doesn’t create any new windows. The –server switch activate the debugger as debugger server, and tcp:port=1234 tells the debugger server to use TCP protocol and listening at port 1234 (you can use other protocol or port too, see debugger documentation).  The last –c ”g;” switch tells the debugger to let managedservice.exe to continue to run upon attachment, and this is because we know the bug is unhandled exception and the exception will cause the debugger to break automatically.

Note: For investigating other kinds of bugs, different debugger commands can be appended after “g;” to enable debugger to act in various situations, for example, “g;sxe ld some.dll”, will break into the debugger automatically when some.dll is loaded into process space.  

Now, we restart the ManagedService again by typing

net start ManagedService

and then we start a new interactive debugger client ntsd.exe (cdb.exe or windbg.exe also works) to connect to the debugger server:

%windbginstallpath%\ntsd.exe –remote tcp:server=localhost,port=1234 

Here, the –remote switch activate the ntsd.exe as debug client, and tcp:server=localhost,port=1234 tells the ntsd.exe to connect to the debugger server locally at port 1234 using TCP protocol.

Note: the Service Control manager has a default 30 seconds timeout period within which the service must have to start, otherwise it would ends the process and the attached debugger server. The timeout value can be adjusted by adding new DWORD value ServicesPipeTimeout under the following registry subkey:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control

Alternative, you can save a crash dump using .dump -ma command, and then debug the dump file in post mortem instead . –ma flag gives a memory dump with everything and it is required by SOS extension.

Once the ntsd.exe attached, we can see that the debugger has already break in because of the unhandled System.Reflection.ReflectionTypeLoadException. To save space, I only show the relevant parts of output from ntsd.exe window:

ModLoad: 000007fe`fac00000 000007fe`fac2d000   C:\Windows\system32\ntmarta.dll
ModLoad: 000007fe`fef50000 000007fe`fefa0000   C:\Windows\system32\WLDAP32.dll
(11f8.f58): CLR exception - code e0434f4d (first chance)
CLR exception type: System.Reflection.ReflectionTypeLoadException
    "Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for m
ore information." 
00000000`7710017a c3              ret
DCMANAGER\administrator (tcp 127.0.0.1:50908) connected at Sat Feb 20 01:41:19 2010
0:007>

It’s time to take a closer look at the ReflectionTypeLoadException. To do that, we first need to load the SOS Debugger Extension. In the ntsd.exe window, type:

.load %windir%\microsoft.net\<architecture>\<version>\sos.dll

In my case, sos.dll is at c:\windows\Microsoft.NET\Framework64\v2.0.50727\. After SOS is loaded, we can use !dso (!dumpstackobject) to see all managed objects found within the bounds of the current stack, which will include our exception:

0:007>!dso

We can easily see that 00000000027c9be0 is the object address of System.Reflection.ReflectionTypeLoadException. To see inside of that exception, we can use !do (!dumpobject) command:

0:007>!do 00000000027c9be0

Name: System.Reflection.ReflectionTypeLoadException
MethodTable: 000007fef9730fc0
EEClass: 000007fef8c3bd18
Size: 152(0x98) bytes
(C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fef8f3ec90  40000b5        8        System.String  0 instance 00000000027ca108 _className
000007fef8f3d038  40000b6       10 ...ection.MethodBase  0 instance 000000000281de00 _exceptionMethod
000007fef8f3ec90  40000b7       18        System.String  0 instance 0000000000000000 _exceptionMethodString
000007fef8f3ec90  40000b8       20        System.String  0 instance 00000000027c9cd0 _message
000007fef8f37b98  40000b9       28 ...tions.IDictionary  0 instance 0000000000000000 _data
000007fef8f3ef58  40000ba       30     System.Exception  0 instance 0000000000000000 _innerException

000007fef8f3ec90  40000bb       38        System.String  0 instance 0000000000000000 _helpURL
000007fef8f3e580  40000bc       40        System.Object  0 instance 00000000027ca4c8 _stackTrace
000007fef8f3ec90  40000bd       48        System.String  0 instance 0000000000000000 _stackTraceString
000007fef8f3ec90  40000be       50        System.String  0 instance 0000000000000000 _remoteStackTraceString
000007fef8f45f00  40000bf       70         System.Int32  1 instance                0 _remoteStackInex
000007fef8f3e580  40000c0       58        System.Object  0 instance 0000000000000000 _dynamicMethods

000007fef8f45f00  40000c1       74         System.Int32  1 instance      -2146232830 _HResult
000007fef8f3ec90  40000c2       60        System.String  0 instance 0000000000000000 _source
000007fef8f41698  40000c3       68        System.IntPtr  1 instance                0 _xptrs
000007fef8f45f00  40000c4       78         System.Int32  1 instance       -532459699 _xcode
000007fef8f34748  4000dad       80      System.Object[]  0 instance 00000000027c6980 _classes
000007fef8f34748  4000dae       88      System.Object[]  0 instance 00000000027c9bb8 _exceptions

We are looking for the value of LoaderExceptions Property, and it is an array of exceptions, so that will be the last field _exceptions. Its corresponding object address is 00000000027c9bb8. This field is an array, so we can see the actually memory layout of this array object by directly dump the memory content using dd command:

0:007>dd 00000000027c9bb8

00000000`027c9bb8  f8f34748 000007fe 00000001 00000000
00000000`027c9bc8  f8f3ef58 000007fe 027c8538 00000000
00000000`027c9bd8  00000000 08000004 f9730fc0 000007fe
00000000`027c9be8  027ca108 00000000 0281d720 00000000
00000000`027c9bf8  00000000 00000000 027c9cd0 00000000
00000000`027c9c08  00000000 00000000 00000000 00000000
00000000`027c9c18  00000000 00000000 027ca4c8 00000000
00000000`027c9c28  00000000 00000000 00000000 00000000

The first 2 DWORDs together 000007fef8f34748 is the address of method table of array type, and then the following 2 DWORDs together 0000000000000001 is number of elements in the array (which is 1 in this case), and then the next 2 DWORDs 000007fef8f3ef58 is the address of method table of the type of element inside the array (which is the method table of exception). And lastly, the final 2 DWORDs 00000000027c8538 is the object address of the first element in the array, which is what we want! So, we use !do on that address:

0:007>!do 00000000027c9bb8

Name: System.TypeLoadException
MethodTable: 000007fef9718088
EEClass: 000007fef8c308b0
Size: 160(0xa0) bytes
(C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fef8f3ec90  40000b5        8        System.String  0 instance 0000000000000000 _className
000007fef8f3d038  40000b6       10 ...ection.MethodBase  0 instance 0000000000000000 _exceptionMethod
000007fef8f3ec90  40000b7       18        System.String  0 instance 0000000000000000 _exceptionMethodString
000007fef8f3ec90  40000b8       20        System.String  0 instance 00000000027a8720 _message
000007fef8f37b98  40000b9       28 ...tions.IDictionary  0 instance 0000000000000000 _data
000007fef8f3ef58  40000ba       30     System.Exception  0 instance 0000000000000000 _innerException
000007fef8f3ec90  40000bb       38        System.String  0 instance 0000000000000000 _helpURL
000007fef8f3e580  40000bc       40        System.Object  0 instance 0000000000000000 _stackTrace
000007fef8f3ec90  40000bd       48        System.String  0 instance 0000000000000000 _stackTraceString
000007fef8f3ec90  40000be       50        System.String  0 instance 0000000000000000 _remoteStackTraceString
000007fef8f45f00  40000bf       70         System.Int32  1 instance                0 _remoteStackIndex
000007fef8f3e580  40000c0       58        System.Object  0 instance 0000000000000000 _dynamicMethods
000007fef8f45f00  40000c1       74         System.Int32  1 instance      -2146233054 _HResult
000007fef8f3ec90  40000c2       60        System.String  0 instance 0000000000000000 _source
000007fef8f41698  40000c3       68        System.IntPtr  1 instance                0 _xptrs
000007fef8f45f00  40000c4       78         System.Int32  1 instance       -532459699 _xcode
000007fef8f3ec90  40003b9       80        System.String  0 instance 00000000027a83e8 ClassName
000007fef8f3ec90  40003ba       88        System.String  0 instance 00000000027a8468 AssemblyName
000007fef8f3ec90  40003bb       90        System.String  0 instance 0000000000000000 MessageArg
000007fef8f45f00  40003bc       7c         System.Int32  1 instance      -2146233054 ResourceId

And then we want to see the actually message. And _message is just another String object, so we just need to use !do one more time:

0:007>!do 00000000027a8720

Name: System.String
MethodTable: 000007fef8f3ec90
EEClass: 000007fef8b4b038
Size: 586(0x24a) bytes
(C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: Could not load type MyCompany.MyProduct.TestImplementation.ITreeNode`1' from assembly 'TestImplementation, Version=1.0.523.0, Culture=neutral, PublicKeyToken=9396306c2be7fcc4'.
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fef8f45f00  4000096        8         System.Int32  1 instance              281 m_arrayLength
000007fef8f45f00  4000097        c         System.Int32  1 instance              179 m_stringLength
000007fef8f406d8  4000098       10          System.Char  1 instance               43 m_firstChar
000007fef8f3ec90  4000099       20        System.String  0   shared           static Empty
                                 >> Domain:Value  000000000009fba0:0000000002771308 <<
000007fef8f40588  400009a       28        System.Char[]  0   shared           static WhitespaceChars
                                 >> Domain:Value  000000000009fba0:0000000002771b28 <<

Mystery solved! it is the ITreeNode type from assembly TestImplementation giving us the trouble. Once we know about the exact type and assembly name, the rest of the debugging is just a piece of cake. Oh, btw, if you want to quit debugging, type single 'q' to quit debugger client, or double "qq" to stop both debugger client and server.

It turned out this bug was actually caused by a mismatched version of assembly, and you probably has guessed it at the beginning.

Takeaway:

All the techniques described above are very basic debugging tricks, including Image File Execution, Debugging Tools for Windows, debugger server and client, SOS extension and .NET object memory model. And you can find more details about each of them by searching their names on the Internet. In this blog, I have shown you how to use these techniques together to solve one particular bug, but of course, you can apply them together in many other creative ways to solve different bugs in various environment.

But I think the most important takeaway should be this:

Make your startup logic as simple as possible! Less codes at startup == Less bugs at startup