You try to install latest Windows SDK and it does not work.

If you look at the setup log file and you see:

8:58:43 AM Wednesday, June 27, 2012: C:\WinSDK\Setup\SFX\vcredist_x64.exe installation failed with return code 5100

Reason for this is you have a newer C++ runtime installed on your machine, for example by Visual Studio 2012.  The dumb Windows SDK installer will not install ANYTHING (even the simplest thing that does not depend on C runtime) if newer C++ runtime is installed.

Fix #1:  Install Windows SDK on clean machine and copy files over (if you just want the tools like me).

Fix #2:  Uninstall Microsoft Visual Studio C++ 2012 runtime (both x86 and x64 versions, then install Windows SDK

Windows Azure Diagnostics does not work, you see following in the Compute Emulator =>

[Diagnostics]: Error starting diagnostics:
System.FormatException: Invalid account string.
    at Microsoft.WindowsAzure.CloudStorageAccount.<Parse>b__0(String err)
    at Microsoft.WindowsAzure.CloudStorageAccount.ParseStringIntoSettings(String s, Action`1 error)
    at Microsoft.WindowsAzure.CloudStorageAccount.TryParse(String s, CloudStorageAccount& accountInformation, Action`1 error)
    at Microsoft.WindowsAzure.CloudStorageAccount.Parse(String value)
    at Microsoft.WindowsAzure.Plugins.Diagnostics.DiagnosticsAgentManager.<StartAgent>b__0()

Reason?  The reason is a connecting string parsing bug in Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString =>

<ConfigurationSettings>
<Setting
   name="Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString"

  value="DefaultEndpointsProtocol=https;AccountName=aaa;AccountKey=bbb;" />
</ConfigurationSettings>

Fix: Remove the trailing semi-colon at the end of the configuration string =>

<ConfigurationSettings>
<Setting
name="Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString"
value="DefaultEndpointsProtocol=https;AccountName=aaa;AccountKey=bbb" />
</ConfigurationSettings>

If you are using the SQL Native Client ODBC driver, there is built-in high performance very detailed driver tracing, here is how you use it.

#1. Create batch file to start tracing (this traces for all three versions of SQL Native Client ODBC Driver) named StartSNACTrace.cmd:

@echo off
echo Starting SNAC tracing...
echo {BA798F36-2325-EC5B-ECF8-76958A2AF9B5} 0xFFFFFFFF 128 SQLNCLI > providers.txt
echo {A9377239-477A-DD22-6E21-75912A95FD08} 0xFFFFFFFF 128 SQLNCLI10 >> providers.txt
echo {2DA81B52-908E-7DB6-EF81-76856BB47C4F} 0xFFFFFFFF 128 SQLNCLI11 >> providers.txt
reg add HKLM\SOFTWARE\Microsoft\BidInterface\Loader /v :Path /t REG_SZ /d "%systemroot%\system32\msdadiag.dll" /f
del /Q *.etl
logman create trace -n SnacTrace -ct perf -pf providers.txt -bs 10000 -nb 10000 50000 -o SnacTrace.etl -max 100 -cnf 00
logman start -n SnacTrace
echo Run StopSnacTrace.cmd to stop tracing.

Note, replace 0xFFFFFFFF with 0x00000000 to do minimum tracing.  The number 128 indicates save trace data as ANSI, which reduces by about 1/2 the space for the ETL files.

Note you MUST start the tracing before the process that uses the SQL Native Client ODBC driver starts (this is VERY IMPORTANT).  

Start the tracing then start the process that is using the SQL Native Client ODBC driver.   You can stop tracing at any time, even before the process exits.

#2. Create this batch file named StopSnacTrace.cmd to stop the tracing:

@echo off
echo Stopping SNAC tracing...
logman stop -n SnacTrace
logman delete -n SnacTrace
reg delete HKLM\SOFTWARE\Microsoft\BidInterface\Loader /v :Path /f
echo SNAC tracing stopped

The above settings will create a new trace file of maximum size 100MB numbered SnacTrace_000001.etl, SnacTrace_000002.etl, etc…

#3.  To process the trace files, you need to register a special mof file on the trace processing machine (you do not have to do this on the trace gathering machine)

#4. Download and register the all.mof file from "Data Access Tracing in SQL Server 2012" MSDN article (last time I checked the URL is here https://technet.microsoft.com/en-us/library/hh880086)

There is a section in this article that says "Download Sample and MOF Files". This file is named Setup.zip.

Download this file to some temporary folder and extract the files.  You will see a folder named MOF_Files, go into this folder and locate the snac*.mof files, then on the client machine where you want to do tracing,

run the following from an elevated command prompt:

C:\>mofcomp all.mof

Microsoft (R) MOF Compiler Version 6.1.7600.16385
Copyright (c) Microsoft Corp. 1997-2006. All rights reserved.
Parsing MOF file: all.mof
MOF file has been successfully parsed
Storing data in the repository...
Done!

This will register all the providers so you can now see them with logman, to verify they are registered, run:

C:\>logman query providers | findstr SQL
SQLNCLI.1                                {BA798F36-2325-EC5B-ECF8-76958A2AF9B5}
SQLNCLI10.1                              {A9377239-477A-DD22-6E21-75912A95FD08}
SQLNCLI11.1                              {2DA81B52-908E-7DB6-EF81-76856BB47C4F}

You should see all 3 SNAC driver versions in the output.  You are now ready to process the traces (the *.etl files generated)

#5.  To process the traces to extract the data to CSV file, use tracerpt tool (comes with Windows) or LogParser (you need to download this tool).

Run following from a command prompt:

tracerpt SnacTrace_000001.etl -of CSV -en ANSI -gmt -o SnacTrace_000001.csv

Note I find the LogParser tool (you can download from Microsoft) provides a nicer, more compact log file, try this command:

LogParser.exe "select eventnumber, eventname, timestamp, userdata into SnacTrace_000001.csv from SnacTrace_000001.etl"
-i:ETW -o:CSV -oTsFormat:"yyyy-MM-dd hh:mm:ss.n" -headers:off

This will give you a really nice looking trace file output, and LogParser is also very fast to process large traces.

Ahh PowerShell, I still cannot fully embrace your powerful weirdness.

With normal NT command prompt I can restart my SQL Server by running =>

net stop mssqlserver  & net start mssqlserver

This does not work in PowerShell, you get the usual cryptic, nonsensical and useless error message:

PS C:\> net stop mssqlserver & net start mssqlserver
Ampersand not allowed. The & operator is reserved for future use; use "&" to pass ampersand as a string.
At line:1 char:23
+ net stop mssqlserver & <<<<  net start mssqlserver
    + CategoryInfo          : ParserError: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : AmpersandNotAllowed

Personally, I would change this error message to:

Ampersand not allowed.  You can use the semi-colon instead for a command separator in PowerShell.

Ok, given this logic, then I try:

PS C:\> net stop mssqlserver "&" net start mssqlserver
The syntax of this command is:

NET STOP
service

Ok, I give up.  Time to search the internet.

Found some mention of an undocumented semi-colon as command separator, this works =>

PS C:\> net stop mssqlserver ; net start mssqlserver
The SQL Server (MSSQLSERVER) service is stopping.
The SQL Server (MSSQLSERVER) service was stopped successfully.

The SQL Server (MSSQLSERVER) service is starting.
The SQL Server (MSSQLSERVER) service was started successfully.

I find with every debugger command I run, I get annoying amounts of debugger logging spew saying “CLRDLL: No CLR image loaded (i.e. mscorwks.dll)”.

For example, when I dump a call stack I see:

0:000> kC

CLRDLL: No CLR image loaded (i.e. mscorwks.dll)

KERNELBASE!RaiseException

myapp!_CxxThrowException

..

CLRDLL: No CLR image loaded (i.e. mscorwks.dll)

CLRDLL: No CLR image loaded (i.e. mscorwks.dll)

CLRDLL: No CLR image loaded (i.e. mscorwks.dll)

CLRDLL: No CLR image loaded (i.e. mscorwks.dll)

CLRDLL: No CLR image loaded (i.e. mscorwks.dll)

CLRDLL: No CLR image loaded (i.e. mscorwks.dll)

CLRDLL: No CLR image loaded (i.e. mscorwks.dll)

CLRDLL: No CLR image loaded (i.e. mscorwks.dll)

WARNING: Stack unwind information not available. Following frames may be wrong.

Etc…

The fix to disable this spew is running:

.cordll –vd

The reason I saw this in my debugger is I have a startup script that runs:

.cordll -ve -u –l

This –ve means “enable vebose logging for CLR”.  Internally the debugger is trying to load CLR debugging support over and over and this is failing.  Running .cordll –vd disables the extended logging.

Note you can disable CLR integration with debugger to speed up the debugger command execution when debugging native code by running:

.cordll -d -u -vd

Read help on .cordll for details.

When setting up read-only routing for SQL Server AlwaysOn, you have to configure each availability replica with a read-only routing URL.

The read-only routing URL is just a “pointer” to the replica and determines what the replica will report back to the client driver when routing to the replica.

Suppose you have an AlwaysOn cluster with 3 cluster nodes, Node1, Node2, and Node3.   On each of these nodes you will have a SQL Server instance running.   For very simple configurations there will be one default SQL Server instance listening on tcp port 1433 on each node.  With a more complex configuration you may have clustered SQL Server instances on these nodes.

Read-only routing uses the following algorithm to locate a readable secondary:

• Client connects to an Availability Group listener endpoint.
• Note this endpoint always points to the primary replica for the availability group
• Client specifies ApplicationIntent=ReadOnly in the connection string, this is transmitted to the server during login
• On server side, server checks that incoming connection is using an Availability Group listener endpoint
• Otherwise, read-only routing is disabled
• Server checks the target database and determines if it is in an availability group
• If database is in an availability group, we check if the read_only_routing_list is set on the primary replica
• If list is not set, routing is disabled
• If list is set, then routing is enforced
• Server then enumerates the replicas in the read_only_routing_list and checks each replica in the list
• First replica it finds that is synchronizing and accepts readers (allow_connections=read_only or all) is the routing target
• Server next reads the read_only_routing_url from this replica and sends this response to the client
• Client reads routing URL and re-directs to the readable secondary instance

Given above, you need to ensure that the read_only_routing_url is properly set for each replica when enabling read-only routing, otherwise the client will be redirected to the wrong instance.

I created a T-SQL script that you can run against any instance and it will tell you what the appropriate read_only_routing_url is for the instance.   The script will detect if the instance is clustered or not an generate appropriate URL.   Script is included below, enjoy!  =>

-- Read-only routing url generation script.
-- Connect to each replica in your AlwaysOn cluster and run this script to get the read_only_routing_url for the replica.
-- Then set this to the read_only_routing_url for the availability group replica =>
--    alter availability group MyAvailabilityGroup modify replica on N'ThisReplica' with (secondary_role(read_only_routing_url=N'<url>'))
print 'Read-only-routing url script v.2012.1.24.1'

print 'This SQL Server instance version is [' + cast(serverproperty('ProductVersion') as varchar(256)) + ']'

if (ServerProperty('IsClustered') = 1)
begin
    print 'This SQL Server instance is a clustered SQL Server instance.'
end
else
begin
    print 'This SQL Server instance is a standard (not clustered) SQL Server instance.'   
end

if (ServerProperty('IsHadrEnabled') = 1)
begin
    print 'This SQL Server instance is enabled for AlwaysOn.'
end
else
begin
    print 'This SQL Server instance is NOT enabled for AlwaysOn.'
end

-- Detect SQL Azure instance.
declare @is_sql_azure bit
set @is_sql_azure = 0

begin try
    set @is_sql_azure = 1
    exec('declare @i int set @i = sql_connection_mode()')
    print 'This SQL Server instance is a Sql Azure instance.'
end try
begin catch
    set @is_sql_azure = 0
    print 'This SQL Server instance is NOT a Sql Azure instance.'
end catch

-- Check that this is SQL 11 or later, otherwise fail fast.
if (@@microsoftversion / 0x01000000 < 11 or @is_sql_azure > 0)
begin
    print 'This SQL Server instance does not support read-only routing, exiting script.'
end
else
begin -- if server supports read-only routing

    -- Fetch the dedicated admin connection (dac) port.
    -- Normally it's always port 1434, but to be safe here we fetch it from the instance.
    -- We use this later to exclude the admin port from read_only_routing_url.
    declare @dac_port int
    declare @reg_value varchar(255)
    exec xp_instance_regread
        N'HKEY_LOCAL_MACHINE',
        N'SOFTWARE\Microsoft\Microsoft SQL Server\\MSSQLServer\SuperSocketNetLib\AdminConnection\Tcp',
        N'TcpDynamicPorts',
        @reg_value output

    set @dac_port = cast(@reg_value as int)

    print 'This SQL Server instance DAC (dedicated admin) port is ' + cast(@dac_port as varchar(255))
    if (@dac_port = 0)
    begin
        print 'Note a DAC port of zero means the dedicated admin port is not enabled.'
    end

    -- Fetch ListenOnAllIPs value.
    -- If set to 1, this means the instance is listening to all IP addresses.
    -- If set to 0, this means the instance is listening to specific IP addresses.
    declare @listen_all int
    exec xp_instance_regread
        N'HKEY_LOCAL_MACHINE',
        N'SOFTWARE\Microsoft\Microsoft SQL Server\\MSSQLServer\SuperSocketNetLib\Tcp',
        N'ListenOnAllIPs',
        @listen_all output

    if (@listen_all = 1)
    begin
        print 'This SQL Server instance is listening to all IP addresses (default mode).'
    end
    else
    begin
        print 'This SQL Server instance is listening to specific IP addresses (ListenOnAllIPs is disabled).'
    end

    -- Check for dynamic port configuration, not recommended with read-only routing.
    declare @tcp_dynamic_ports varchar(255)
    exec xp_instance_regread
        N'HKEY_LOCAL_MACHINE',
        N'SOFTWARE\Microsoft\Microsoft SQL Server\\MSSQLServer\SuperSocketNetLib\Tcp\IPAll',
        N'TcpDynamicPorts',
        @tcp_dynamic_ports output

    if (@tcp_dynamic_ports = '0')
    begin
        print 'This SQL Server instance is listening on a dynamic tcp port, this is NOT A RECOMMENDED CONFIGURATION when using read-only routing, because the instance port can change each time the instance is restarted.'
    end
    else
    begin
        print 'This SQL Server instance is listening on fixed tcp port(s) (it is not configured for dynamic ports), this is a recommended configuration when using read-only routing.'
    end

    -- Calculate the server domain and instance FQDN.
    -- We use @server_domain later to build the FQDN to the clustered instance.
    declare @instance_fqdn varchar(255)
    declare @server_domain varchar(255)

    -- Get the instance FQDN using the xp_getnetname API
    -- Note all cluster nodes must be in same domain, so this works for calculating cluster FQDN.
    set @instance_fqdn = ''
    exec xp_getnetname @instance_fqdn output, 1

    -- Remove embedded null character at end if found.
    declare @terminator int
    set @terminator = charindex(char(0), @instance_fqdn) - 1
    if (@terminator > 0)
    begin
        set @instance_fqdn = substring(@instance_fqdn, 1, @terminator)
    end

    -- Build @server_domain using @instance_fqdn.
    set @server_domain = @instance_fqdn

    -- Remove trailing portion to extract domain name.
    set @terminator = charindex('.', @server_domain)
    if (@terminator > 0)
    begin
        set @server_domain = substring(@server_domain, @terminator+1, datalength(@server_domain))
    end
    print 'This SQL Server instance resides in domain ''' +  @server_domain + ''''

    if (ServerProperty('IsClustered') = 1)
    begin
        -- Fetch machine name, which for a clustered SQL instance returns the network name of the virtual server.
        -- Append @server_domain to build the FQDN.
        set @instance_fqdn = cast(serverproperty('MachineName') as varchar(255)) + '.' + @server_domain
    end

    declare @ror_url varchar(255)
    declare @instance_port int

    set @ror_url = ''

    -- Get first available port for instance.
    select
    top 1    -- Select first matching port
    @instance_port = port
    from sys.dm_tcp_listener_states
    where
    type=0 -- Type 0 = TSQL (to avoid mirroring endpoint)
    and
    state=0    --  State 0 is online   
    and
    port <> @dac_port -- Avoid DAC port (admin port)
    and
    -- Avoid availability group listeners
    ip_address not in (select ip_address from sys.availability_group_listener_ip_addresses agls)
    group by port       
    order by port asc  -- Pick first port in ascending order

    -- Check if there are multiple ports and warn if this is the case.
    declare @list_of_ports varchar(max)
    set @list_of_ports = ''

    select
    @list_of_ports = @list_of_ports +
        case datalength(@list_of_ports)
        when 0 then cast(port as varchar(max))
        else ',' +  cast(port as varchar(max))
        end
    from sys.dm_tcp_listener_states
    where
    type=0    --     Type 0 = TSQL (to avoid mirroring endpoint)
    and
    state=0    --  State 0 is online   
    and
    port <> @dac_port -- Avoid DAC port (admin port)
    and
    -- Avoid availability group listeners
    ip_address not in (select ip_address from sys.availability_group_listener_ip_addresses agls)
    group by port       
    order by port asc

    print 'This SQL Server instance FQDN (Fully Qualified Domain Name) is ''' + @instance_fqdn + ''''
    print 'This SQL Server instance port is ' + cast(@instance_port as varchar(10))

    set @ror_url = 'tcp://' + @instance_fqdn + ':' + cast(@instance_port as varchar(10))

    print '****************************************************************************************************************'
    print 'The read_only_routing_url for this SQL Server instance is ''' + @ror_url + ''''
    print '****************************************************************************************************************'

    -- If there is more than one instance port (unusual) list them out just in case.
    if (charindex(',', @list_of_ports) > 0)
    begin
        print 'Note there is more than one instance port, the list of available instance ports for read_only_routing_url is (' + @list_of_ports + ')'
        print 'The above URL just uses the first port in the list, but you can use any of these available ports.'
    end

end -- if server supports read-only routing
go

The root cause of this problem was code calling the Win32 API CloseHandle on a handle that it did not own.   Since Windows can aggresively reuse handle values, it’s possible to hit a situation like below:

Thread1: CreateFile –> Get handle #1.

Thread1: CloseHandle(#1)

Thread2: CreateFile –> Get handle #1, store in member variable.

Thread1: CloseHandle (#1) <- Mistake in code.

Thread1: CreateEvent –> Get handle #1.

Now my code on Thread2 thinks it has a file handle, but it has an event handle!   Thread2 did not make any coding mistake, the Thread1 code corrupted Thread2’s handle value.

This is an inherent issue when using handles, they are inherently dangerous you have to be very careful to only call CloseHandle 1 time.  Now the operating system could mitigate this by handling out random values for handles, but in general the operating system does not always do this for performance reasons.  The tradeoff is fast performance for all applications that behave correctly.

Now if you have a process where you suspect handles are getting misused (leaked or double closed), you can use Application Verifier to help track this down very quickly, this is what I want to talk about.

First download both Microsoft Application Verifier and Debugging Tools for Windows, both are free downloads from Microsoft.  I would provide links but these always change, get the latest versions of both.  Get the version that matches your operating system bitness as well for best results.

Install Microsoft Application Verifier and Debugging Tools for Windows on the machine where you have the problem.  Both are very clean installers and remove easily so don’t worry about installing on production machine, they are easy to remove and don’t require any reboot during removal.   If you are really worried you can install them on a separate machine and xcopy the files to the production machine, this works as well.

Next, figure out what process you want to debug.  If you are running IIS application with .NET, most likely this is the w3wp.exe process.  You don’t need the process ID yet.

First run Application Verifier to setup handle tracking, it looks like below:

Not too fancy.  :)  You want to create the entry for the process name next, select File | Add Application.   Let’s do the demonstration using notepad.exe for now.   Just type in notepad.exe and click Open.  You do not have to find and select the correct binary, just typing in the process name is sufficient, it does not need the full path:

Now that you’ve added the process, go to the right and select ONLY handles under the Basics options, like below:

The reason I only select handles here is to reduce the performance implications of all the other tracking inside the process.  I just want to track handles, this is all.

Once you’ve selected handles only, click Save, then close Application Verifier.  You will see this message:

This is fine, we will do this work later on (attach debugger).  Click OK and exit App Verifier.

Now, VERY IMPORTANT!  Re-open App Verifier.  Verify you see your process name correctly and you see only Handles is checked.  Do this step as a sanity check to ensure you selected the right thing AND it was saved correctly to registry.

Exit App Verifier again.

If your process is currently running, stop and restart the process.   If for example you are using IIS, you want to run iisreset at this point to restart IIS.  This step is required as the process reads the Application Verifier flags ONLY during startup.

In our case we are debugging notepad, so lets startup an instance of notepad.exe at this point (Start|Run –> Notepad.exe)

Now open Windbg.exe (from Debugging Tools For Windows install).  Select File | Attach To Process…

VERY IMPORTANT!  First thing to do is check the Non-invasive check box on the Attach to Process dialog, as below:

You should ALWAYS check this when doing handle tracking work because this allows you to attach to the process in a special mode called non-invasive mode.  This allows the debugger to be later detached and the process will continue normally.  If you don’t check this checkbox, the debugger will fully attach to the process and you cannot shut down the debugger without terminating the process.

Next, locate your process, you will see here my process is notepad.exe, select this process:

Click OK to attach the debugger non-invasively.   Once you are attached, go to the command window (select View|Command) and run the following commands:

0:000> !htrace -enable
Handle tracing enabled.
Handle tracing information snapshot successfully taken.

This command is required to enable the tracing of handles.  Once this is set, you can exit the debugger by running the q (quit) command:

0:000:> q

At this point handle tracking is enabled for the process.  For this experiment, now go to notepad and open a single text file but do not exit notepad.exe yet.

Now we want to see what handle tracking can do for us.  Startup Windbg.exe again and attach to notepad again in non-invasive mode.  Run the following commands:

0:000> .dump /ma c:\temp\notepad.dmp
Creating c:\temp\notepad.exe - mini user dump
Dump successfully written

Note the command is .dump /ma <filename> where <filename> can be anything you want here.  I called my dump file c:\temp\notepad.dmp.

This creates a dump of the process.  Now again run q (quit) command to exit debugger.

Re-open Windbg again, we are next going to examine the dump file.

In Windbg select File | Open Crash Dump… and select c:\temp\notepad.dmp and click Open.

In the command window, run the following 2 commands to load symbols:

0:000> .sympath srv*https://msdl.microsoft.com/download/symbols
Symbol search path is: srv*https://msdl.microsoft.com/download/symbols
Expanded Symbol search path is: srv*https://msdl.microsoft.com/download/symbols
0:000> .reload /f
..........................................*** ERROR: Symbol file could not be found.  Defaulted to export symbols for AvCheck.dll -
.....
Loading unloaded module list

Note you may see errors here, this is fine for now.

Now we want to see a trace of our handle activity, run the !htrace command with no arguments:

0:000> !htrace

Next you will see tons of debugger spew like below:

--------------------------------------
Handle = 0x00000000000002f0 - OPEN
Thread ID = 0x0000000000001390, Process ID = 0x0000000000000b6c

0x0000000077835e3a: ntdll!ZwDuplicateObject+0x000000000000000a
0x00000000776148a8: kernel32!DuplicateHandle+0x0000000000000055
0x000007fefe6cb435: rpcrt4!THREAD::THREAD+0x00000000000000b5
0x000007fefe6cb35c: rpcrt4!ThreadSelfHelper+0x0000000000000028
0x000007fefe77444c: rpcrt4!NdrpClientCall2+0x0000000000000698
0x000007fefe774ddd: rpcrt4!NdrClientCall2+0x000000000000001d
0x000007fefe4c0e9a: ole32!BulkUpdateOIDs+0x000000000000008a
0x000007fefe4c0d89: ole32!CRpcResolver::BulkUpdateOIDs+0x0000000000000109
0x000007fefe4c0c36: ole32!CROIDTable::ClientBulkUpdateOIDWithPingServer+0x0000000000000224
0x000007fefe4c94c3: ole32!CROIDTable::WorkerThreadLoop+0x0000000000000057
0x000007fefe4c9406: ole32!CRpcThread::WorkerLoop+0x000000000000001e
0x000007fefe4c9682: ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x000000000000001a
0x000000007761466d: kernel32!BaseThreadInitThunk+0x000000000000000d
0x0000000077818791: ntdll!RtlUserThreadStart+0x000000000000001d
--------------------------------------
Handle = 0x000000000000031c – OPEN

This shows you all the handle activity in the process.

You can locate a double close by searching for the CLOSE lines:

Handle = 0x000000000000033c - CLOSE

This will also have the stack trace of the code that closed the handle.

Now in my case what I did was gather a dump when some error case occurred in my process, then I examined the dump to determine what handle my code was using and then used !htrace to find out who closed it.  All of the information is kept in the crash dump nicely for you.

You can also pass a parameter to !htrace to dump activity for a single handle, like below:

0:000> !htrace 33c
--------------------------------------
Handle = 0x000000000000033c - CLOSE
Thread ID = 0x0000000000000744, Process ID = 0x0000000000000b6c

0x0000000077835faa: ntdll!ZwTerminateThread+0x000000000000000a
0x000000007780b508: ntdll!RtlExitUserThread+0x0000000000000048
0x000000007780b8a7: ntdll!TppWorkerThread+0x0000000000000a23
0x000000007761466d: kernel32!BaseThreadInitThunk+0x000000000000000d
0x0000000077818791: ntdll!RtlUserThreadStart+0x000000000000001d
--------------------------------------
Handle = 0x000000000000033c - OPEN
Thread ID = 0x0000000000000744, Process ID = 0x0000000000000b6c

0x000000007773d57a: user32!ZwUserCallOneParam+0x000000000000000a
0x000000007773d95f: user32!RealMsgWaitForMultipleObjectsEx+0x00000000000000b3
0x000000007773da2e: user32!MsgWaitForMultipleObjectsEx+0x0000000000000046
0x0000000077732af4: user32!MsgWaitForMultipleObjects+0x0000000000000020
0x000007fef5fbb198: browseui!CShellTaskScheduler::_TT_MsgWaitForMultipleObjects+0x0000000000000038
0x000007fef5fbb11d: browseui!CShellTaskScheduler::TT_TransitionThreadToRunningOrTerminating+0x00000000000000a1
0x000007fef5fbaea1: browseui!CShellTaskThread::ThreadProc+0x0000000000000076
--------------------------------------

You will notice a specific handle is closed and reopened many times over a period of time, this is normal.  What you want to look for is cases where CLOSE is called 2 times in a row to find cases of handle misuse.  Likewise if a handle is leaked you will not see a final CLOSE called, the last trace will be an OPEN.

Now, when you are finished debugging, BE ABSOLUTELY SURE to start App Verifier again and DELETE the process entry for your process!  You want to always do this since the App Verifier settings are persisted in the registry, they will survive even a server reboot.  Right click on the Image Name and select “Delete Application”.  Then click Save and close and re-open App Verifier to ensure that the entry is gone.  As well if you are debugging some service you want to restart the service to stop the tracking, run iisreset for example again to reset IIS to normal state if you are debugging IIS.

Hope this helps you debug the next handle misuse case you encounter, have fun!

Run dumpbin /headers on the binary, then look at the magic number.

This is an amd64 binary:

c:\>dumpbin /headers c:\amd64\msvcr100.dll | findstr magic
             20B magic # (PE32+)

This is a 32-bit binary:

c:\DPMedusa1>dumpbin /headers c:\x86\msvcr100.dll | findstr magic
             10B magic # (PE32)

dumpbin also works on executables, for example on my amd64 machine:

c:\>dumpbin /headers c:\Windows\System32\calc.exe | findstr magic
             20B magic # (PE32+)

c:\>dumpbin /headers c:\Windows\SysWOW64\calc.exe | findstr magic
             10B magic # (PE32)

    NT SERVICE\TrustedInstaller

C:\Windows>icacls c:\windows\winsxs
c:\windows\winsxs

                  NT SERVICE\TrustedInstaller:(OI)(CI)(F)
                  BUILTIN\Administrators:(OI)(CI)(RX)
                  NT AUTHORITY\SYSTEM:(OI)(CI)(RX)
                  BUILTIN\Users:(OI)(CI)(RX)

The security UI dialogs do not recognize TrustedInstaller just by itself (don't know why).