The good news is that it is easy to recover from this bad state. You simply need to open services.msc, and start "World Wide Web Publishing Service" manually youself.

Log Name:      Application
Source:        TFS Services
Date:          1/27/2013 8:53:37 AM
Event ID:      7006
Task Category: None
Level:         Warning
Detailed Message: There are no active requests for service host DefaultCollection that exceed the warning threshold of 30.

Log Name:      Application
Source:        TFS Services
Date:          1/27/2013 8:52:22 AM
Event ID:      7005
Task Category: None
Level:         Warning
Detailed Message: A request for service host DefaultCollection has been executing for 37 seconds, exceeding the warning threshold of 30.

After you verify that your TFS server is performing well and there is no performance issues end users are really experiencing, you may want to get rid of these warnings. The default threshold is set to 30 seconds. However, it is configureable through a TFS registry key. e.g., The setting below will change the threshold to 300 seconds:

DECLARE @registryUpdates typ_KeyValuePairStringTableNullable
INSERT @registryUpdates ([Key], [Value])
SELECT '#\Configuration\ServiceHostMonitor\TotalExecutionElapsedThreshold\',
'300'

EXEC prc_UpdateRegistry 1, 'Test', @registryUpdates, 1 

Earlier this week, I had a need to ngen several TFS assemblies on TFS 2010 AT so that I can get rid of broken call stacks in PerfView profiler. How hard could it be? Just run ngen.exe on these assemblies. That's what I throught as well. Well, I was wrong. :)

Tip #1. Pick the right version of ngen.exe

These are TFS assemblies I was trying to ngen:

• Microsoft.Teamfoundation.Framework.Server.dll
• Microsoft.Teamfoundation.VersionControler.Server.dll
• Microsoft.TeamFoundation.VersionControl.Server.Proxy.dll

Well, these assemblies are compiled against .NET V2.0. But appPool is configured to run with .NET V4.0 as a 64-bit. So we need to use the NGen.exe under d:\windows\microsoft .net\framework64\v.4.0.30319. 

To avoid typing the full path, I navigated to the above path and Set Path=%CD%;%PATH%

Tip #2: Use /nodepdencies

So I navigated to web service\bin folder where TFS assemblies are located and ran "ngen.exe install %TfsAssemblyName%". This failed with a bunch of dependenies not found warnings.

Warning: ngen'ed image is still getting created though. So make sure you uninstall them properly so that your ngen folder is not clogged with a bunch of "bad" images that won't get used by your application anyway. And to uninstall them properyl, you need to use the same version of ngen.exe you used to install it, and pass in the same parameters.  Anyway, I forgot to do this and later on I just couldn't uninstall a few of them from NGEN folder. So I just use brutal force and navigated to these folders and deleted them manually. (It didn't seem to cause any lingering issues. well so far. :))

Well, I'm aware of /nodependencies option for ngen.exe. So I ran "ngen.exe install %TfsAssemblyName% /nodependencies".

This still failed and I noticed it is looking up .NET V2.0 dependencies. Hmmm, so how do I tell ngen.exe to use .NET V4.0 for these TFS assemblies which are compiled against .NET V2.0?

I know MS.TF.Framework.Server.dll is used by TfsMgmt.exe as well and TfsMgmt.exe is compiled with .NET V4.0. It is doing so with these entries in the TfsMgmt.exe.config file:

<configuration>
  <startup useLegacyV2RuntimeActivationPolicy="true">
    <supportedRuntime version="v4.0"/>
  </startup>
</configuration>

So maybe we can pass this along to ngen.exe?

Tip #3: Use /ExeConfig

Now I'm running this command: "ngen.exe install %TfsAssemblyName% /nodependencies /execonfig:" D:\Program Files\Microsoft Team Foundation Server 2010\Tools\Tfsmgmt.exe"".

This seemed to work!!!! :)

Tip#4: Use "ngen.exe display /verbose"

Now I did a "IISreset" and use Process Explorer to verify my ngen assemblies are loaded. Unfortunatelly, MS.TF.VersionControl.Server.dll is still not using the ngen'ed version. :(

It turns out Ngen.exe has a nice command "ngen.exe display /verbose > out.txt" that give you a lot of good information on any ngen'ed assemblies. It tells you which version of ngen images are installed and which options were
used while it was installed:

• ScenarioNoDependencies: this means /NoDependencies is used
• ConfigExe: this means /configExe is used
• RunTime Version: this tellyou which version of Ngen.exe was used.
• Processor or ngen install path: This tells you if 32 bit or 64-bit ngen was used
• ** Missing dependency assembly **: this tells you while ngen image was created, it failed to locate a dependent assembly.  This ngen image will failed to load by
       your application.

With this, I found that there are two missing dependency dlls for MS.TF.VersionControl.Server.dll while other two ngen'ed assemblies have no missing dependencies:

• Microsoft.TeamFoundaton.VersionControl.Server.Proxy.dll
• Microsoft.TeamFoundation.WorkItemTracking.Server.DataAccessLayer.dll

But both these dlls are there on the "web Service\bin" directory, the same directory where the MS.TF.VersionControl.Server.dll is. So why does ngen.exe failed to find these assemblies?

Tip #5: Use FusLogVw.exe to troubleshoot

To figure out why ngen.exe failed to locate these two dlls while ngen'ing MS.TF.Versioncontrol.Server.dll, I enabled FusLogVw.exe. (copied D:\Program Files\Microsoft SDKs\Windows\v7.0A\bin\NETFX 4.0 Tools from a VS client machine).

The log revealed the reason. Becuase I was using /execonfig option on ngen.exe: ngen.exe install  Microsoft.Teamfoundation.VersionControler.Server.dll /nodependencies /execonfig:" D:\Program Files\Microsoft Team Foundation Server 2010\Tools\Tfsmgmt.exe"

So why wasn't thin an issue for other two assemblies? (Framework.Server and Proxy). Because their dependent dlls are also used by TfsMgmt.exe so they are all avaailbe under D:\Program Files\Microsoft Team Foundation Server 2010\Tools!!!

Armed with this knownledge, I fixed the ngen issue on MS.TF.VersionControl.Server.dll  with these steps:

1. Copy TfsMgmt.exe and TfsMgmt.exe.config from D:\Program Files\Microsoft Team Foundation Server 2010\Tools to D:\Program Files\Microsoft Team Foundation Server 2010\Web Services\bin

2. ngen.exe install  Microsoft.Teamfoundation.VersionControler.Server.dll /nodependencies /execonfig:" D:\Program Files\Microsoft Team Foundation Server 2010\Web Services\bin\Tfsmgmt.exe"

Now I have ngen'ed TFS assemblies loaded in my appPool!!!

*** Assembly Binder Log Entry  (8/8/2012 @ 1:52:33 PM) ***

The operation failed.
Bind result: hr = 0x80070002. The system cannot find the file specified.

Assembly manager loaded from:  D:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
Running under executable  D:\Windows\Microsoft.NET\Framework64\v4.0.30319\mscorsvw.exe
--- A detailed error log follows. 

=== Pre-bind state information ===
LOG: User = DDNET\VSTSLAB
LOG: DisplayName = Microsoft.TeamFoundation.VersionControl.Server.Proxy, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
 (Fully-specified)
LOG: Appbase = file:///D:/Program Files/Microsoft Team Foundation Server 2010/Tools/ LOG: Initial PrivatePath = NULL
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = mscorsvw.exe
Calling assembly : (Unknown).
===
LOG: This bind starts in default load context.
LOG: Using application configuration file: D:\Program Files\Microsoft Team Foundation Server 2010\Tools\Tfsmgmt.exe.config LOG: Using host configuration file: 
LOG: Using machine configuration file from D:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config.
LOG: GAC Lookup was unsuccessful.
LOG: Attempting download of new URL file:///D:/Program Files/Microsoft Team Foundation Server 2010/Tools/Microsoft.TeamFoundation.VersionControl.Server.Proxy.DLL. LOG: Attempting download of new URL file:///D:/Program Files/Microsoft Team Foundation Server 2010/Tools/Microsoft.TeamFoundation.VersionControl.Server.Proxy/Microsoft.TeamFoundation.VersionControl.Server.Proxy.DLL. LOG: Attempting download of new URL file:///D:/Program Files/Microsoft Team Foundation Server 2010/Tools/Microsoft.TeamFoundation.VersionControl.Server.Proxy.EXE. LOG: Attempting download of new URL file:///D:/Program Files/Microsoft Team Foundation Server 2010/Tools/Microsoft.TeamFoundation.VersionControl.Server.Proxy/Microsoft.TeamFoundation.VersionControl.Server.Proxy.EXE. LOG: All probing URLs attempted and failed. 

Tip #6: You can also use FusLogVw.exe to find out why an ngen'ed image is not been used.

a. Make sure you check "Native Image" option in settings.

b. For a failed ngen assembliy, you will see something like this:

WRN: Dependency assembly was not found at ngen time, but is found at binding time. Disallow using this native image.
WRN: No matching native image found.
LOG: Bind to native image assembly did not succeed. Use IL image.
LOG: IL assembly loaded from D:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\ad61b6f7\f7e77928\assembly\dl3\76ac4941\00c26601_b5c7ca01\Microsoft.TeamFoundation.VersionControl.Server.dll.

Have a good day!

Jiange

System.InvalidOperationException: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool. 

This may have occurred because all pooled connections were in use and max pool size was reached.

Basically this means that the connection pool is full and that all connections are in use.

1. Verified SQL connections are indeed leaked

I used Perf counter ".Net Data Provider for SqlServer"\NumberOfRelaimedConnections to verify we indeed have a leak. This perf counter keeps going up during my load run, which means that a lot of SQL connections are not closed properly by our code.

2. Verify SQL connection pool is maxed out

0:079> !dumpheap -stat -type System.Data.ProviderBase.DbConnectionPool
Statistics:
              MT    Count    TotalSize Class Name
000007fedf56fe00        1           24 System.Collections.Generic.ObjectEqualityComparer`1[[System.Data.ProviderBase.DbConnectionPoolIdentity, System.Data]]
...
000007fedf56ffd8        2          192 System.Collections.Generic.Dictionary`2+Entry[[System.Data.Common.DbConnectionPoolKey, System.Data],[System.Data.ProviderBase.DbConnectionPoolGroup, System.Data]][]
000007fedf56fe88        4          192 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Data.ProviderBase.DbConnectionPoolIdentity, System.Data],[System.Data.ProviderBase.DbConnectionPool, System.Data]]
000007fedf386ec0        4          224 System.Collections.Concurrent.ConcurrentDictionary`2[[System.Data.ProviderBase.DbConnectionPoolIdentity, System.Data],[System.Data.ProviderBase.DbConnectionPool, System.Data]]
000007fedf56ed90       14          336 System.Data.ProviderBase.DbConnectionPoolCounters+Counter
000007fedf56bfc8        2          336 System.Data.ProviderBase.DbConnectionPool
000007fee1cbeb28       10         2944 System.Object[]
Total 63 objects

0:079> !dumpheap -mt 000007fedf56bfc8       
         Address               MT     Size
000000047fd3a260 000007fedf56bfc8      168    
000000047fd5f468 000007fedf56bfc8      168    

Statistics:
              MT    Count    TotalSize Class Name
000007fedf56bfc8        2          336 System.Data.ProviderBase.DbConnectionPool
Total 2 objects

0:079> !do 000000047fd3a260
Name:        System.Data.ProviderBase.DbConnectionPool
MethodTable: 000007fedf56bfc8
EEClass:     000007fedf3cfd48
Size:        168(0xa8) bytes
File:        D:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fee1cf6408  4000e70       80         System.Int32  1 instance           190000 _cleanupWait
000007fedf56ef90  4000e71        8 ...ctionPoolIdentity  0 instance 000000047fd3a240 _identity
000007fedf56a3c8  4000e72       10 ...ConnectionFactory  0 instance 000000047fd31438 _connectionFactory
000007fedf56df40  4000e73       18 ...nnectionPoolGroup  0 instance 000000047fd36d10 _connectionPoolGroup
000007fedf56cec0  4000e74       20 ...nPoolGroupOptions  0 instance 000000047fd36ce8 _connectionPoolGroupOptions
000007fedf57e7a8  4000e75       28 ...nPoolProviderInfo  0 instance 0000000000000000 _connectionPoolProviderInfo
000007fedf577368  4000e76       84         System.Int32  1 instance                1 _state
000007fedf387188  4000e77       30 ...al, System.Data]]  0 instance 000000047fd3a3e8 _stackOld
000007fedf387188  4000e78       38 ...al, System.Data]]  0 instance 000000047fd3a408 _stackNew
000007fedf387240  4000e79       40 ...on, System.Data]]  0 instance 000000047fd3a428 _pendingOpens
000007fee1cf6408  4000e7a       88         System.Int32  1 instance                0 _pendingOpensWaiting
000007fee1ce3b90  4000e7b       48 ...ding.WaitCallback  0 instance 000000047fd3add0 _poolCreateRequest
000007fee1cf6408  4000e7c       8c         System.Int32  1 instance                0 _waitCount
000007fedf568ff0  4000e7d       50 ...l+PoolWaitHandles  0 instance 000000047fd3a708 _waitHandles
000007fee1cf3888  4000e7e       58     System.Exception  0 instance 0000000000000000 _resError
000007fee1cf4f30  4000e7f       9c       System.Boolean  1 instance                0 _errorOccurred
000007fee1cf6408  4000e80       90         System.Int32  1 instance             5000 _errorWait
000007fee1cf1fe8  4000e81       60 ...m.Threading.Timer  0 instance 0000000000000000 _errorTimer
000007fee1cf1fe8  4000e82       68 ...m.Threading.Timer  0 instance 000000047fd3aef0 _cleanupTimer
000007fedf569850  4000e83       70 ...tedConnectionPool  0 instance 000000047fd3abb8 _transactedConnectionPool
000007fedf387308  4000e84       78 ...al, System.Data]]  0 instance 000000047fd3a850 _objectList
000007fee1cf6408  4000e85       94         System.Int32  1 instance                7 _totalObjects
000007fee1cf6408  4000e87       98         System.Int32  1 instance                1 _objectID
000007fee1ce4810  4000e6f      b88        System.Random  0   shared           static _random
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd3a5f0 <<
000007fee1cf6408  4000e86      9e0         System.Int32  1   shared           static _objectTypeCount
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:4 <<

Ok, this Connection pool is not maxed out. Check next one.

0:079> !do 00000002bfe10368   
Free Object
Size:        638840(0x9bf78) bytes
0:079> !do 000000047fd5f468
Name:        System.Data.ProviderBase.DbConnectionPool
MethodTable: 000007fedf56bfc8
EEClass:     000007fedf3cfd48
Size:        168(0xa8) bytes
File:        D:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fee1cf6408  4000e70       80         System.Int32  1 instance           170000 _cleanupWait
000007fedf56ef90  4000e71        8 ...ctionPoolIdentity  0 instance 000000047fd3a240 _identity
000007fedf56a3c8  4000e72       10 ...ConnectionFactory  0 instance 000000047fd31438 _connectionFactory
000007fedf56df40  4000e73       18 ...nnectionPoolGroup  0 instance 000000047fd5f090 _connectionPoolGroup
000007fedf56cec0  4000e74       20 ...nPoolGroupOptions  0 instance 000000047fd5f068 _connectionPoolGroupOptions
000007fedf57e7a8  4000e75       28 ...nPoolProviderInfo  0 instance 0000000000000000 _connectionPoolProviderInfo
000007fedf577368  4000e76       84         System.Int32  1 instance                1 _state
000007fedf387188  4000e77       30 ...al, System.Data]]  0 instance 000000047fd5f510 _stackOld
000007fedf387188  4000e78       38 ...al, System.Data]]  0 instance 000000047fd5f530 _stackNew
000007fedf387240  4000e79       40 ...on, System.Data]]  0 instance 000000047fd5f550 _pendingOpens
000007fee1cf6408  4000e7a       88         System.Int32  1 instance                0 _pendingOpensWaiting
000007fee1ce3b90  4000e7b       48 ...ding.WaitCallback  0 instance 000000047fd5fc40 _poolCreateRequest
000007fee1cf6408  4000e7c       8c         System.Int32  1 instance               20 _waitCount
000007fedf568ff0  4000e7d       50 ...l+PoolWaitHandles  0 instance 000000047fd5f718 _waitHandles
000007fee1cf3888  4000e7e       58     System.Exception  0 instance 0000000000000000 _resError
000007fee1cf4f30  4000e7f       9c       System.Boolean  1 instance                0 _errorOccurred
000007fee1cf6408  4000e80       90         System.Int32  1 instance             5000 _errorWait
000007fee1cf1fe8  4000e81       60 ...m.Threading.Timer  0 instance 0000000000000000 _errorTimer
000007fee1cf1fe8  4000e82       68 ...m.Threading.Timer  0 instance 000000047fd5fcc0 _cleanupTimer
000007fedf569850  4000e83       70 ...tedConnectionPool  0 instance 000000047fd5fbc8 _transactedConnectionPool
000007fedf387308  4000e84       78 ...al, System.Data]]  0 instance 000000047fd5f860 _objectList
000007fee1cf6408  4000e85       94         System.Int32  1 instance              100 _totalObjects
000007fee1cf6408  4000e87       98         System.Int32  1 instance                3 _objectID
000007fee1ce4810  4000e6f      b88        System.Random  0   shared           static _random
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd3a5f0 <<
000007fee1cf6408  4000e86      9e0         System.Int32  1   shared           static _objectTypeCount
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:4 <<

This connection pool appears to be maxed out (default pool size is 100. Check _connectionPoolGroupOptions to see if default pool size is used).

0:079> !do 000000047fd5f068
Name:        System.Data.ProviderBase.DbConnectionPoolGroupOptions
MethodTable: 000007fedf56cec0
EEClass:     000007fedf3d0438
Size:        40(0x28) bytes
File:        D:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fee1cf4f30  4000ed4       14       System.Boolean  1 instance                1 _poolByIdentity
000007fee1cf6408  4000ed5        8         System.Int32  1 instance                0 _minPoolSize
000007fee1cf6408  4000ed6        c         System.Int32  1 instance              100 _maxPoolSize
000007fee1cf6408  4000ed7       10         System.Int32  1 instance            15000 _creationTimeout
000007fee1cf0658  4000ed8       18      System.TimeSpan  1 instance 000000047fd5f080 _loadBalanceTimeout
000007fee1cf4f30  4000ed9       15       System.Boolean  1 instance                1 _hasTransactionAffinity
000007fee1cf4f30  4000eda       16       System.Boolean  1 instance                0 _useLoadBalancing

Yep, one of the SQL connection pool is indeed maxed out.

3. Figure out which SQL connections are leaked and who is leaking them

Leaked SQL Connections are not GC rooted and their _closedCount field is set to 0.

0:079> !dumpheap -stat -type System.Data.SqlClient.SqlConnection
Statistics:
              MT    Count    TotalSize Class Name
000007fedf56cac8        1           64 System.Data.SqlClient.SqlConnectionFactory
000007fedf3a5ce0        1           80 System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[System.Data.SqlClient.SqlConnectionStringBuilder+Keywords, System.Data]]
000007fedf56ccc0        2          112 System.Data.SqlClient.SqlConnectionPoolGroupProviderInfo
000007fedf56ae10        2          400 System.Data.SqlClient.SqlConnectionString
000007fedfb11a88        1         1440 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[System.Data.SqlClient.SqlConnectionStringBuilder+Keywords, System.Data]][]
000007fedf56a818      107         4280 System.Data.SqlClient.SqlConnectionTimeoutErrorInternal
000007fee1cbeb28      107        11128 System.Object[]
000007fedf56daf0      749        17976 System.Data.SqlClient.SqlConnectionTimeoutPhaseDuration
000007fedf56a620      559        22360 System.Data.SqlClient.SqlConnectionPoolKey
000007fedf566170      588        75264 System.Data.SqlClient.SqlConnection
000007fedf577a00      557       102488 System.Data.SqlClient.SqlConnectionStringBuilder
Total 2674 objects

Dump out all Sql Connections so we can examine their _closedCount field: (We're looking for the ones with _ClosedCount=0)

0:079> .foreach (conn {!DumpHeap -short -MT 000007fedf566170}) {.echo foo; !do conn}
0000000100030770
Name:        System.Data.SqlClient.SqlConnection
MethodTable: 000007fedf566170
EEClass:     000007fedf3a8110
Size:        128(0x80) bytes
File:        D:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fee1cf3c38  4000198        8        System.Object  0 instance 0000000000000000 __identity
000007fee0d6f2e8  40002d2       10 ...ponentModel.ISite  0 instance 0000000000000000 site
000007fee0d7dbe0  40002d3       18 ....EventHandlerList  0 instance 0000000100032178 events
000007fee1cf3c38  40002d1      1b8        System.Object  0   shared           static EventDisposed
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd55f50 <<
000007fedf581510  40002c0       20 ...hangeEventHandler  0 instance 0000000000000000 _stateChangeEventHandler
000007fedf57e4c0  400107c       28 ...t.SqlDebugContext  0 instance 0000000000000000 _sdc
000007fee1cf4f30  400107d       70       System.Boolean  1 instance                0 _AsyncCommandInProgress
000007fedf5863e0  400107e       30 ...ent.SqlStatistics  0 instance 0000000000000000 _statistics
000007fee1cf4f30  400107f       71       System.Boolean  1 instance                0 _collectstats
000007fee1cf4f30  4001080       72       System.Boolean  1 instance                0 _fireInfoMessageEventOnUserErrors
000007fedfb0de48  4001081       38 ....Task, mscorlib]]  0 instance 0000000000000000 _currentCompletion
000007fedf5698b0  4001082       40 ...ent.SqlCredential  0 instance 0000000000000000 _credential
000007fee1cf3658  4001083       48        System.String  0 instance 00000001000306a8 _connectionString
000007fedf56c0e0  4001086       50 ...ConnectionOptions  0 instance 0000000000000000 _userConnectionOptions
000007fedf56df40  4001087       58 ...nnectionPoolGroup  0 instance 0000000000000000 _poolGroup
000007fedf56d730  4001088       60 ...onnectionInternal  0 instance 000000047fd43d80 _innerConnection
000007fee1cf6408  4001089       68         System.Int32  1 instance                1 _closeCount
000007fee1cf6408  400108b       6c         System.Int32  1 instance           322025 ObjectID
000007fee1cf3c38  400107b      ea0        System.Object  0   shared           static EventInfoMessage
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd31420 <<
000007fedf56a3c8  4001084      ea8 ...ConnectionFactory  0   shared           static _connectionFactory
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd31438 <<
000007fee1ce8290  4001085      eb0 ...eAccessPermission  0   shared           static ExecutePermission
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd352f8 <<
000007fee1cf6408  400108a      a08         System.Int32  1   shared           static _objectTypeCount
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:322560 <<
...
00000004c0280050
Name:        System.Data.SqlClient.SqlConnection
MethodTable: 000007fedf566170
EEClass:     000007fedf3a8110
Size:        128(0x80) bytes
File:        D:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fee1cf3c38  4000198        8        System.Object  0 instance 0000000000000000 __identity
000007fee0d6f2e8  40002d2       10 ...ponentModel.ISite  0 instance 0000000000000000 site
000007fee0d7dbe0  40002d3       18 ....EventHandlerList  0 instance 00000004c02819b8 events
000007fee1cf3c38  40002d1      1b8        System.Object  0   shared           static EventDisposed
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd55f50 <<
000007fedf581510  40002c0       20 ...hangeEventHandler  0 instance 0000000000000000 _stateChangeEventHandler
000007fedf57e4c0  400107c       28 ...t.SqlDebugContext  0 instance 0000000000000000 _sdc
000007fee1cf4f30  400107d       70       System.Boolean  1 instance                0 _AsyncCommandInProgress
000007fedf5863e0  400107e       30 ...ent.SqlStatistics  0 instance 0000000000000000 _statistics
000007fee1cf4f30  400107f       71       System.Boolean  1 instance                0 _collectstats
000007fee1cf4f30  4001080       72       System.Boolean  1 instance                0 _fireInfoMessageEventOnUserErrors
000007fedfb0de48  4001081       38 ....Task, mscorlib]]  0 instance 0000000000000000 _currentCompletion
000007fedf5698b0  4001082       40 ...ent.SqlCredential  0 instance 0000000000000000 _credential
000007fee1cf3658  4001083       48        System.String  0 instance 00000004c027ff88 _connectionString
000007fedf56c0e0  4001086       50 ...ConnectionOptions  0 instance 000000047fd5ed68 _userConnectionOptions
000007fedf56df40  4001087       58 ...nnectionPoolGroup  0 instance 000000047fd5f090 _poolGroup
000007fedf56d730  4001088       60 ...onnectionInternal  0 instance 00000003c00425d0 _innerConnection
000007fee1cf6408  4001089       68         System.Int32  1 instance                0 _closeCount
000007fee1cf6408  400108b       6c         System.Int32  1 instance           322129 ObjectID
000007fee1cf3c38  400107b      ea0        System.Object  0   shared           static EventInfoMessage
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd31420 <<
000007fedf56a3c8  4001084      ea8 ...ConnectionFactory  0   shared           static _connectionFactory
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd31438 <<
000007fee1ce8290  4001085      eb0 ...eAccessPermission  0   shared           static ExecutePermission
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd352f8 <<
000007fee1cf6408  400108a      a08         System.Int32  1   shared           static _objectTypeCount
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:322560 <<

Picked one Sql Connection object that is still open (_closeCount is zero!). Now let's make sure it's unreachable (i.e. leaked!).

0:079> !GCRoot 00000004c08c36c8
Thread 254c:
    0000000014fbc2e0 000007fedf4f4229 System.Data.ProviderBase.DbConnectionPool.TryGetConnection(System.Data.Common.DbConnection, UInt32, Boolean, Boolean, System.Data.Common.DbConnectionOptions, System.Data.ProviderBase.DbConnectionInternal ByRef)
        rbp+c8: 0000000014fbc3d8
            ->  00000004c08c36c8 System.Data.SqlClient.SqlConnection

...

    0000000014fbc6a0 000007fedbb6a55d Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.Execute(ExecuteType, System.Data.CommandBehavior)
        rbp+210: 0000000014fbc920
            ->  00000004c08c2b00 Microsoft.TeamFoundation.WorkItemTracking.Server.DalSqlResourceComponent6
            ->  00000004c08c36c8 System.Data.SqlClient.SqlConnection

Found 7 unique roots (run '!GCRoot -all' to see all roots).

Well, that SQL connection is still in-use. (not leaked.) Let's check another one.

0:079> !gcroot 00000004c0280050
Found 0 unique roots (run '!GCRoot -all' to see all roots).

Ok, so it's unreachable and open, it's therefore a leaked connection. We can tell more than just the connection string by dumping the inner connection:

0:079> !do 00000004c0280050
Name:        System.Data.SqlClient.SqlConnection
MethodTable: 000007fedf566170
EEClass:     000007fedf3a8110
Size:        128(0x80) bytes
File:        D:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fee1cf3c38  4000198        8        System.Object  0 instance 0000000000000000 __identity
000007fee0d6f2e8  40002d2       10 ...ponentModel.ISite  0 instance 0000000000000000 site
000007fee0d7dbe0  40002d3       18 ....EventHandlerList  0 instance 00000004c02819b8 events
000007fee1cf3c38  40002d1      1b8        System.Object  0   shared           static EventDisposed
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd55f50 <<
000007fedf581510  40002c0       20 ...hangeEventHandler  0 instance 0000000000000000 _stateChangeEventHandler
000007fedf57e4c0  400107c       28 ...t.SqlDebugContext  0 instance 0000000000000000 _sdc
000007fee1cf4f30  400107d       70       System.Boolean  1 instance                0 _AsyncCommandInProgress
000007fedf5863e0  400107e       30 ...ent.SqlStatistics  0 instance 0000000000000000 _statistics
000007fee1cf4f30  400107f       71       System.Boolean  1 instance                0 _collectstats
000007fee1cf4f30  4001080       72       System.Boolean  1 instance                0 _fireInfoMessageEventOnUserErrors
000007fedfb0de48  4001081       38 ....Task, mscorlib]]  0 instance 0000000000000000 _currentCompletion
000007fedf5698b0  4001082       40 ...ent.SqlCredential  0 instance 0000000000000000 _credential
000007fee1cf3658  4001083       48        System.String  0 instance 00000004c027ff88 _connectionString
000007fedf56c0e0  4001086       50 ...ConnectionOptions  0 instance 000000047fd5ed68 _userConnectionOptions
000007fedf56df40  4001087       58 ...nnectionPoolGroup  0 instance 000000047fd5f090 _poolGroup
000007fedf56d730  4001088       60 ...onnectionInternal  0 instance 00000003c00425d0 _innerConnection
000007fee1cf6408  4001089       68         System.Int32  1 instance                0 _closeCount
000007fee1cf6408  400108b       6c         System.Int32  1 instance           322129 ObjectID
000007fee1cf3c38  400107b      ea0        System.Object  0   shared           static EventInfoMessage
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd31420 <<
000007fedf56a3c8  4001084      ea8 ...ConnectionFactory  0   shared           static _connectionFactory
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd31438 <<
000007fee1ce8290  4001085      eb0 ...eAccessPermission  0   shared           static ExecutePermission
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd352f8 <<
000007fee1cf6408  400108a      a08         System.Int32  1   shared           static _objectTypeCount
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:322560 <<

0:079> !do 00000003c00425d0
Name:        System.Data.SqlClient.SqlInternalConnectionTds
MethodTable: 000007fedf56e8a8
EEClass:     000007fedf3d0cd0
Size:        296(0x128) bytes
File:        D:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fee1cf6408  40008b8       38         System.Int32  1 instance              869 _objectID
000007fee1cf4f30  40008bb       44       System.Boolean  1 instance                0 _allowSetConnectionString
000007fee1cf4f30  40008bc       45       System.Boolean  1 instance                1 _hidePassword
000007fedf567c68  40008bd       3c         System.Int32  1 instance                1 _state
000007fee1cf0850  40008be        8 System.WeakReference  0 instance 00000003c0042768 _owningObject
000007fedf56bfc8  40008bf       10 ....DbConnectionPool  0 instance 000000047fd5f468 _connectionPool
000007fedf568ca8  40008c0       18 ...ctionPoolCounters  0 instance 000000047fd31478 _performanceCounters
000007fedf56a6c8  40008c1       20 ...ferenceCollection  0 instance 00000003c0047278 _referenceCollection
000007fee1cf6408  40008c2       40         System.Int32  1 instance                0 _pooledCount
000007fee1cf4f30  40008c3       46       System.Boolean  1 instance                0 _connectionIsDoomed
000007fee1cf4f30  40008c4       47       System.Boolean  1 instance                0 _cannotBePooled
000007fee1cf4f30  40008c5       48       System.Boolean  1 instance                0 _isInStasis
000007fee1cf0980  40008c6       50      System.DateTime  1 instance 00000003c0042620 _createTime
000007fef53a39d8  40008c7       28 ...tions.Transaction  0 instance 0000000000000000 _enlistedTransaction
000007fef53a39d8  40008c8       30 ...tions.Transaction  0 instance 0000000000000000 _enlistedTransactionOriginal
000007fee1cf6408  40008b7      6e0         System.Int32  1   shared           static _objectTypeCount
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:969 <<
000007fedf56f5d8  40008b9      618 ...teChangeEventArgs  0   shared           static StateChangeClosed
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd36570 <<
000007fedf56f5d8  40008ba      620 ...teChangeEventArgs  0   shared           static StateChangeOpen
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd36588 <<
000007fedf56ae10  4001258       58 ...lConnectionString  0 instance 000000047fd5ed68 _connectionOptions
000007fee1cf4f30  4001259       88       System.Boolean  1 instance                0 _isEnlistedInTransaction
000007fee1cf77d0  400125a       60        System.Byte[]  0 instance 0000000000000000 _promotedDTCToken
000007fee1cf77d0  400125b       68        System.Byte[]  0 instance 0000000000000000 _whereAbouts
000007fee1cf3658  400125c       70        System.String  0 instance 00000003c00432b8 <CurrentDatabase>k__BackingField
000007fee1cf3658  400125d       78        System.String  0 instance 000000047fd5ee80 <CurrentDataSource>k__BackingField
000007fedfb22358  400125e       80 ...egatedTransaction  0 instance 0000000000000000 <DelegatedTransaction>k__BackingField
000007fedf56ccc0  400126e       90 ...GroupProviderInfo  0 instance 000000047fd5f0d8 _poolGroupProviderInfo
000007fedf569ad8  400126f       98 ...lClient.TdsParser  0 instance 00000003c0042850 _parser
000007fedf56dbb0  4001270       a0 ...lient.SqlLoginAck  0 instance 00000003c0043340 _loginAck
000007fedf5698b0  4001271       a8 ...ent.SqlCredential  0 instance 0000000000000000 _credential
000007fee1cf4f30  4001272       89       System.Boolean  1 instance                1 _fConnectionOpen
000007fee1cf4f30  4001273       8a       System.Boolean  1 instance                1 _fResetConnection
000007fee1cf3658  4001274       b0        System.String  0 instance 00000003c00432b8 _originalDatabase
000007fee1cf3658  4001275       b8        System.String  0 instance 0000000000000000 _currentFailoverPartner
000007fee1cf3658  4001276       c0        System.String  0 instance 00000003c0043310 _originalLanguage
000007fee1cf3658  4001277       c8        System.String  0 instance 00000003c0043310 _currentLanguage
000007fee1cf6408  4001278      100         System.Int32  1 instance             8000 _currentPacketSize
000007fee1cf6408  4001279      104         System.Int32  1 instance                0 _asyncCommandCount
000007fee1cf3658  400127a       d0        System.String  0 instance 00000002ffcd1420 _instanceName
000007fedf56ef90  400127b       d8 ...ctionPoolIdentity  0 instance 000000047fd3a240 _identity
000007fedf56f4d8  400127c       e0 ...Tds+SyncAsyncLock  0 instance 00000003c00426f8 _parserLock
000007fee1cf6408  400127d      108         System.Int32  1 instance               -1 _threadIdOwningParserLock
000007fedf56a818  400127e       e8 ...eoutErrorInternal  0 instance 00000003c0042780 timeoutErrorInternal
000007fee16612d0  400127f       f0 ...rence, mscorlib]]  0 instance 0000000000000000 _preparedCommands
000007fee1cf0da8  4001280      110          System.Guid  1 instance 00000003c00426e0 _clientConnectionId
000007fedf5860d0  4001281       f8 ...lient.RoutingInfo  0 instance 0000000000000000 _routingInfo

0:079> !do 00000003c0042850
Name:        System.Data.SqlClient.TdsParser
MethodTable: 000007fedf569ad8
EEClass:     000007fedf3cf318
Size:        160(0xa0) bytes
File:        D:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fee1cf6408  40014d6       70         System.Int32  1 instance              864 _objectID
000007fedf56dcc0  40014d8        8 ...ParserStateObject  0 instance 00000003c00428f0 _physicalStateObj
000007fedf56dcc0  40014d9       10 ...ParserStateObject  0 instance 0000000000000000 _pMarsPhysicalConObj
000007fedf589758  40014da       74         System.Int32  1 instance                2 _state
000007fee1cf3658  40014db       18        System.String  0 instance 000000047fd5ee80 _server
000007fee1cf4f30  40014dc       88       System.Boolean  1 instance                0 _fResetConnection
000007fee1cf4f30  40014dd       89       System.Boolean  1 instance                0 _fPreserveTransaction
000007fedf569618  40014de       20 ...ient.SqlCollation  0 instance 00000003c00432f8 _defaultCollation
000007fee1cf6408  40014df       78         System.Int32  1 instance             1252 _defaultCodePage
000007fee1cf6408  40014e0       7c         System.Int32  1 instance             1033 _defaultLCID
000007fee1cf7528  40014e1       28 System.Text.Encoding  0 instance 000000047fd3e630 _defaultEncoding
000007fedf57b7e8  40014e3       80         System.Int32  1 instance                0 _encryptionOption
000007fedf58c368  40014e4       30 ...ternalTransaction  0 instance 0000000000000000 _currentTransaction
000007fedf58c368  40014e5       38 ...ternalTransaction  0 instance 0000000000000000 _pendingTransaction
000007fee1ce2060  40014e6       68         System.Int64  1 instance 0 _retainedTransactionId
000007fee1cf6408  40014e7       84         System.Int32  1 instance                0 _nonTransactedOpenResultCount
000007fedf56e8a8  40014e8       40 ...rnalConnectionTds  0 instance 00000003c00425d0 _connHandler
000007fee1cf4f30  40014e9       8a       System.Boolean  1 instance                0 _fMARS
000007fee1cf4f30  40014ea       8b       System.Boolean  1 instance                0 _loginWithFailover
000007fee1cf0918  40014eb       48 ...ng.AutoResetEvent  0 instance 0000000000000000 _resetConnectionEvent
000007fedf584368  40014ec       50 ...ParserSessionPool  0 instance 0000000000000000 _sessionPool
000007fee1cf4f30  40014ed       8c       System.Boolean  1 instance                1 _isShiloh
000007fee1cf4f30  40014ee       8d       System.Boolean  1 instance                1 _isShilohSP1
000007fee1cf4f30  40014ef       8e       System.Boolean  1 instance                1 _isYukon
000007fee1cf4f30  40014f0       8f       System.Boolean  1 instance                1 _isKatmai
000007fee1cf4f30  40014f1       90       System.Boolean  1 instance                0 _isDenali
000007fee1cf77d0  40014f2       58        System.Byte[]  0 instance 00000003c0042be8 _sniSpnBuffer
000007fedf5863e0  40014f3       60 ...ent.SqlStatistics  0 instance 0000000000000000 _statistics
000007fee1cf4f30  40014f4       91       System.Boolean  1 instance                0 _statisticsIsInTransaction
000007fee1cf4f30  40014fb       92       System.Boolean  1 instance                0 _asyncWrite
000007fee1cf6408  40014d5      a40         System.Int32  1   shared           static _objectTypeCount
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:964 <<
000007fee1ce8c48  40014d7     1100 ...eading.Tasks.Task  0   shared           static completedTask
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:0000000000000000 <<
000007fedf57b7e8  40014e2      a44         System.Int32  1   shared           static _sniSupportedEncryptionOption
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:0 <<
000007fee1cf77d0  40014f5     1108        System.Byte[]  0   shared           static s_nicAddress
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd3de68 <<
000007fee1cf4f30  40014f6      a4c       System.Boolean  1   shared           static s_fSSPILoaded
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:1 <<
000007fee1cdffa0  40014f7      a48        System.UInt32  1   shared           static s_maxSSPILength
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:12256 <<
000007fee1cf77d0  40014f8     1110        System.Byte[]  0   shared           static s_longDataHeader
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd3cf80 <<
000007fee1cf3c38  40014f9     1118        System.Object  0   shared           static s_tdsParserLock
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd3cfb8 <<
000007fee1cf77d0  40014fa     1120        System.Byte[]  0   shared           static s_xmlMetadataSubstituteSequence
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd3cfd0 <<
000007fedf3905c8  40014fc     1128 ...rd, System.Data]]  0   shared           static __tvpEmptyValue
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd3d038 <<

0:079> !do 00000003c00428f0
Name:        System.Data.SqlClient.TdsParserStateObject
MethodTable: 000007fedf56dcc0
EEClass:     000007fedf3d09b8
Size:        424(0x1a8) bytes
File:        D:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fee1cf6408  40015a4      128         System.Int32  1 instance              864 _objectID
000007fedf569ad8  40015a5        8 ...lClient.TdsParser  0 instance 00000003c0042850 _parser
000007fedf569528  40015a6       10 ...lClient.SNIHandle  0 instance 00000003c0043150 _sessionHandle
000007fee1cf0850  40015a7       18 System.WeakReference  0 instance 00000003c0042a98 _owner
000007fee1cf6408  40015a8      12c         System.Int32  1 instance                0 _activateCount
000007fee1cf6408  40015a9      130         System.Int32  1 instance                8 _inputHeaderLen
000007fee1cf6408  40015aa      134         System.Int32  1 instance                8 _outputHeaderLen
000007fee1cf77d0  40015ab       20        System.Byte[]  0 instance 00000003c0045300 _outBuff
000007fee1cf6408  40015ac      138         System.Int32  1 instance                8 _outBytesUsed
000007fee1cf77d0  40015ad       28        System.Byte[]  0 instance 00000003c00433a8 _inBuff
000007fee1cf6408  40015ae      13c         System.Int32  1 instance              676 _inBytesUsed
000007fee1cf6408  40015af      140         System.Int32  1 instance              676 _inBytesRead
000007fee1cf6408  40015b0      144         System.Int32  1 instance                0 _inBytesPacket
000007fee1cf5ab8  40015b1      170          System.Byte  1 instance                3 _outputMessageType
000007fee1cf5ab8  40015b2      171          System.Byte  1 instance                1 _messageStatus
000007fee1cf5ab8  40015b3      172          System.Byte  1 instance                1 _outputPacketNumber
000007fee1cf4f30  40015b4      173       System.Boolean  1 instance                0 _pendingData
000007fee1cf4f30  40015b5      174       System.Boolean  1 instance                0 _fResetEventOwned
000007fee1cf4f30  40015b6      175       System.Boolean  1 instance                0 _fResetConnectionSent
000007fee1cf4f30  40015b7      176       System.Boolean  1 instance                0 _errorTokenReceived
000007fee1cf4f30  40015b8      177       System.Boolean  1 instance                0 _bulkCopyOpperationInProgress
000007fee1cf4f30  40015b9      178       System.Boolean  1 instance                0 _bulkCopyWriteTimeout
000007fedf56ece8  40015ba       30 ...lClient.SNIPacket  0 instance 00000003c0047258 _sniPacket
000007fedf56ece8  40015bb       38 ...lClient.SNIPacket  0 instance 00000001ffdc1038 _sniAsyncAttnPacket
000007fedf56e0c8  40015bc       40 ....WritePacketCache  0 instance 00000003c0042ab0 _writePacketCache
000007fedf387940  40015bd       48 ...et, System.Data]]  0 instance 00000003c0042af8 _pendingWritePackets
000007fee1cf3c38  40015be       50        System.Object  0 instance 00000003c0042b48 _writePacketLockObject
000007fee1cf21b0  40015bf      188 ...Services.GCHandle  1 instance 00000003c0042a78 _gcHandle
000007fee1cf6408  40015c0      148         System.Int32  1 instance                1 _pendingCallbacks
000007fee1ce2060  40015c1      100         System.Int64  1 instance 3600000 _timeoutMilliseconds
000007fee1ce2060  40015c2      108         System.Int64  1 instance 0 _timeoutTime
000007fee1cf4f30  40015c3      179       System.Boolean  1 instance                0 _attentionSent
000007fee1cf4f30  40015c4      17a       System.Boolean  1 instance                0 _attentionReceived
000007fee1cf4f30  40015c5      17b       System.Boolean  1 instance                0 _attentionSending
000007fee1cf4f30  40015c6      17c       System.Boolean  1 instance                0 _internalTimeout
000007fee1ce2060  40015c7      110         System.Int64  1 instance 634759841647582060 _lastSuccessfulIOTime
000007fee1cbeb28  40015c8       58      System.Object[]  0 instance 00000003c0042b60 _securePasswords
000007fee1cf63a8  40015c9       60       System.Int32[]  0 instance 00000003c0042b90 _securePasswordOffsetsInBuffer
000007fee1cf4f30  40015ca      17d       System.Boolean  1 instance                0 _cancelled
000007fee1cf6408  40015cb      14c         System.Int32  1 instance               -1 _allowObjectID
000007fee1cf4f30  40015cc      17e       System.Boolean  1 instance                0 _hasOpenResult
000007fedf58c368  40015cd       68 ...ternalTransaction  0 instance 0000000000000000 _executedUnderTransaction
000007fee1cd3ea8  40015ce      118        System.UInt64  1 instance 0 _longlen
000007fee1cd3ea8  40015cf      120        System.UInt64  1 instance 0 _longlenleft
000007fee1cf63a8  40015d0       70       System.Int32[]  0 instance 0000000000000000 _decimalBits
000007fee1cf77d0  40015d1       78        System.Byte[]  0 instance 00000004bff10550 _bTmp
000007fee1cf6408  40015d2      150         System.Int32  1 instance                0 _bTmpRead
000007fee1cd5ab0  40015d3       80  System.Text.Decoder  0 instance 0000000000000000 _plpdecoder
000007fee1cf4f30  40015d4      17f       System.Boolean  1 instance                0 _accumulateInfoEvents
000007fedf39a538  40015d5       88 ...or, System.Data]]  0 instance 0000000000000000 _pendingInfoEvents
000007fee1cf77d0  40015d6       90        System.Byte[]  0 instance 00000003c0042bb0 _partialHeaderBuffer
000007fee1cf6408  40015d7      154         System.Int32  1 instance                0 _partialHeaderBytesRead
000007fedf56c2d8  40015d8       98 ...t._SqlMetaDataSet  0 instance 00000004c0284568 _cleanupMetaData
000007fedf5828e0  40015d9       a0 ...DataSetCollection  0 instance 0000000000000000 _cleanupAltMetaDataSetArray
000007fee1cf6408  40015da      158         System.Int32  1 instance                0 _tracePasswordOffset
000007fee1cf6408  40015db      15c         System.Int32  1 instance                0 _tracePasswordLength
000007fee1cf6408  40015dc      160         System.Int32  1 instance                0 _traceChangePasswordOffset
000007fee1cf6408  40015dd      164         System.Int32  1 instance                0 _traceChangePasswordLength
000007fee1cf4f30  40015de      180       System.Boolean  1 instance                0 _receivedColMetaData
000007fedf571848  40015df      168         System.Int32  1 instance               11 _sniContext
000007fee1cf4f30  40015e0      181       System.Boolean  1 instance                0 _bcpLock
000007fedf56f070  40015e1      190 ...Object+NullBitmap  1 instance 00000003c0042a80 _nullBitmapInfo
000007fee1cfc0e0  40015e2       a8 ...bject, mscorlib]]  0 instance 0000000000000000 _networkPacketTaskSource
000007fee1cf1fe8  40015e3       b0 ...m.Threading.Timer  0 instance 0000000000000000 _networkPacketTimeout
000007fee1cf4f30  40015e4      182       System.Boolean  1 instance                1 _syncOverAsync
000007fee1cf4f30  40015e5      183       System.Boolean  1 instance                0 _snapshotReplay
000007fedf5729f0  40015e6       b8 ...ect+StateSnapshot  0 instance 0000000000000000 _snapshot
000007fee1cf1e50  40015e7       c0 ....ExecutionContext  0 instance 0000000000000000 _executionContext
000007fee1cf4f30  40015e8      184       System.Boolean  1 instance                0 _asyncReadWithoutSnapshot
000007fedf56c1b8  40015e9       c8 ...qlErrorCollection  0 instance 0000000000000000 _errors
000007fedf56c1b8  40015ea       d0 ...qlErrorCollection  0 instance 0000000000000000 _warnings
000007fee1cf3c38  40015eb       d8        System.Object  0 instance 00000003c0042bd0 _errorAndWarningsLock
000007fee1cf4f30  40015ec      185       System.Boolean  1 instance                0 _hasErrorOrWarning
000007fedf56c1b8  40015ed       e0 ...qlErrorCollection  0 instance 0000000000000000 _preAttentionErrors
000007fedf56c1b8  40015ee       e8 ...qlErrorCollection  0 instance 0000000000000000 _preAttentionWarnings
000007fee1cfc0e0  40015ef       f0 ...bject, mscorlib]]  0 instance 0000000000000000 _writeCompletionSource
000007fee1cf6408  40015f0      16c         System.Int32  1 instance                0 _asyncWriteCount
000007fee1cf3888  40015f1       f8     System.Exception  0 instance 0000000000000000 _delayedWriteAsyncCallbackException
000007fee1cf4f30  40015f2      186       System.Boolean  1 instance                0 _reading
000007fee1cf6408  40015a3      a54         System.Int32  1   shared           static _objectTypeCount
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:NotInit  <<

_outBuff is our output buffer, i.e. what we sent to the server, and it has our command text... Now we know who's leaking!

0:079> !do 00000003c0045300
Name:        System.Byte[]
MethodTable: 000007fee1cf77d0
EEClass:     000007fee16fa4b8
Size:        8024(0x1f58) bytes
Array:       Rank 1, Number of elements 8000, Type Byte
Content:     ...6............................G.e.t.C.o.n.s.t.a.n.t.S.e.t.s....@.p.a.r.t.i.t.i.o.n.I.d..&.......@.u.s.e.r.S.I.D...X.....4X.S.-
Fields:
None

1. Baseline: Normal WAN (1Gbps)

2. Comparison Run: Slow WAN (300 Kbps).

As expected, almost all tests are slower due to slower network. But overall perf are still pretty good for these small operations.

Microsoft.TeamFoundation.TeamFoundationServiceUnavailableException: TF31002: Unable to connect to this Team Foundation Server: https://TfsServerName:8080/.

Team Foundation Server Url: https://msTfsServerName:8080/.

 Possible reasons for failure include:

- The name, port number, or protocol for the Team Foundation Server is incorrect.

- The Team Foundation Server is offline.

- The password has expired or is incorrect.

Technical information (for administrator):

The request failed with HTTP status 404: Not Found. ---> System.Net.WebException: The request failed with HTTP status 404: Not Found.

at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)

at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)

at Microsoft.TeamFoundation.Proxy.BisRegistrationServiceProxyWsdl.GetRegistrationEntries(String toolId)

at Microsoft.TeamFoundation.Proxy.BisRegistrationProxy.GetRegistrationEntries(String toolId)

at Microsoft.TeamFoundation.Proxy.BisRegistrationService.RefreshMemoryCache()

at Microsoft.TeamFoundation.Proxy.BisRegistrationService.RefreshCachesIfNeeded(Boolean direct)

at Microsoft.TeamFoundation.Proxy.BisRegistrationService.GetRegistrationEntries(String toolId)

at Microsoft.TeamFoundation.Framework.Client.PreFrameworkServerDataProvider.FindServiceLocation(String serviceType, String toolId)

at Microsoft.TeamFoundation.Framework.Client.PreFrameworkServerDataProvider.LocationForCurrentConnection(String serviceType, Guid serviceIdentifier)

Across a “normal” mix of operations, we see about a 25-30% improvement in overall performance.  This is the throughput comparison result for 4 load run configs:

There are 79 perf benchmark tests that existed in both TFS 2008 and TFS 2010 and our testing shows that almost 50% (37 out of 79) of them improved 25% or more in TFS 2010. This is the detailed perf result comparison for these 74 perf scenarios:

One issue we found is that on config #3 and config #4, a lot of tests that access TFS reports timed out. The issue is that AS has four thread pools. Cube processing uses threads from the Processing thread pool. But queries also need to use threads from this thread pool. So if AS is starved for threads, you can get very poor query response time during cube processing. Increasing the max size of this thread pool prevents this starvation, and allows queries to continue running without much degradation in performance during cube processing. There is a document on MSDN called Analysis Services Performance Guide that includes the follow suggestion on setting this value:

The best way to see this is to use Performance Monitor to watch these two performance counters and increase the number of threads so it's just a little bit higher than what's being used. We tried this on two machines. Pioneer had MaxThreads set to 120. Setting it to 150 is high enough to allow queries to run and not get blocked while processing the cube. On another server, the default was 80, and setting it to 100 was enough of this machine.

You can follow these simple steps to configure your AS properly:

• First measure how many threads are being used during cube processing:
  • Open the Performance (Server 2003) or Performance Monitor (Server 2008) application
  • Remove all counters
  • Add a performance counter
    • Click on the Add button
    • Open the MSAS 2008:Threads section
    • Add the Processing pool busy threads performance count
  • During cube processing, see how high this goes.
• Check/change the thread limit:
  • Open SQL Server Management Studio
  • Connect to your Analysis Services machine
  • Right click on the AS instance (not the database) and click Properties. This displays the Analysis Services Properties dialog box
  • Check the box Show Advanced (All) Properties
  • Near the very bottom is a property called ThreadPool \ Process \ MaxThreads. If this value is close to the maximum number of threads used during cube processing, increase the value in the Value column and try again. You want this number to be high enough so cube processing doesn't or only very briefly reaches this limit. You might start by increasing it 10%.

Have a good day.

 Jiange

1. In Windows Explorer, delete some files. These files seem to go away

2. Check available free space, it didn't show any increase.

3. In the Windows Explorer, cilck on refresh, these "deleted" files are showing up again.

I tried to bing this issue but found no hits. I eventually just rebooted the machine and it took a long time for ther reboot to completes. But the good news is that after the reboot, I'm able to deleted unwanted files again on this Win2k8 server to free up more spaces. I have not figure out what's the root cause for this behavior yet though.

Jiange

1. Is this a native memory leak or .NET memory leak?

0:072> !address -summary

-------------------- Usage SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots) Pct(Busy)   Usage
   390e8000 (  934816) : 44.58%    65.56%    : RegionUsageIsVAD
   28f79000 (  671204) : 32.01%    00.00%    : RegionUsageFree
    9c2f000 (  159932) : 07.63%    11.22%    : RegionUsageImage
    30cd000 (   49972) : 02.38%    03.50%    : RegionUsageStack
      49000 (     292) : 00.01%    00.02%    : RegionUsageTeb
   11246000 (  280856) : 13.39%    19.70%    : RegionUsageHeap
          0 (       0) : 00.00%    00.00%    : RegionUsagePageHeap
       1000 (       4) : 00.00%    00.00%    : RegionUsagePeb
       1000 (       4) : 00.00%    00.00%    : RegionUsageProcessParametrs
       2000 (       8) : 00.00%    00.00%    : RegionUsageEnvironmentBlock
       Tot: 7fff0000 (2097088 KB) Busy: 57077000 (1425884 KB)

-------------------- Type SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots)  Usage
   28f79000 (  671204) : 32.01%   : <free>
    9eb7000 (  162524) : 07.75%   : MEM_IMAGE
    1c9c000 (   29296) : 01.40%   : MEM_MAPPED
   4b524000 ( 1234064) : 58.85%   : MEM_PRIVATE

-------------------- State SUMMARY --------------------------
    TotSize (      KB)   Pct(Tots)  Usage
   407c5000 ( 1056532) : 50.38%   : MEM_COMMIT
   28f79000 (  671204) : 32.01%   : MEM_FREE
   168b2000 (  369352) : 17.61%   : MEM_RESERVE

RegionUsageIsVAD is big and VirtualAlloc goes here (e.g., .NET heap).

0:072> !EEHeap -gc
Number of GC Heaps: 4
------------------------------
...
------------------------------
GC Heap Size:            Size: 0x2611db9c (638704540) bytes.

Ok, .NET memory usage is pretty high at 630 MB.

2. Which .NET objects are we leaking

 0:072> !dumpheap -stat
------------------------------
Heap 0
total 0 objects
------------------------------
Heap 1
total 0 objects
------------------------------
Heap 2
total 0 objects
------------------------------
Heap 3
total 0 objects
------------------------------
total 0 objects
Statistics:
      MT    Count    TotalSize Class Name
7aef7c48        1           12 System.Collections.Generic.SortedList`2+KeyList[[System.String, mscorlib],[System.Object, mscorlib]]
...

79b57ee8    84541    117150884 System.Object[]
1d6af048 15013412    420375536 Microsoft.VisualStudio.TestTools.Common.TimerResult

Ok, there are 15 millions of Microsoft.VisualStudio.TestTools.Common.TimerResult objects. !gcroot confirmed they are rooted.

3. Where are we leaking it? 

Use "!dumpheap -mt" and "!do" to examine a few of them:

0:072> !dumpheap -mt 1d6af048
------------------------------
Heap 0
 Address       MT     Size
009aa8e8 1d6af048       28    
009aabf8 1d6af048       28    

...

0133f0e8 1d6af048       28    
0133f104 1d6af048       28

 072> !dumpobj 0133f0e8
Name:        Microsoft.VisualStudio.TestTools.Common.TimerResult
MethodTable: 1d6af048
EEClass:     1e969b88
Size:        28(0x1c) bytes
File:        D:\WINDOWS\assembly\GAC_MSIL\Microsoft.VisualStudio.QualityTools.Common\10.0.0.0__b03f5f7f11d50a3a\Microsoft.VisualStudio.QualityTools.Common.dll
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
79b9d470  40003d2        4        System.String  0 instance 08aa3eb4 m_name
79ba5af0  40003d3        8      System.DateTime  1 instance 0133f0f0 m_startTime
79ba4a94  40003d4       10      System.TimeSpan  1 instance 0133f0f8 m_duration
0:072> !dumpobj -nofields 08aa3eb4
Name:        System.String
MethodTable: 79b9d470
EEClass:     798d5ec0
Size:        58(0x3a) bytes
File:        D:\WINDOWS\Microsoft.Net\assembly\GAC_32\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String:      _TWI_GetRandomWorkItem

Almost all TimerResult have the same name "_TWI_GetRandomWorkItem". And this is enough for us to track down the "leak". It turns out that in our test code, we have a poorly written algorithm to get a random sequence of items from a list. And it might leads to a "infinite" loop and keeps calling an method that calls TestContext.BeginTimer/TestContext.EndTimer. So each call will create a new Timer Result object. This will eventually cause the app to crash due to OutOfMemoryException.

We can also drill down further on these 15 million objects. Are they stored in a few collections? Yes, 4 big object[] are holding about 11 million of them.

0:022> !dumpheap -mt 79b57ee8 -min 1000000
------------------------------
Heap 0
 Address       MT     Size
11863098 79b57ee8 16777232    
total 0 objects
------------------------------
Heap 1
 Address       MT     Size
129d3cf8 79b57ee8 30433544    
58fc0038 79b57ee8 33554448    
total 0 objects
------------------------------
Heap 2
 Address       MT     Size
54420038 79b57ee8 33554448    
total 0 objects
------------------------------
Heap 3
 Address       MT     Size
total 0 objects
------------------------------
total 0 objects
Statistics:
      MT    Count    TotalSize Class Name
79b57ee8        4    114319672 System.Object[]
Total 4 objects

Spot check some entries in these huge object[] and verified that they are the same TimerResult objects with the same name _TWI_GetRandomWorkItem:

0:022> !dumparray -start 10000 -length 2 -details 11863098
Name:        Microsoft.VisualStudio.TestTools.Common.TimerResult[]
MethodTable: 79b57ee8
EEClass:     798d69a8
Size:        16777232(0x1000010) bytes
Array:       Rank 1, Number of elements 4194304, Type CLASS
Element Methodtable: 1d6af048
[10000] 0c7fab28
    Name:        Microsoft.VisualStudio.TestTools.Common.TimerResult
    MethodTable: 1d6af048
    EEClass:     1e969b88
    Size:        28(0x1c) bytes
    File:        D:\WINDOWS\assembly\GAC_MSIL\Microsoft.VisualStudio.QualityTools.Common\10.0.0.0__b03f5f7f11d50a3a\Microsoft.VisualStudio.QualityTools.Common.dll
    Fields:
              MT    Field   Offset                 Type VT     Attr    Value Name
        79b9d470  40003d2        4            System.String      0     instance     08aa3eb4     m_name
        79ba5af0  40003d3        8          System.DateTime      1     instance     0c7fab30     m_startTime
        79ba4a94  40003d4       10          System.TimeSpan      1     instance     0c7fab38     m_duration
[10001] 0c7fab44
    Name:        Microsoft.VisualStudio.TestTools.Common.TimerResult
    MethodTable: 1d6af048
    EEClass:     1e969b88
    Size:        28(0x1c) bytes
    File:        D:\WINDOWS\assembly\GAC_MSIL\Microsoft.VisualStudio.QualityTools.Common\10.0.0.0__b03f5f7f11d50a3a\Microsoft.VisualStudio.QualityTools.Common.dll
    Fields:
              MT    Field   Offset                 Type VT     Attr    Value Name
        79b9d470  40003d2        4            System.String      0     instance     08aa3eb4     m_name
        79ba5af0  40003d3        8          System.DateTime      1     instance     0c7fab4c     m_startTime
        79ba4a94  40003d4       10          System.TimeSpan      1     instance     0c7fab54     m_duration

0:022> !do -nofields 08aa3eb4    
Name:        System.String
MethodTable: 79b9d470
EEClass:     798d5ec0
Size:        58(0x3a) bytes
File:        D:\WINDOWS\Microsoft.Net\assembly\GAC_32\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String:      _TWI_GetRandomWorkItem

Anyway, we fixed the issue by switching to a better algorithm of picking a random sequence of items from a collection.

Have a good day.

Jiange