Share via


A case study on System.InvalidOperationException due to SQLConnection leak

I ran into this exception below while I was load testing TFS last week:

System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. 

This may have occurred because all pooled connections were in use and max pool size was reached.

 

Basically this means that the connection pool is full and that all connections are in use.

1. Verified SQL connections are indeed leaked

I used Perf counter ".Net Data Provider for SqlServer"\NumberOfRelaimedConnections to verify we indeed have a leak. This perf counter keeps going up during my load run, which means that a lot of SQL connections are not closed properly by our code.

2. Verify SQL connection pool is maxed out

0:079> !dumpheap -stat -type System.Data.ProviderBase.DbConnectionPool
Statistics:
              MT    Count    TotalSize Class Name
000007fedf56fe00        1           24 System.Collections.Generic.ObjectEqualityComparer`1[[System.Data.ProviderBase.DbConnectionPoolIdentity, System.Data]]
...
000007fedf56ffd8        2          192 System.Collections.Generic.Dictionary`2+Entry[[System.Data.Common.DbConnectionPoolKey, System.Data],[System.Data.ProviderBase.DbConnectionPoolGroup, System.Data]][]
000007fedf56fe88        4          192 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Data.ProviderBase.DbConnectionPoolIdentity, System.Data],[System.Data.ProviderBase.DbConnectionPool, System.Data]]
000007fedf386ec0        4          224 System.Collections.Concurrent.ConcurrentDictionary`2[[System.Data.ProviderBase.DbConnectionPoolIdentity, System.Data],[System.Data.ProviderBase.DbConnectionPool, System.Data]]
000007fedf56ed90       14          336 System.Data.ProviderBase.DbConnectionPoolCounters+Counter
000007fedf56bfc8        2          336 System.Data.ProviderBase.DbConnectionPool
000007fee1cbeb28       10         2944 System.Object[]
Total 63 objects

0:079> !dumpheap -mt 000007fedf56bfc8       
         Address               MT     Size
000000047fd3a260 000007fedf56bfc8      168    
000000047fd5f468 000007fedf56bfc8      168    

Statistics:
              MT    Count    TotalSize Class Name
000007fedf56bfc8        2          336 System.Data.ProviderBase.DbConnectionPool
Total 2 objects

0:079> !do 000000047fd3a260
Name:        System.Data.ProviderBase.DbConnectionPool
MethodTable: 000007fedf56bfc8
EEClass:     000007fedf3cfd48
Size:        168(0xa8) bytes
File:        D:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fee1cf6408  4000e70       80         System.Int32  1 instance           190000 _cleanupWait
000007fedf56ef90  4000e71        8 ...ctionPoolIdentity  0 instance 000000047fd3a240 _identity
000007fedf56a3c8  4000e72       10 ...ConnectionFactory  0 instance 000000047fd31438 _connectionFactory
000007fedf56df40  4000e73       18 ...nnectionPoolGroup  0 instance 000000047fd36d10 _connectionPoolGroup
000007fedf56cec0  4000e74       20 ...nPoolGroupOptions  0 instance 000000047fd36ce8 _connectionPoolGroupOptions
000007fedf57e7a8  4000e75       28 ...nPoolProviderInfo  0 instance 0000000000000000 _connectionPoolProviderInfo
000007fedf577368  4000e76       84         System.Int32  1 instance                1 _state
000007fedf387188  4000e77       30 ...al, System.Data]]  0 instance 000000047fd3a3e8 _stackOld
000007fedf387188  4000e78       38 ...al, System.Data]]  0 instance 000000047fd3a408 _stackNew
000007fedf387240  4000e79       40 ...on, System.Data]]  0 instance 000000047fd3a428 _pendingOpens
000007fee1cf6408  4000e7a       88         System.Int32  1 instance                0 _pendingOpensWaiting
000007fee1ce3b90  4000e7b       48 ...ding.WaitCallback  0 instance 000000047fd3add0 _poolCreateRequest
000007fee1cf6408  4000e7c       8c         System.Int32  1 instance                0 _waitCount
000007fedf568ff0  4000e7d       50 ...l+PoolWaitHandles  0 instance 000000047fd3a708 _waitHandles
000007fee1cf3888  4000e7e       58     System.Exception  0 instance 0000000000000000 _resError
000007fee1cf4f30  4000e7f       9c       System.Boolean  1 instance                0 _errorOccurred
000007fee1cf6408  4000e80       90         System.Int32  1 instance             5000 _errorWait
000007fee1cf1fe8  4000e81       60 ...m.Threading.Timer  0 instance 0000000000000000 _errorTimer
000007fee1cf1fe8  4000e82       68 ...m.Threading.Timer  0 instance 000000047fd3aef0 _cleanupTimer
000007fedf569850  4000e83       70 ...tedConnectionPool  0 instance 000000047fd3abb8 _transactedConnectionPool
000007fedf387308  4000e84       78 ...al, System.Data]]  0 instance 000000047fd3a850 _objectList
000007fee1cf6408  4000e85       94         System.Int32  1 instance                7 _totalObjects
000007fee1cf6408  4000e87       98         System.Int32  1 instance                1 _objectID
000007fee1ce4810  4000e6f      b88        System.Random  0   shared           static _random
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd3a5f0 <<
000007fee1cf6408  4000e86      9e0         System.Int32  1   shared           static _objectTypeCount
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:4 <<

Ok, this Connection pool is not maxed out. Check next one.

0:079> !do 00000002bfe10368   
Free Object
Size:        638840(0x9bf78) bytes
0:079> !do 000000047fd5f468
Name:        System.Data.ProviderBase.DbConnectionPool
MethodTable: 000007fedf56bfc8
EEClass:     000007fedf3cfd48
Size:        168(0xa8) bytes
File:        D:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fee1cf6408  4000e70       80         System.Int32  1 instance           170000 _cleanupWait
000007fedf56ef90  4000e71        8 ...ctionPoolIdentity  0 instance 000000047fd3a240 _identity
000007fedf56a3c8  4000e72       10 ...ConnectionFactory  0 instance 000000047fd31438 _connectionFactory
000007fedf56df40  4000e73       18 ...nnectionPoolGroup  0 instance 000000047fd5f090 _connectionPoolGroup
000007fedf56cec0  4000e74       20 ...nPoolGroupOptions  0 instance 000000047fd5f068 _connectionPoolGroupOptions
000007fedf57e7a8  4000e75       28 ...nPoolProviderInfo  0 instance 0000000000000000 _connectionPoolProviderInfo
000007fedf577368  4000e76       84         System.Int32  1 instance                1 _state
000007fedf387188  4000e77       30 ...al, System.Data]]  0 instance 000000047fd5f510 _stackOld
000007fedf387188  4000e78       38 ...al, System.Data]]  0 instance 000000047fd5f530 _stackNew
000007fedf387240  4000e79       40 ...on, System.Data]]  0 instance 000000047fd5f550 _pendingOpens
000007fee1cf6408  4000e7a       88         System.Int32  1 instance                0 _pendingOpensWaiting
000007fee1ce3b90  4000e7b       48 ...ding.WaitCallback  0 instance 000000047fd5fc40 _poolCreateRequest
000007fee1cf6408  4000e7c       8c         System.Int32  1 instance               20 _waitCount
000007fedf568ff0  4000e7d       50 ...l+PoolWaitHandles  0 instance 000000047fd5f718 _waitHandles
000007fee1cf3888  4000e7e       58     System.Exception  0 instance 0000000000000000 _resError
000007fee1cf4f30  4000e7f       9c       System.Boolean  1 instance                0 _errorOccurred
000007fee1cf6408  4000e80       90         System.Int32  1 instance             5000 _errorWait
000007fee1cf1fe8  4000e81       60 ...m.Threading.Timer  0 instance 0000000000000000 _errorTimer
000007fee1cf1fe8  4000e82       68 ...m.Threading.Timer  0 instance 000000047fd5fcc0 _cleanupTimer
000007fedf569850  4000e83       70 ...tedConnectionPool  0 instance 000000047fd5fbc8 _transactedConnectionPool
000007fedf387308  4000e84       78 ...al, System.Data]]  0 instance 000000047fd5f860 _objectList
000007fee1cf6408  4000e85       94         System.Int32  1 instance              100 _totalObjects
000007fee1cf6408  4000e87       98         System.Int32  1 instance                3 _objectID
000007fee1ce4810  4000e6f      b88        System.Random  0   shared           static _random
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd3a5f0 <<
000007fee1cf6408  4000e86      9e0         System.Int32  1   shared           static _objectTypeCount
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:4 <<

This connection pool appears to be maxed out (default pool size is 100. Check _connectionPoolGroupOptions to see if default pool size is used).

0:079> !do 000000047fd5f068
Name:        System.Data.ProviderBase.DbConnectionPoolGroupOptions
MethodTable: 000007fedf56cec0
EEClass:     000007fedf3d0438
Size:        40(0x28) bytes
File:        D:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fee1cf4f30  4000ed4       14       System.Boolean  1 instance                1 _poolByIdentity
000007fee1cf6408  4000ed5        8         System.Int32  1 instance                0 _minPoolSize
000007fee1cf6408  4000ed6        c         System.Int32  1 instance              100 _maxPoolSize
000007fee1cf6408  4000ed7       10         System.Int32  1 instance            15000 _creationTimeout
000007fee1cf0658  4000ed8       18      System.TimeSpan  1 instance 000000047fd5f080 _loadBalanceTimeout
000007fee1cf4f30  4000ed9       15       System.Boolean  1 instance                1 _hasTransactionAffinity
000007fee1cf4f30  4000eda       16       System.Boolean  1 instance                0 _useLoadBalancing

Yep, one of the SQL connection pool is indeed maxed out.

 

3. Figure out which SQL connections are leaked and who is leaking them

Leaked SQL Connections are not GC rooted and their _closedCount field is set to 0.

0:079> !dumpheap -stat -type System.Data.SqlClient.SqlConnection
Statistics:
              MT    Count    TotalSize Class Name
000007fedf56cac8        1           64 System.Data.SqlClient.SqlConnectionFactory
000007fedf3a5ce0        1           80 System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[System.Data.SqlClient.SqlConnectionStringBuilder+Keywords, System.Data]]
000007fedf56ccc0        2          112 System.Data.SqlClient.SqlConnectionPoolGroupProviderInfo
000007fedf56ae10        2          400 System.Data.SqlClient.SqlConnectionString
000007fedfb11a88        1         1440 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[System.Data.SqlClient.SqlConnectionStringBuilder+Keywords, System.Data]][]
000007fedf56a818      107         4280 System.Data.SqlClient.SqlConnectionTimeoutErrorInternal
000007fee1cbeb28      107        11128 System.Object[]
000007fedf56daf0      749        17976 System.Data.SqlClient.SqlConnectionTimeoutPhaseDuration
000007fedf56a620      559        22360 System.Data.SqlClient.SqlConnectionPoolKey
000007fedf566170      588        75264 System.Data.SqlClient.SqlConnection
000007fedf577a00      557       102488 System.Data.SqlClient.SqlConnectionStringBuilder
Total 2674 objects

Dump out all Sql Connections so we can examine their _closedCount field: (We're looking for the ones with _ClosedCount=0)

0:079> .foreach (conn {!DumpHeap -short -MT 000007fedf566170}) {.echo foo; !do conn}
0000000100030770
Name:        System.Data.SqlClient.SqlConnection
MethodTable: 000007fedf566170
EEClass:     000007fedf3a8110
Size:        128(0x80) bytes
File:        D:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fee1cf3c38  4000198        8        System.Object  0 instance 0000000000000000 __identity
000007fee0d6f2e8  40002d2       10 ...ponentModel.ISite  0 instance 0000000000000000 site
000007fee0d7dbe0  40002d3       18 ....EventHandlerList  0 instance 0000000100032178 events
000007fee1cf3c38  40002d1      1b8        System.Object  0   shared           static EventDisposed
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd55f50 <<
000007fedf581510  40002c0       20 ...hangeEventHandler  0 instance 0000000000000000 _stateChangeEventHandler
000007fedf57e4c0  400107c       28 ...t.SqlDebugContext  0 instance 0000000000000000 _sdc
000007fee1cf4f30  400107d       70       System.Boolean  1 instance                0 _AsyncCommandInProgress
000007fedf5863e0  400107e       30 ...ent.SqlStatistics  0 instance 0000000000000000 _statistics
000007fee1cf4f30  400107f       71       System.Boolean  1 instance                0 _collectstats
000007fee1cf4f30  4001080       72       System.Boolean  1 instance                0 _fireInfoMessageEventOnUserErrors
000007fedfb0de48  4001081       38 ....Task, mscorlib]]  0 instance 0000000000000000 _currentCompletion
000007fedf5698b0  4001082       40 ...ent.SqlCredential  0 instance 0000000000000000 _credential
000007fee1cf3658  4001083       48        System.String  0 instance 00000001000306a8 _connectionString
000007fedf56c0e0  4001086       50 ...ConnectionOptions  0 instance 0000000000000000 _userConnectionOptions
000007fedf56df40  4001087       58 ...nnectionPoolGroup  0 instance 0000000000000000 _poolGroup
000007fedf56d730  4001088       60 ...onnectionInternal  0 instance 000000047fd43d80 _innerConnection
000007fee1cf6408  4001089       68         System.Int32  1 instance                1 _closeCount
000007fee1cf6408  400108b       6c         System.Int32  1 instance           322025 ObjectID
000007fee1cf3c38  400107b      ea0        System.Object  0   shared           static EventInfoMessage
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd31420 <<
000007fedf56a3c8  4001084      ea8 ...ConnectionFactory  0   shared           static _connectionFactory
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd31438 <<
000007fee1ce8290  4001085      eb0 ...eAccessPermission  0   shared           static ExecutePermission
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd352f8 <<
000007fee1cf6408  400108a      a08         System.Int32  1   shared           static _objectTypeCount
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:322560 <<
...
00000004c0280050
Name:        System.Data.SqlClient.SqlConnection
MethodTable: 000007fedf566170
EEClass:     000007fedf3a8110
Size:        128(0x80) bytes
File:        D:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fee1cf3c38  4000198        8        System.Object  0 instance 0000000000000000 __identity
000007fee0d6f2e8  40002d2       10 ...ponentModel.ISite  0 instance 0000000000000000 site
000007fee0d7dbe0  40002d3       18 ....EventHandlerList  0 instance 00000004c02819b8 events
000007fee1cf3c38  40002d1      1b8        System.Object  0   shared           static EventDisposed
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd55f50 <<
000007fedf581510  40002c0       20 ...hangeEventHandler  0 instance 0000000000000000 _stateChangeEventHandler
000007fedf57e4c0  400107c       28 ...t.SqlDebugContext  0 instance 0000000000000000 _sdc
000007fee1cf4f30  400107d       70       System.Boolean  1 instance                0 _AsyncCommandInProgress
000007fedf5863e0  400107e       30 ...ent.SqlStatistics  0 instance 0000000000000000 _statistics
000007fee1cf4f30  400107f       71       System.Boolean  1 instance                0 _collectstats
000007fee1cf4f30  4001080       72       System.Boolean  1 instance                0 _fireInfoMessageEventOnUserErrors
000007fedfb0de48  4001081       38 ....Task, mscorlib]]  0 instance 0000000000000000 _currentCompletion
000007fedf5698b0  4001082       40 ...ent.SqlCredential  0 instance 0000000000000000 _credential
000007fee1cf3658  4001083       48        System.String  0 instance 00000004c027ff88 _connectionString
000007fedf56c0e0  4001086       50 ...ConnectionOptions  0 instance 000000047fd5ed68 _userConnectionOptions
000007fedf56df40  4001087       58 ...nnectionPoolGroup  0 instance 000000047fd5f090 _poolGroup
000007fedf56d730  4001088       60 ...onnectionInternal  0 instance 00000003c00425d0 _innerConnection
000007fee1cf6408  4001089       68         System.Int32  1 instance                0 _closeCount
000007fee1cf6408  400108b       6c         System.Int32  1 instance           322129 ObjectID
000007fee1cf3c38  400107b      ea0        System.Object  0   shared           static EventInfoMessage
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd31420 <<
000007fedf56a3c8  4001084      ea8 ...ConnectionFactory  0   shared           static _connectionFactory
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd31438 <<
000007fee1ce8290  4001085      eb0 ...eAccessPermission  0   shared           static ExecutePermission
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd352f8 <<
000007fee1cf6408  400108a      a08         System.Int32  1   shared           static _objectTypeCount
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:322560 <<

Picked one Sql Connection object that is still open (_closeCount is zero!). Now let's make sure it's unreachable (i.e. leaked!).

0:079> !GCRoot 00000004c08c36c8
Thread 254c:
    0000000014fbc2e0 000007fedf4f4229 System.Data.ProviderBase.DbConnectionPool.TryGetConnection(System.Data.Common.DbConnection, UInt32, Boolean, Boolean, System.Data.Common.DbConnectionOptions, System.Data.ProviderBase.DbConnectionInternal ByRef)
        rbp+c8: 0000000014fbc3d8
            ->  00000004c08c36c8 System.Data.SqlClient.SqlConnection

...

    0000000014fbc6a0 000007fedbb6a55d Microsoft.TeamFoundation.Framework.Server.TeamFoundationSqlResourceComponent.Execute(ExecuteType, System.Data.CommandBehavior)
        rbp+210: 0000000014fbc920
            ->  00000004c08c2b00 Microsoft.TeamFoundation.WorkItemTracking.Server.DalSqlResourceComponent6
            ->  00000004c08c36c8 System.Data.SqlClient.SqlConnection

Found 7 unique roots (run '!GCRoot -all' to see all roots).

Well, that SQL connection is still in-use. (not leaked.) Let's check another one.

0:079> !gcroot 00000004c0280050
Found 0 unique roots (run '!GCRoot -all' to see all roots).

Ok, so it's unreachable and open, it's therefore a leaked connection. We can tell more than just the connection string by dumping the inner connection:

0:079> !do 00000004c0280050
Name:        System.Data.SqlClient.SqlConnection
MethodTable: 000007fedf566170
EEClass:     000007fedf3a8110
Size:        128(0x80) bytes
File:        D:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fee1cf3c38  4000198        8        System.Object  0 instance 0000000000000000 __identity
000007fee0d6f2e8  40002d2       10 ...ponentModel.ISite  0 instance 0000000000000000 site
000007fee0d7dbe0  40002d3       18 ....EventHandlerList  0 instance 00000004c02819b8 events
000007fee1cf3c38  40002d1      1b8        System.Object  0   shared           static EventDisposed
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd55f50 <<
000007fedf581510  40002c0       20 ...hangeEventHandler  0 instance 0000000000000000 _stateChangeEventHandler
000007fedf57e4c0  400107c       28 ...t.SqlDebugContext  0 instance 0000000000000000 _sdc
000007fee1cf4f30  400107d       70       System.Boolean  1 instance                0 _AsyncCommandInProgress
000007fedf5863e0  400107e       30 ...ent.SqlStatistics  0 instance 0000000000000000 _statistics
000007fee1cf4f30  400107f       71       System.Boolean  1 instance                0 _collectstats
000007fee1cf4f30  4001080       72       System.Boolean  1 instance                0 _fireInfoMessageEventOnUserErrors
000007fedfb0de48  4001081       38 ....Task, mscorlib]]  0 instance 0000000000000000 _currentCompletion
000007fedf5698b0  4001082       40 ...ent.SqlCredential  0 instance 0000000000000000 _credential
000007fee1cf3658  4001083       48        System.String  0 instance 00000004c027ff88 _connectionString
000007fedf56c0e0  4001086       50 ...ConnectionOptions  0 instance 000000047fd5ed68 _userConnectionOptions
000007fedf56df40  4001087       58 ...nnectionPoolGroup  0 instance 000000047fd5f090 _poolGroup
000007fedf56d730  4001088       60 ...onnectionInternal  0 instance 00000003c00425d0 _innerConnection
000007fee1cf6408  4001089       68         System.Int32  1 instance                0 _closeCount
000007fee1cf6408  400108b       6c         System.Int32  1 instance           322129 ObjectID
000007fee1cf3c38  400107b      ea0        System.Object  0   shared           static EventInfoMessage
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd31420 <<
000007fedf56a3c8  4001084      ea8 ...ConnectionFactory  0   shared           static _connectionFactory
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd31438 <<
000007fee1ce8290  4001085      eb0 ...eAccessPermission  0   shared           static ExecutePermission
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd352f8 <<
000007fee1cf6408  400108a      a08         System.Int32  1   shared           static _objectTypeCount
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:322560 <<

0:079> !do 00000003c00425d0
Name:        System.Data.SqlClient.SqlInternalConnectionTds
MethodTable: 000007fedf56e8a8
EEClass:     000007fedf3d0cd0
Size:        296(0x128) bytes
File:        D:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fee1cf6408  40008b8       38         System.Int32  1 instance              869 _objectID
000007fee1cf4f30  40008bb       44       System.Boolean  1 instance                0 _allowSetConnectionString
000007fee1cf4f30  40008bc       45       System.Boolean  1 instance                1 _hidePassword
000007fedf567c68  40008bd       3c         System.Int32  1 instance                1 _state
000007fee1cf0850  40008be        8 System.WeakReference  0 instance 00000003c0042768 _owningObject
000007fedf56bfc8  40008bf       10 ....DbConnectionPool  0 instance 000000047fd5f468 _connectionPool
000007fedf568ca8  40008c0       18 ...ctionPoolCounters  0 instance 000000047fd31478 _performanceCounters
000007fedf56a6c8  40008c1       20 ...ferenceCollection  0 instance 00000003c0047278 _referenceCollection
000007fee1cf6408  40008c2       40         System.Int32  1 instance                0 _pooledCount
000007fee1cf4f30  40008c3       46       System.Boolean  1 instance                0 _connectionIsDoomed
000007fee1cf4f30  40008c4       47       System.Boolean  1 instance                0 _cannotBePooled
000007fee1cf4f30  40008c5       48       System.Boolean  1 instance                0 _isInStasis
000007fee1cf0980  40008c6       50      System.DateTime  1 instance 00000003c0042620 _createTime
000007fef53a39d8  40008c7       28 ...tions.Transaction  0 instance 0000000000000000 _enlistedTransaction
000007fef53a39d8  40008c8       30 ...tions.Transaction  0 instance 0000000000000000 _enlistedTransactionOriginal
000007fee1cf6408  40008b7      6e0         System.Int32  1   shared           static _objectTypeCount
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:969 <<
000007fedf56f5d8  40008b9      618 ...teChangeEventArgs  0   shared           static StateChangeClosed
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd36570 <<
000007fedf56f5d8  40008ba      620 ...teChangeEventArgs  0   shared           static StateChangeOpen
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd36588 <<
000007fedf56ae10  4001258       58 ...lConnectionString  0 instance 000000047fd5ed68 _connectionOptions
000007fee1cf4f30  4001259       88       System.Boolean  1 instance                0 _isEnlistedInTransaction
000007fee1cf77d0  400125a       60        System.Byte[]  0 instance 0000000000000000 _promotedDTCToken
000007fee1cf77d0  400125b       68        System.Byte[]  0 instance 0000000000000000 _whereAbouts
000007fee1cf3658  400125c       70        System.String  0 instance 00000003c00432b8 <CurrentDatabase>k__BackingField
000007fee1cf3658  400125d       78        System.String  0 instance 000000047fd5ee80 <CurrentDataSource>k__BackingField
000007fedfb22358  400125e       80 ...egatedTransaction  0 instance 0000000000000000 <DelegatedTransaction>k__BackingField
000007fedf56ccc0  400126e       90 ...GroupProviderInfo  0 instance 000000047fd5f0d8 _poolGroupProviderInfo
000007fedf569ad8  400126f       98 ...lClient.TdsParser  0 instance 00000003c0042850 _parser
000007fedf56dbb0  4001270       a0 ...lient.SqlLoginAck  0 instance 00000003c0043340 _loginAck
000007fedf5698b0  4001271       a8 ...ent.SqlCredential  0 instance 0000000000000000 _credential
000007fee1cf4f30  4001272       89       System.Boolean  1 instance                1 _fConnectionOpen
000007fee1cf4f30  4001273       8a       System.Boolean  1 instance                1 _fResetConnection
000007fee1cf3658  4001274       b0        System.String  0 instance 00000003c00432b8 _originalDatabase
000007fee1cf3658  4001275       b8        System.String  0 instance 0000000000000000 _currentFailoverPartner
000007fee1cf3658  4001276       c0        System.String  0 instance 00000003c0043310 _originalLanguage
000007fee1cf3658  4001277       c8        System.String  0 instance 00000003c0043310 _currentLanguage
000007fee1cf6408  4001278      100         System.Int32  1 instance             8000 _currentPacketSize
000007fee1cf6408  4001279      104         System.Int32  1 instance                0 _asyncCommandCount
000007fee1cf3658  400127a       d0        System.String  0 instance 00000002ffcd1420 _instanceName
000007fedf56ef90  400127b       d8 ...ctionPoolIdentity  0 instance 000000047fd3a240 _identity
000007fedf56f4d8  400127c       e0 ...Tds+SyncAsyncLock  0 instance 00000003c00426f8 _parserLock
000007fee1cf6408  400127d      108         System.Int32  1 instance               -1 _threadIdOwningParserLock
000007fedf56a818  400127e       e8 ...eoutErrorInternal  0 instance 00000003c0042780 timeoutErrorInternal
000007fee16612d0  400127f       f0 ...rence, mscorlib]]  0 instance 0000000000000000 _preparedCommands
000007fee1cf0da8  4001280      110          System.Guid  1 instance 00000003c00426e0 _clientConnectionId
000007fedf5860d0  4001281       f8 ...lient.RoutingInfo  0 instance 0000000000000000 _routingInfo

0:079> !do 00000003c0042850
Name:        System.Data.SqlClient.TdsParser
MethodTable: 000007fedf569ad8
EEClass:     000007fedf3cf318
Size:        160(0xa0) bytes
File:        D:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fee1cf6408  40014d6       70         System.Int32  1 instance              864 _objectID
000007fedf56dcc0  40014d8        8 ...ParserStateObject  0 instance 00000003c00428f0 _physicalStateObj
000007fedf56dcc0  40014d9       10 ...ParserStateObject  0 instance 0000000000000000 _pMarsPhysicalConObj
000007fedf589758  40014da       74         System.Int32  1 instance                2 _state
000007fee1cf3658  40014db       18        System.String  0 instance 000000047fd5ee80 _server
000007fee1cf4f30  40014dc       88       System.Boolean  1 instance                0 _fResetConnection
000007fee1cf4f30  40014dd       89       System.Boolean  1 instance                0 _fPreserveTransaction
000007fedf569618  40014de       20 ...ient.SqlCollation  0 instance 00000003c00432f8 _defaultCollation
000007fee1cf6408  40014df       78         System.Int32  1 instance             1252 _defaultCodePage
000007fee1cf6408  40014e0       7c         System.Int32  1 instance             1033 _defaultLCID
000007fee1cf7528  40014e1       28 System.Text.Encoding  0 instance 000000047fd3e630 _defaultEncoding
000007fedf57b7e8  40014e3       80         System.Int32  1 instance                0 _encryptionOption
000007fedf58c368  40014e4       30 ...ternalTransaction  0 instance 0000000000000000 _currentTransaction
000007fedf58c368  40014e5       38 ...ternalTransaction  0 instance 0000000000000000 _pendingTransaction
000007fee1ce2060  40014e6       68         System.Int64  1 instance 0 _retainedTransactionId
000007fee1cf6408  40014e7       84         System.Int32  1 instance                0 _nonTransactedOpenResultCount
000007fedf56e8a8  40014e8       40 ...rnalConnectionTds  0 instance 00000003c00425d0 _connHandler
000007fee1cf4f30  40014e9       8a       System.Boolean  1 instance                0 _fMARS
000007fee1cf4f30  40014ea       8b       System.Boolean  1 instance                0 _loginWithFailover
000007fee1cf0918  40014eb       48 ...ng.AutoResetEvent  0 instance 0000000000000000 _resetConnectionEvent
000007fedf584368  40014ec       50 ...ParserSessionPool  0 instance 0000000000000000 _sessionPool
000007fee1cf4f30  40014ed       8c       System.Boolean  1 instance                1 _isShiloh
000007fee1cf4f30  40014ee       8d       System.Boolean  1 instance                1 _isShilohSP1
000007fee1cf4f30  40014ef       8e       System.Boolean  1 instance                1 _isYukon
000007fee1cf4f30  40014f0       8f       System.Boolean  1 instance                1 _isKatmai
000007fee1cf4f30  40014f1       90       System.Boolean  1 instance                0 _isDenali
000007fee1cf77d0  40014f2       58        System.Byte[]  0 instance 00000003c0042be8 _sniSpnBuffer
000007fedf5863e0  40014f3       60 ...ent.SqlStatistics  0 instance 0000000000000000 _statistics
000007fee1cf4f30  40014f4       91       System.Boolean  1 instance                0 _statisticsIsInTransaction
000007fee1cf4f30  40014fb       92       System.Boolean  1 instance                0 _asyncWrite
000007fee1cf6408  40014d5      a40         System.Int32  1   shared           static _objectTypeCount
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:964 <<
000007fee1ce8c48  40014d7     1100 ...eading.Tasks.Task  0   shared           static completedTask
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:0000000000000000 <<
000007fedf57b7e8  40014e2      a44         System.Int32  1   shared           static _sniSupportedEncryptionOption
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:0 <<
000007fee1cf77d0  40014f5     1108        System.Byte[]  0   shared           static s_nicAddress
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd3de68 <<
000007fee1cf4f30  40014f6      a4c       System.Boolean  1   shared           static s_fSSPILoaded
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:1 <<
000007fee1cdffa0  40014f7      a48        System.UInt32  1   shared           static s_maxSSPILength
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:12256 <<
000007fee1cf77d0  40014f8     1110        System.Byte[]  0   shared           static s_longDataHeader
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd3cf80 <<
000007fee1cf3c38  40014f9     1118        System.Object  0   shared           static s_tdsParserLock
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd3cfb8 <<
000007fee1cf77d0  40014fa     1120        System.Byte[]  0   shared           static s_xmlMetadataSubstituteSequence
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd3cfd0 <<
000007fedf3905c8  40014fc     1128 ...rd, System.Data]]  0   shared           static __tvpEmptyValue
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:000000047fd3d038 <<

0:079> !do 00000003c00428f0
Name:        System.Data.SqlClient.TdsParserStateObject
MethodTable: 000007fedf56dcc0
EEClass:     000007fedf3d09b8
Size:        424(0x1a8) bytes
File:        D:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fee1cf6408  40015a4      128         System.Int32  1 instance              864 _objectID
000007fedf569ad8  40015a5        8 ...lClient.TdsParser  0 instance 00000003c0042850 _parser
000007fedf569528  40015a6       10 ...lClient.SNIHandle  0 instance 00000003c0043150 _sessionHandle
000007fee1cf0850  40015a7       18 System.WeakReference  0 instance 00000003c0042a98 _owner
000007fee1cf6408  40015a8      12c         System.Int32  1 instance                0 _activateCount
000007fee1cf6408  40015a9      130         System.Int32  1 instance                8 _inputHeaderLen
000007fee1cf6408  40015aa      134         System.Int32  1 instance                8 _outputHeaderLen
000007fee1cf77d0  40015ab       20        System.Byte[]  0 instance 00000003c0045300 _outBuff
000007fee1cf6408  40015ac      138         System.Int32  1 instance                8 _outBytesUsed
000007fee1cf77d0  40015ad       28        System.Byte[]  0 instance 00000003c00433a8 _inBuff
000007fee1cf6408  40015ae      13c         System.Int32  1 instance              676 _inBytesUsed
000007fee1cf6408  40015af      140         System.Int32  1 instance              676 _inBytesRead
000007fee1cf6408  40015b0      144         System.Int32  1 instance                0 _inBytesPacket
000007fee1cf5ab8  40015b1      170          System.Byte  1 instance                3 _outputMessageType
000007fee1cf5ab8  40015b2      171          System.Byte  1 instance                1 _messageStatus
000007fee1cf5ab8  40015b3      172          System.Byte  1 instance                1 _outputPacketNumber
000007fee1cf4f30  40015b4      173       System.Boolean  1 instance                0 _pendingData
000007fee1cf4f30  40015b5      174       System.Boolean  1 instance                0 _fResetEventOwned
000007fee1cf4f30  40015b6      175       System.Boolean  1 instance                0 _fResetConnectionSent
000007fee1cf4f30  40015b7      176       System.Boolean  1 instance                0 _errorTokenReceived
000007fee1cf4f30  40015b8      177       System.Boolean  1 instance                0 _bulkCopyOpperationInProgress
000007fee1cf4f30  40015b9      178       System.Boolean  1 instance                0 _bulkCopyWriteTimeout
000007fedf56ece8  40015ba       30 ...lClient.SNIPacket  0 instance 00000003c0047258 _sniPacket
000007fedf56ece8  40015bb       38 ...lClient.SNIPacket  0 instance 00000001ffdc1038 _sniAsyncAttnPacket
000007fedf56e0c8  40015bc       40 ....WritePacketCache  0 instance 00000003c0042ab0 _writePacketCache
000007fedf387940  40015bd       48 ...et, System.Data]]  0 instance 00000003c0042af8 _pendingWritePackets
000007fee1cf3c38  40015be       50        System.Object  0 instance 00000003c0042b48 _writePacketLockObject
000007fee1cf21b0  40015bf      188 ...Services.GCHandle  1 instance 00000003c0042a78 _gcHandle
000007fee1cf6408  40015c0      148         System.Int32  1 instance                1 _pendingCallbacks
000007fee1ce2060  40015c1      100         System.Int64  1 instance 3600000 _timeoutMilliseconds
000007fee1ce2060  40015c2      108         System.Int64  1 instance 0 _timeoutTime
000007fee1cf4f30  40015c3      179       System.Boolean  1 instance                0 _attentionSent
000007fee1cf4f30  40015c4      17a       System.Boolean  1 instance                0 _attentionReceived
000007fee1cf4f30  40015c5      17b       System.Boolean  1 instance                0 _attentionSending
000007fee1cf4f30  40015c6      17c       System.Boolean  1 instance                0 _internalTimeout
000007fee1ce2060  40015c7      110         System.Int64  1 instance 634759841647582060 _lastSuccessfulIOTime
000007fee1cbeb28  40015c8       58      System.Object[]  0 instance 00000003c0042b60 _securePasswords
000007fee1cf63a8  40015c9       60       System.Int32[]  0 instance 00000003c0042b90 _securePasswordOffsetsInBuffer
000007fee1cf4f30  40015ca      17d       System.Boolean  1 instance                0 _cancelled
000007fee1cf6408  40015cb      14c         System.Int32  1 instance               -1 _allowObjectID
000007fee1cf4f30  40015cc      17e       System.Boolean  1 instance                0 _hasOpenResult
000007fedf58c368  40015cd       68 ...ternalTransaction  0 instance 0000000000000000 _executedUnderTransaction
000007fee1cd3ea8  40015ce      118        System.UInt64  1 instance 0 _longlen
000007fee1cd3ea8  40015cf      120        System.UInt64  1 instance 0 _longlenleft
000007fee1cf63a8  40015d0       70       System.Int32[]  0 instance 0000000000000000 _decimalBits
000007fee1cf77d0  40015d1       78        System.Byte[]  0 instance 00000004bff10550 _bTmp
000007fee1cf6408  40015d2      150         System.Int32  1 instance                0 _bTmpRead
000007fee1cd5ab0  40015d3       80  System.Text.Decoder  0 instance 0000000000000000 _plpdecoder
000007fee1cf4f30  40015d4      17f       System.Boolean  1 instance                0 _accumulateInfoEvents
000007fedf39a538  40015d5       88 ...or, System.Data]]  0 instance 0000000000000000 _pendingInfoEvents
000007fee1cf77d0  40015d6       90        System.Byte[]  0 instance 00000003c0042bb0 _partialHeaderBuffer
000007fee1cf6408  40015d7      154         System.Int32  1 instance                0 _partialHeaderBytesRead
000007fedf56c2d8  40015d8       98 ...t._SqlMetaDataSet  0 instance 00000004c0284568 _cleanupMetaData
000007fedf5828e0  40015d9       a0 ...DataSetCollection  0 instance 0000000000000000 _cleanupAltMetaDataSetArray
000007fee1cf6408  40015da      158         System.Int32  1 instance                0 _tracePasswordOffset
000007fee1cf6408  40015db      15c         System.Int32  1 instance                0 _tracePasswordLength
000007fee1cf6408  40015dc      160         System.Int32  1 instance                0 _traceChangePasswordOffset
000007fee1cf6408  40015dd      164         System.Int32  1 instance                0 _traceChangePasswordLength
000007fee1cf4f30  40015de      180       System.Boolean  1 instance                0 _receivedColMetaData
000007fedf571848  40015df      168         System.Int32  1 instance               11 _sniContext
000007fee1cf4f30  40015e0      181       System.Boolean  1 instance                0 _bcpLock
000007fedf56f070  40015e1      190 ...Object+NullBitmap  1 instance 00000003c0042a80 _nullBitmapInfo
000007fee1cfc0e0  40015e2       a8 ...bject, mscorlib]]  0 instance 0000000000000000 _networkPacketTaskSource
000007fee1cf1fe8  40015e3       b0 ...m.Threading.Timer  0 instance 0000000000000000 _networkPacketTimeout
000007fee1cf4f30  40015e4      182       System.Boolean  1 instance                1 _syncOverAsync
000007fee1cf4f30  40015e5      183       System.Boolean  1 instance                0 _snapshotReplay
000007fedf5729f0  40015e6       b8 ...ect+StateSnapshot  0 instance 0000000000000000 _snapshot
000007fee1cf1e50  40015e7       c0 ....ExecutionContext  0 instance 0000000000000000 _executionContext
000007fee1cf4f30  40015e8      184       System.Boolean  1 instance                0 _asyncReadWithoutSnapshot
000007fedf56c1b8  40015e9       c8 ...qlErrorCollection  0 instance 0000000000000000 _errors
000007fedf56c1b8  40015ea       d0 ...qlErrorCollection  0 instance 0000000000000000 _warnings
000007fee1cf3c38  40015eb       d8        System.Object  0 instance 00000003c0042bd0 _errorAndWarningsLock
000007fee1cf4f30  40015ec      185       System.Boolean  1 instance                0 _hasErrorOrWarning
000007fedf56c1b8  40015ed       e0 ...qlErrorCollection  0 instance 0000000000000000 _preAttentionErrors
000007fedf56c1b8  40015ee       e8 ...qlErrorCollection  0 instance 0000000000000000 _preAttentionWarnings
000007fee1cfc0e0  40015ef       f0 ...bject, mscorlib]]  0 instance 0000000000000000 _writeCompletionSource
000007fee1cf6408  40015f0      16c         System.Int32  1 instance                0 _asyncWriteCount
000007fee1cf3888  40015f1       f8     System.Exception  0 instance 0000000000000000 _delayedWriteAsyncCallbackException
000007fee1cf4f30  40015f2      186       System.Boolean  1 instance                0 _reading
000007fee1cf6408  40015a3      a54         System.Int32  1   shared           static _objectTypeCount
                                 >> Domain:Value  000000000179a3a0:NotInit  000000000ffb56e0:NotInit  <<

_outBuff is our output buffer, i.e. what we sent to the server, and it has our command text... Now we know who's leaking!

0:079> !do 00000003c0045300
Name:        System.Byte[]
MethodTable: 000007fee1cf77d0
EEClass:     000007fee16fa4b8
Size:        8024(0x1f58) bytes
Array:       Rank 1, Number of elements 8000, Type Byte
Content:     <...6............................G.e.t.C.o.n.s.t.a.n.t.S.e.t.s....@.p.a.r.t.i.t.i.o.n.I.d..&.......@.u.s.e.r.S.I.D...X.....4X.S>.-
Fields:
None