Remi's World https://docs.microsoft.com/archive/blogs/remis_world/feed.xml Wed, 17 Apr 2019 03:50:39 GMT en-US hourly 1 The case of the stunted recovery https://docs.microsoft.com/archive/blogs/remis_world/the-case-of-the-stunted-recovery Tue, 11 Sep 2018 20:30:41 GMT https://blogs.msdn.microsoft.com/remis_world/?p=145 here that we were having a bad couple of days in the Azure South Central region after a bad storm. We had finally recovered (or so we thought) when we realized that one of the two big TFS scale units was showing signs of distress: Availability was down sharply: Mostly because of slow commands (as opposed to just failures): CPU was high but not high enough to explain such performance problems (note that I am writing this after the fact so you see the recovery around 19:00) - you'll also notice the fact that we recycled AT_IN_2 and got a memory dump on AT_IN_1): And Queued Requests were high (not a good sign, consistent with performance issues): So the next step is to go look at a PerfView log (which our automated agent grabs for us in those cases). In the case of High Requests Queued, I also start with the ASP.net statistics node: ASP.Net Statistics • Total Requests: 8,926.00 • Trace Duration (Sec): 169.0 • Average Request/Sec: 52.80 • Number of CPUs: 20 • Maximum Number of requests recieved but not replied to: 1165 • Maximum Number of requests queued waiting for processing: 749 • Maximum Number of requests concurrently being worked on: 955 Those numbers are pathetic (given the D15v2 machine we're running), we can usually push 800+ requests/sec with this kind of hardware – in this case, we’re barely doing 52… I also notice a weird pattern: the CPU is all over the place, and the queueing seems to be happening when the CPU is low: One known way to get this kind of pattern is excessive GCs which may cause beat frequency issues, so naturally, I look at the GC Stats section, and I see this: • CommandLine: d:\windows\system32\inetsrv\w3wp.exe • Total CPU Time: 420,750 msec • Total GC CPU Time: 24,694 msec • Total GC Pause: 13,354.6 msec • % Time paused for Garbage Collection: 48.9% • % CPU Time spent Garbage Collecting: 5.9% • Max GC Heap Size: 97,452.790 MB • Peak Process Working Set: 105,898.787 MB • Peak Virtual Memory Usage: 2,339,444.720 MB So what I see is: • We’re using a lot of CPU overall (420s/30s = 14 cores out of 20) but it’s not atrocious • We’re not spending a lot of CPU in the GC (5.9% is good) • We’re spending close to half of the time paused (48.9%) and clearly not doing much since the CPU in GC is 5.9%! So the next thing to look at is why we’re spending so much time suspended so I look for a long pause and focus on that: GC Index Pause Start Trigger Reason Gen Suspend Msec Pause Msec 32952 3,773.61 AllocSmall 1N 1,304.97 1,582.67 • GC Index is just the # of the current GC, it’s the 32,952th GC for this process • Pause Start is when the CLR starts the suspension • Trigger Reason is why the GC was triggered (by a small allocation) • Gen is the Generation • Suspend Msec is how long in milliseconds the suspension took (way too long!) • Pause Msec is the total length of the pause What I am looking for now is why it’s taking so long to suspend the Execution Engine (EE) – and the way to do that is to see what is actually running while we’re supposed to be pausing. Here are the events I am interested in: So I go to the CPU view, pick my process (w3wp) and restrict the timeline to between the events above: I observe this weird pattern of contention on a native CLR lock while we’re gathering a stack trace while printing Exception.ToString() – so I call my colleagues and they tell me they don’t see any traces showing this exception and that I must be mistaken… I have to agree with them that we indeed don’t see any of those traces so we’re kind of stuck – still, I share my findings with the CLR perf folks and figure they probably know better. In the meantime, I grab a dump of the process (111 GB) and look for large objects (the process is really bloated and since I am investigating GC issues, it seems like a good idea) – I notice a lot of large strings in the LOH: 0:272> !DumpHeap -min 81000 -MT 00007fff4e176830 Address MT Size 000002f8791bb080 00007fff4e176830 4362836 000002f8798540f8 00007fff4e176830 4431142 000002f87f326c58 00007fff4e176830 4431358 0000030101531c18 00007fff4e176830 5026698 000003010273ecd8 00007fff4e176830 4430314 0000030103f9f838 00007fff4e176830 4434006 00000301083defc0 00007fff4e176830 5026698 000003010cd7ab80 00007fff4e176830 5027526 000003010de528c0 00007fff4e176830 4360256 000003010e4d93b0 00007fff4e176830 4363424 000003010f93e598 00007fff4e176830 5027526 0000030166df1020 00007fff4e176830 10058324 … So I dump one of those strings, and then another, and another and they all look like this: 0:000> !do 0000030166df1020 Name: System.String MethodTable: 00007fff4e176830 EEClass: 00007fff4da56cb8 Size: 10,058,324(0x997a54) bytes String: Fields: MT Field Offset Type VT Attr Value Name 00007fff4e179180 4000273 8 System.Int32 1 instance 5,029,149 m_stringLength 00007fff4e1779e8 4000274 c System.Char 1 instance 53 m_firstChar 0:000> du 0000030166df1020+c 00000301`66df102c "System.AggregateException: One o" 00000301`66df106c "r more errors occurred. ---> Mic" 00000301`66df10ac "rosoft.ServiceBus.Messaging.Mess" 00000301`66df10ec "agingException: The service was " 00000301`66df112c "unable to process the request; p" 00000301`66df116c "lease retry the operation. For m" 00000301`66df11ac "ore information on exception typ" 00000301`66df11ec "es and proper exception handling" 00000301`66df122c ", please refer to http://go.micr" 00000301`66df126c "osoft.com/fwlink/?LinkId=761101 " 00000301`66df12ac "TrackingId:b8337bc1bb0d4f26909e4" 00000301`66df12ec "a73054ae986_G16, SystemTracker:g" So this is indeed the string generated by Exception.ToString() and it’s almost 10MB because the call stack is so deep and the method names are so long – that explains why we’re spending so much time in GetStackStrace – it all makes sense now. The next morning, I get a reply from the CLR folks who do confirm that getting a callstack can be slow and is done in cooperative mode (i.e. it’s not interruptible) and that it’s also not particularly efficient because the hash table that maps method addresses to method names for is not sized correctly – since I have dump, I take a look and sure enough, they’re right (those guys are really good): 0:000> x clr!g_pDebugInterface 00007fff`516851d0 clr!g_pDebugInterface = 0x000002f3`6f08f8e0 0:000> dt -t clr!Debugger 0x000002f3`6f08f8e0 +0x000 __VFN_table : 0x00007fff`51478af0 +0x018 m_processId : 0x4ae4 +0x01c m_trappingRuntimeThreads : 0n0 +0x020 m_stopped : 0n0 +0x024 m_unrecoverableError : 0n0 +0x028 m_ignoreThreadDetach : 0n0 +0x030 m_pMethodInfos : 0x000002f3`6f981540 DebuggerMethodInfoTable 0:000> dt -t clr!DebuggerMethodInfoTable 0x000002f3`6f981540 +0x000 __VFN_table : 0x00007fff`5147fb08 +0x008 m_pcEntries : 0x000002f3`3731f040 +0x010 m_iEntrySize : 0x38 +0x014 m_iBuckets : 0x65 +0x018 m_piBuckets : 0x000002f3`6f104960 -> 0x171b1 +0x020 m_iFree : 0x171bd +0x024 m_iEntries : 0x258b4 0:000> ? 0x258b4 Evaluate expression: 153,780 = 00000000`000258b4 Even better, the CLR folks open an issue on GitHub to track the bug! But you will ask, why didn't we see those exceptions in our telemetry? Well that's because those strings are way over the ETW limit (32KB) which means they just get dropped on the floor (yeah, that's bad and I have fixed it since). ]]> The case of the not so ConcurrentDictionary https://docs.microsoft.com/archive/blogs/remis_world/the-case-of-the-not-so-concurrentdictionary Fri, 16 Mar 2018 19:16:35 GMT https://blogs.msdn.microsoft.com/remis_world/?p=85 So I pinged my colleague who owns this service and he noticed it was actual very predictable: Like clockwork, once a minute - he went further and got a PerfView which showed high contention on a newly added ConcurrentDictionary: He then asked me to take a look since that ConcurrentDictionary was added on my suggestion to work around another issue (which I will blog about one day). Having had that problem before, I figured we either had a hot spot or a hashing function problem - so I got a dump of the process to see which (I could have saved some time and looked at the source...but as they say, there's nothing like a good dump). 
0:000> !do 0000015d19e676f0
Name: System.Collections.Concurrent.ConcurrentDictionary`2
MethodTable: 00007ff8e18f9728
EEClass: 00007ff8e18c5de0
Size: 64(0x40) bytes
File: D:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
 MT              Field   Offset Type                 VT Attr     Value            Name
00007ff8e187d930 4001830 8      ....Byte, mscorlib]] 0  instance 0000015e3a819078 m_tables
00007ff93b3c4300 4001831 10     ...Canon, mscorlib]] 0  instance 0000000000000000 m_comparer
00007ff93b3b1f28 4001832 30     System.Boolean       1  instance 1                m_growLockArray
00007ff93b3a9288 4001833 20     System.Int32         1  instance 0                m_keyRehashCount
00007ff93b3a9288 4001834 24     System.Int32         1  instance 256              m_budget
0000000000000000 4001835 18     SZARRAY              0  instance 0000000000000000 m_serializationArray
00007ff93b3a9288 4001836 28     System.Int32         1  instance 0                m_serializationConcurrencyLevel
00007ff93b3a9288 4001837 2c     System.Int32         1  instance 0                m_serializationCapacity
00007ff93b3b1f28 400183b 10     System.Boolean       1  static   <no information>
0:000> !do 0000015e3a819078
Name: System.Collections.Concurrent.ConcurrentDictionary`2+Tables
MethodTable: 00007ff8e18fafd0
EEClass: 00007ff8e18c6ab0
Size: 48(0x30) bytes
File: D:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
 MT              Field   Offset Type            VT Attr     Value            Name
0000000000000000 400341d 8      SZARRAY         0  instance 0000015e3a816ca8 m_buckets
00007ff93b3a6fc0 400341e 10     System.Object[] 0  instance 0000015e3a816290 m_locks
00007ff93b3a9220 400341f 18     System.Int32[]  0  instance 0000015e3a817e28 m_countPerLock
00007ff93b3c4300 4003420 20...Canon, mscorlib]] 0  instance 0000015d19e677a0 m_comparer
0:000> !DumpArray 0000015e3a816ca8
Name: System.Collections.Concurrent.ConcurrentDictionary`2+Node
MethodTable: 00007ff8e18fae50
EEClass: 00007ff93ad6aa00
Size: 4480(0x1180) bytes
Array: Rank 1, Number of elements 557, Type CLASS
Element Methodtable: 00007ff8e18fad88
[0] null
[1] null
<...>
[428] null
[429] 0000015d46de8280
[430] null
<...>
[556] null
0:000> !do 0000015d46de8280
Name: System.Collections.Concurrent.ConcurrentDictionary`2+Node
MethodTable: 00007ff8e18fad88
EEClass: 00007ff8e18c6990
Size: 40(0x28) bytes
File: D:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
 MT              Field   Offset Type             VT Attr     Value            Name
00007ff93b3abf10 4003421 8      System.__Canon   0  instance 0000015d46de8240 m_key
00007ff93b3a8940 4003422 1c     System.Byte      1  instance 0                m_value
00007ff8e187d800 4003423 10 ....Byte, mscorlib]] 0  instance 0000015e47c672e8 m_next
00007ff93b3a9288 4003424 18     System.Int32     1  instance 37103870         m_hashcode
0:000> !do 0000015e47c672e8
Name: System.Collections.Concurrent.ConcurrentDictionary`2+Node
MethodTable: 00007ff8e18fad88
EEClass: 00007ff8e18c6990
Size: 40(0x28) bytes
File: D:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
 MT              Field   Offset  Type            VT Attr    Value            Name
00007ff93b3abf10 4003421 8       System.__Canon  0 instance 0000015e47c672a8 m_key
00007ff93b3a8940 4003422 1c      System.Byte     1 instance 0                m_value
00007ff8e187d800 4003423 10 ....Byte, mscorlib]] 0 instance 0000015e4781f3f8 m_next
00007ff93b3a9288 4003424 18      System.Int32    1 instance 37103870         m_hashcode
Sure enough, there's only one bucket occupied, and all of the items in that bucket have the same hash code, so our ConcurrentDictionary is really a giant linked list, with a giant lock on top... Our ConcurrentDictionary's keys are System.EventHandler which is really a delegate - which default HashCode implementation is...the hash code of the underlying type which means all of our delegates have the same hashcode, hence the same bucket....DOH!]]> The mysterious case of the 1 minute pauses https://docs.microsoft.com/archive/blogs/remis_world/the-mysterious-case-of-the-1-minute-pauses Thu, 15 Feb 2018 19:38:54 GMT https://blogs.msdn.microsoft.com/remis_world/?p=35 Our DRIs had been tracking this issue but couldn't get a usable PerfView log - so I got called in to help out - and I saw the same behavior - i.e. the ETL traces that we auto-collect were not helpful - mostly because they either didn't show high CPU or because the symbols were messed up  - the dreaded ?!? in PerfView (that happens sometimes with dynamically generated code, like XML Serializers or compiled regex) but this was worse than usual. So I hopped onto the machines and captured a more detail log of requests/sec and queued requests:     And lo and behold, we have a one minute gap where nothing happens, followed by a massive increase in queued requests, then everything goes back to normal. I was pretty sure I was onto something so I figured I had to get a trace during that exact minute. So I fired up PerfView and waited - once I saw requests/sec plummet to zero, I clicked "Collect" and this is what I saw in the CPU Stacks view - just one thread doing work (and lot of it - basically burning an entire core):   That was still puzzling - we have one thread getting called by an ETW callback and doing rundown (rundown is basically the process that tells the CLR to dump type information in the ETW stream so that profilers can make sense of things like dynamic types/assemblies) and absolutely nothing else going on... I decided a dump might be helpful, so I waited again and collected a reflected process dump with ProcDump. I opened it up and saw tons of these types of threads: 
ntdll!ZwWaitForSingleObject+0x14
KERNELBASE!WaitForSingleObjectEx+0x8f
clr!CLREventWaitHelper2+0x3c
clr!CLREventWaitHelper+0x1f
clr!CLREventBase::WaitEx+0x7c
clr!CLREventBase::Wait+0x15
clr!SVR::GCHeap::WaitUntilGCComplete+0x2f
clr!Thread::RareDisablePreemptiveGC+0x180
clr!Thread::DisablePreemptiveGC+0x17bafc
clr!GCHolderBase::EnterInternalCoop+0x17bb0c
clr!GCCoop::GCCoop+0x3d
clr!Thread::OnThreadTerminate+0x7e
clr!DestroyThread+0xd2
clr!ThreadpoolMgr::WorkerThreadStart+0x1ba4d0
clr!Thread::intermediateThreadProc+0x86
kernel32!BaseThreadInitThunk+0x14
ntdll!RtlUserThreadStart+0x21
Except for one thread whose stack didn't fully resolve but did match the thread id of my PerfView above (i.e the rundown thread): 
clr!ExecutionManager::GetRangeSection+0x59
clr!ExecutionManager::FindCodeRangeWithLock+0xa7
0x00007ff9`58fe28b0
0x00000253`eecb0848
clr!ClrFreeInProcessHeap+0x17
clr!operator delete[]+0x20
clr!SBuffer::DeleteBuffer+0x24
clr!SBuffer::{dtor}+0xffffffff`ffeefbbb
clr!SString::~SString+0x3e
So here's what we know so far:
  • All threads are stuck waiting for a GC
  • One thread is burning an entire core doing rundown, enumerating all of the types in the process.
The questions that remain are:
  1. Why is the rundown taking so long?
  2. Why is it holding up the GC threads?
  3. Why is there a rundown at all?
Since I have a process dump, I am going to look at the types in there: 
0:092> !DumpDomain
--------------------------------------
normal output followed by thousands of these guys:
Assembly: 0000025879e43270 (Dynamic) []
ClassLoader: 000002587c924da0
SecurityDescriptor: 0000025879cd5220
 Module Name
00007ff939994ea8 Dynamic Module

Assembly: 0000025879e44230 (Dynamic) []
ClassLoader: 000002587c924e50
SecurityDescriptor: 0000025879cd5d60
 Module Name
00007ff939995b08 Dynamic Module
So that confirms the dynamic types going crazy theory - now we just have to figure out what they are and what is causing the problem - so I pick a couple of random modules and dump them (I also notice there's a cool -MT option to !DumpModule) 
0:092> !DumpModule -MT 00007ff93bc7f7a0
Name: Unknown Module
Attributes: Reflection Supports
<snip>
Types defined in this module
MT TypeDef Name
------------------------------------------------------------------------------
00007ff93bc7fed0 0x02000002 <Unloaded Type>
00007ff93bcc0068 0x02000003 <Unloaded Type>
00007ff93bcc0150 0x02000004 <Unloaded Type>
00007ff93bcc0260 0x02000005 <Unloaded Type>
00007ff93bcc03b0 0x02000006 <Unloaded Type>
Then I dump the types themselves and lo and behold, it's a bunch of XML Serializers: 
0:092> !DumpMT -MD 00007ff93bcc03b0
EEClass: 00007ff93bcbdac8
Module: 00007ff93bc7f7a0
Name: <Unloaded Type>
mdToken: 0000000002000006
File: Unknown Module
BaseSize: 0x28
ComponentSize: 0x0
Slots in VTable: 12
Number of IFaces in IFaceMap: 0
--------------------------------------
MethodDesc Table
 Entry MethodDesc JIT Name
00007ff987692c20 00007ff9871f7de0 PreJIT System.Object.ToString()
00007ff987696f40 00007ff9871f7de8 PreJIT System.Object.Equals(System.Object)
00007ff9876960a0 00007ff9871f7e10 PreJIT System.Object.GetHashCode()
00007ff9876bb7b0 00007ff9871f7e28 PreJIT System.Object.Finalize()
00007ff93b0f1e58 00007ff93bcc0358 NONE Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializerContract.get_Reader()
00007ff93b0f2280 00007ff93bcc0360 JIT Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializerContract.get_Writer()
00007ff93b0f1e68 00007ff93bcc0368 NONE Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializerContract.get_ReadMethods()
00007ff93b0f22c0 00007ff93bcc0370 JIT Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializerContract.get_WriteMethods()
00007ff93b0f1e78 00007ff93bcc0378 NONE Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializerContract.get_TypedSerializers()
00007ff93b0f1e80 00007ff93bcc0380 NONE Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializerContract.CanSerialize(System.Type)
00007ff93b0f1e88 00007ff93bcc0388 NONE Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializerContract.GetSerializer(System.Type)
00007ff93b0f2260 00007ff93bcc0390 JIT Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializerContract..ctor()
But wait, those serializers are supposed to be cached right? Well, yes and no, in this case, we're using the constructor  that does not use the built-in cache so a new serializer is created on every call to Serialize which means that we're generating a new assembly every couple of seconds... But why does the CLR rundown block the GC? Well, that was a bug and that was fixed last fall! We just hadn't picked up the new build yet. But wait, why are we even doing rundown? That's because we're running a super sweet background profiler which wakes up every so often and collects some data that gets aggregated later.    ]]> Troubleshooting ReaderWriterLockSlim issues https://docs.microsoft.com/archive/blogs/remis_world/troubleshooting-readerwriterlockslim-issues Tue, 14 Mar 2017 19:49:58 GMT https://blogs.msdn.microsoft.com/remis_world/?p=25 
0:1199> !ClrStack
OS Thread Id: 0x294c (1199)
Call Site
[HelperMethodFrame_1OBJ: 0000001b1667de98] System.Threading.WaitHandle.WaitOneNative(System.Runtime.InteropServices.SafeHandle, UInt32, Boolean, Boolean)
System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean)
System.Threading.WaitHandle.WaitOne(Int32, Boolean)
System.Threading.ReaderWriterLockSlim.WaitOnEvent(System.Threading.EventWaitHandle, UInt32 ByRef, TimeoutTracker)
System.Threading.ReaderWriterLockSlim.TryEnterReadLockCore(TimeoutTracker)
System.Threading.ReaderWriterLockSlim.TryEnterReadLock(TimeoutTracker)
[...]
(let's also assume most of the other threads are in this state) - let's take a closer look: 
0:1199> !dso
OS Thread Id: 0x294c (1199)
RSP/REG Object Name
0000001B1667DDE0 000001b2701f7810 Microsoft.Win32.SafeHandles.SafeWaitHandle
0000001B1667DEE0 000001b2701f77e0 System.Threading.ManualResetEvent
0000001B1667DF88 000001b82c2f6bf0 System.Threading.ReaderWriterCount
0000001B1667E060 000001b220284198 System.Threading.ReaderWriterLockSlim
[...]
Let's check out this ReaderWriterLockSlim object: 
0:1199> !do 000001b220284198
Name: System.Threading.ReaderWriterLockSlim
MethodTable: 00007ffab8dac760
EEClass: 00007ffab8ad3da0
Size: 96(0x60) bytes
Fields:
MT               Field   Offset Type VT Attr Value Name
00007ffabc3d24b8 40001d2 50  System.Boolean 1 instance 1 fIsReentrant
00007ffabc403e98 40001d3 30  System.Int32   1 instance 0 myLock
00007ffabc3fc9a0 40001d7 34  System.UInt32  1 instance 1 numWriteWaiters
00007ffabc3fc9a0 40001d8 38  System.UInt32  1 instance 1340 numReadWaiters
00007ffabc3fc9a0 40001d9 3c  System.UInt32  1 instance 0 numWriteUpgradeWaiters
00007ffabc3fc9a0 40001da 40  System.UInt32  1 instance 0 numUpgradeWaiters
00007ffabc3d24b8 40001db 51  System.Boolean 1 instance 0 fNoWaiters
00007ffabc403e98 40001dc 44  System.Int32   1 instance -1 upgradeLockOwnerId
00007ffabc403e98 40001dd 48  System.Int32   1 instance -1 writeLockOwnerId
00007ffabc4086c8 40001de 8   ...g.EventWaitHandle 0 instance 000001b183d85990 writeEvent
00007ffabc4086c8 40001df 10  ...g.EventWaitHandle 0 instance 000001b2701f77e0 readEvent
00007ffabc4086c8 40001e0 18  ...g.EventWaitHandle 0 instance 0000000000000000 upgradeEvent
00007ffabc4086c8 40001e1 20  ...g.EventWaitHandle 0 instance 0000000000000000 waitUpgradeEvent
00007ffabc3fb628 40001e3 28  System.Int64   1 instance 1951 lockID
00007ffabc3d24b8 40001e5 52  System.Boolean 1 instance 0 fUpgradeThreadHoldingRead
00007ffabc3fc9a0 40001e7 4c  System.UInt32  1 instance 1073741825 owners
00007ffabc3d24b8 40001ed 53  System.Boolean 1 instance 0 fDisposed
00007ffabc3fb628 40001e2 438 System.Int64   1 shared static s_nextLockID
>> Domain:Value 000001af5f46feb0:NotInit 000001b473428820:NotInit << 00007ffab8dab368 40001e4 0 ...ReaderWriterCount 0 shared TLstatic t_rwc >> Thread:Value <<
That's very interesting - if you're not familiar with the ReaderWriterLockSlim, you can see the source code here. Like all locks, ReaderWriterLockSlim is optimized for the happy case (i.e. the lock is free). In our case, there are 1340 waiting readers and 1 waiting writer - but what's the state of the lock? 
0:1199> ? 0n1073741825
Evaluate expression: 1073741825 = 00000000`40000001
Which means: Writer(s) waiting, one reader currently in the lock. Here's the decoder ring (from the source code linked above): 
        //Various R/W masks
        //Note:
        //The Uint is divided as follows:
        //
        //Writer-Owned  Waiting-Writers   Waiting Upgraders     Num-Readers
        //    31          30                 29                 28.......0
        //
        private const uint WRITER_HELD = 0x80000000;
        private const uint WAITING_WRITERS = 0x40000000;
        private const uint WAITING_UPGRADER = 0x20000000;
So, we have a reader that's not coming out - but which one? For that, we need to understand how the ReaderWriterLockSlim keeps track of the lock holders - again, it's optimized for the happy case: 
        private ReaderWriterCount GetThreadRWCount(bool dontAllocate)
        {
	    ReaderWriterCount rwc = t_rwc;
            ReaderWriterCount empty = null;
            while (rwc != null)
            {
                if (rwc.lockID == this.lockID)
		    return rwc;
           [...]
In summary, there's a thread local (no contention!) linked list of items (for all ReaderWriterLockSlim's), identified by the lockID of the lock you're holding. With that information, we can find the lone reader: Here's what a ReaderWriterCount looks like: 
0:1199> !do 000001b82c2f6bf0
Name:        System.Threading.ReaderWriterCount
MethodTable: 00007ffab8dab368
EEClass:     00007ffab8ad3980
Size:        48(0x30) bytes
File:        D:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Core\v4.0_4.0.0.0__b77a5c561934e089\System.Core.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007ffabc3fb628  40001cd       10         System.Int64  1 instance 1951 lockID
00007ffabc403e98  40001ce       18         System.Int32  1 instance                0 readercount
00007ffabc403e98  40001cf       1c         System.Int32  1 instance                0 writercount
00007ffabc403e98  40001d0       20         System.Int32  1 instance                0 upgradecount
00007ffab8dab368  40001d1        8 ...ReaderWriterCount  0 instance 000001b82c2f4b60 next
So, the idea is to dump all of the instances of that class that have the right lockID and the readerCount set to 1 - I don't know about you, but I don't like to grovel through thousands of objects, I'd rather spend the time writing a command I can reuse: 
0:1199> .foreach (rwc {!DumpHeap -MT 00007ffab8dab368 -short}){.if ((qwo(${rwc}+10) == 0n1951) and (dwo(${rwc}+0x18) > 0)) {!do rwc}}
Which returns...nothing... (the command above basically says: foreach instance of ReaderWriterCount, if the lockId is 1951 and the readerCount is greater than 0, dump the object) - that means the lock was orphaned - i.e. the thread that called EnterReaderLock exited without ever calling Leave :-( Note that sosex figures that out without resorting to arcane debugger commands: 
0:1199> !rwlock 000001b220284198
WriteLockOwnerThread:             None
UpgradableReadLockOwnerThread:    None
ReaderCount:                      1
ReaderThreadIds:
WaitingReaderCount:               1340
WaitingReaderThreadIds:           0x6b,0xdc,0x134,0x113,0x12d,0x193,0xcc,0x100,0x14[...]
WaitingWriterCount:               1340
WaitingWriterThreadIds:           0x7d
WaitingUpgradableReaderCount:     0
WaitingUpgradableReaderThreadIds: None
WaitingWriterUpgradeCount:        0
WaitingWriterUpgradeThreadIds:    None
*This lock has 1 orphaned reader lock.
Unfortunately, there isn't much more we can tell at this point - but if we had found one, we could just gcroot on it to see which thread owned it: 
0:1199> !gcroot 000001b82c2f6bf0
Thread 294c:
    0000001b1667e080 00007ffab92d932c System.Threading.ReaderWriterLockSlim.TryEnterReadLockCore(TimeoutTracker) [f:\dd\ndp\fx\src\Core\system\threading\ReaderWriterLockSlim\ReaderWriterLockSlim.cs @ 305]
        rbx:  (interior)
            ->  000001b82c2f4b38 System.Object[]
            ->  000001b82c2f6bf0 System.Threading.ReaderWriterCount

    0000001b1667e080 00007ffab92d932c System.Threading.ReaderWriterLockSlim.TryEnterReadLockCore(TimeoutTracker) [f:\dd\ndp\fx\src\Core\system\threading\ReaderWriterLockSlim\ReaderWriterLockSlim.cs @ 305]
        r14:
            ->  000001b82c2f6bf0 System.Threading.ReaderWriterCount
I hope this was useful - let me know in the comment sections if you want to see more of this kind of stuff.]]> How to diagnose SqlConnection Leaks https://docs.microsoft.com/archive/blogs/remis_world/how-to-diagnose-sqlconnection-leaks Mon, 25 Jun 2012 07:52:00 GMT https://blogs.msdn.microsoft.com/remis_world/2012/06/25/how-to-diagnose-sqlconnection-leaks/ This is a fairly common problem - you test your app, everything works great. Throw it into production and then it stops working and you see a lot of these:

System.InvalidOperationException: Timeout expired.  The timeout period elapsed prior to obtaining a connection from the pool.

You then check the number of connections in Perfmon and see that you have 100 connections used up...oops...something is not closing the connection....

You can follow the instructions on this blog to figure out which pool you're leaking from, but sometimes that's not quite enough, and you have to figure out where exactly in the code the connection is getting leaked. Here's how:

First, dump all the SqlConnection objects in the heap:

To do that more efficiently, I use !DumpHeap -MT <MethodTable> but I have to resolve the method table for SqlConnection like so:

0:108> !Name2EE System.Data.dll System.Data.SqlClient.SqlConnection
Module:      000007fee4db1000
Assembly:    System.Data.dll
Token:       0000000002000281
MethodTable: 000007fee4f96170
EEClass:     000007fee4dd8110
Name:        System.Data.SqlClient.SqlConnection

Then, use this little debugger command to dump all those connections (you might want to use a log file by doing ".logopen mylogfile.txt"):

0:108> .foreach (foo {!DumpHeap -short -MT 000007fee4f96170}) {.echo foo; !do foo}

00000001000ce1a8

Name:        System.Data.SqlClient.SqlConnection
MethodTable: 000007fee4f96170
EEClass:     000007fee4dd8110
Size:        128(0x80) bytes
File:        D:WindowsMicrosoft.NetassemblyGAC_64System.Datav4.0_4.0.0.0__b77a5c561934e089System.Data.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fee3ec3c38  4000198        8        System.Object  0 instance 0000000000000000 __identity
000007fee5a5f2e8  40002d2       10 ...ponentModel.ISite  0 instance 0000000000000000 site
000007fee5a6dbe0  40002d3       18 ....EventHandlerList  0 instance 0000000100dc3ea0 events
000007fee3ec3c38  40002d1      1b8        System.Object  0   shared           static EventDisposed
                                 >> Domain:Value  0000000001770530:NotInit  000000000fe9e740:00000003ffe0a4b0 <<
000007fee4fb1510  40002c0       20 ...hangeEventHandler  0 instance 0000000000000000 _stateChangeEventHandler
000007fee4fae4c0  400107c       28 ...t.SqlDebugContext  0 instance 0000000000000000 _sdc
000007fee3ec4f30  400107d       70       System.Boolean  1 instance                0 _AsyncCommandInProgress
000007fee4fb63e0  400107e       30 ...ent.SqlStatistics  0 instance 0000000000000000 _statistics
000007fee3ec4f30  400107f       71       System.Boolean  1 instance                0 _collectstats
000007fee3ec4f30  4001080       72       System.Boolean  1 instance                0 _fireInfoMessageEventOnUserErrors
000007fee553de48  4001081       38 ....Task, mscorlib]]  0 instance 0000000000000000 _currentCompletion
000007fee4f998b0  4001082       40 ...ent.SqlCredential  0 instance 0000000000000000 _credential
000007fee3ec3658  4001083       48        System.String  0 instance 0000000100dc2a10 _connectionString
000007fee4f9c0e0  4001086       50 ...ConnectionOptions  0 instance 0000000000000000 _userConnectionOptions
000007fee4f9df40  4001087       58 ...nnectionPoolGroup  0 instance 0000000000000000 _poolGroup
000007fee4f9d730  4001088       60 ...onnectionInternal  0 instance 00000003ffdf82e0 _innerConnection
000007fee3ec6408  4001089       68         System.Int32  1 instance                1 _closeCount
000007fee3ec6408  400108b       6c         System.Int32  1 instance            75433 ObjectID

00000004c0710338

Name:        System.Data.SqlClient.SqlConnection
MethodTable: 000007fee4f96170
EEClass:     000007fee4dd8110
Size:        128(0x80) bytes
File:        D:WindowsMicrosoft.NetassemblyGAC_64System.Datav4.0_4.0.0.0__b77a5c561934e089System.Data.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fee3ec3c38  4000198        8        System.Object  0 instance 0000000000000000 __identity
000007fee5a5f2e8  40002d2       10 ...ponentModel.ISite  0 instance 0000000000000000 site
000007fee5a6dbe0  40002d3       18 ....EventHandlerList  0 instance 00000004c0711d08 events
000007fee3ec3c38  40002d1      1b8        System.Object  0   shared           static EventDisposed
                                 >> Domain:Value  0000000001770530:NotInit  000000000fe9e740:00000003ffe0a4b0 <<
000007fee4fb1510  40002c0       20 ...hangeEventHandler  0 instance 0000000000000000 _stateChangeEventHandler
000007fee4fae4c0  400107c       28 ...t.SqlDebugContext  0 instance 0000000000000000 _sdc
000007fee3ec4f30  400107d       70       System.Boolean  1 instance                0 _AsyncCommandInProgress
000007fee4fb63e0  400107e       30 ...ent.SqlStatistics  0 instance 0000000000000000 _statistics
000007fee3ec4f30  400107f       71       System.Boolean  1 instance                0 _collectstats
000007fee3ec4f30  4001080       72       System.Boolean  1 instance                0 _fireInfoMessageEventOnUserErrors
000007fee553de48  4001081       38 ....Task, mscorlib]]  0 instance 0000000000000000 _currentCompletion
000007fee4f998b0  4001082       40 ...ent.SqlCredential  0 instance 0000000000000000 _credential
000007fee3ec3658  4001083       48        System.String  0 instance 00000004c0710270 _connectionString
000007fee4f9c0e0  4001086       50 ...ConnectionOptions  0 instance 000000023fdf8250 _userConnectionOptions
000007fee4f9df40  4001087       58 ...nnectionPoolGroup  0 instance 000000023fdf8578 _poolGroup
000007fee4f9d730  4001088       60 ...onnectionInternal  0 instance 0000000400005d30 _innerConnection
000007fee3ec6408  4001089       68         System.Int32  1 instance                0 _closeCount
000007fee3ec6408  400108b       6c         System.Int32  1 instance            74597 ObjectID
000007fee3ec3c38  400107b      ea0        System.Object  0   shared           static EventInfoMessage

<...>

You will notice this last connection object is still open (_closeCount is zero!). Now let's make sure it's unreachable (i.e. leaked!).

0:108> !GCRoot 00000004c0710338
Found 0 unique roots (run '!GCRoot -all' to see all roots).

Ok, so it's unreachable and open, it's therefore a leaked connection - now let's see if we can tell more than just the connection string by dumping the inner connection (the outer is just a shell):

0:108> !do  0000000400005d30
Name:        System.Data.SqlClient.SqlInternalConnectionTds
MethodTable: 000007fee4f9e8a8
EEClass:     000007fee4e00cd0
Size:        296(0x128) bytes
File:        D:WindowsMicrosoft.NetassemblyGAC_64System.Datav4.0_4.0.0.0__b77a5c561934e089System.Data.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fee3ec6408  40008b8       38         System.Int32  1 instance              280 _objectID
000007fee3ec4f30  40008bb       44       System.Boolean  1 instance                0 _allowSetConnectionString
000007fee3ec4f30  40008bc       45       System.Boolean  1 instance                1 _hidePassword
000007fee4f97c68  40008bd       3c         System.Int32  1 instance                1 _state
000007fee3ec0850  40008be        8 System.WeakReference  0 instance 0000000400005ec8 _owningObject
000007fee4f9bfc8  40008bf       10 ....DbConnectionPool  0 instance 000000023fdf8950 _connectionPool
000007fee4f98ca8  40008c0       18 ...ctionPoolCounters  0 instance 000000047ff5dc88 _performanceCounters
000007fee4f9a6c8  40008c1       20 ...ferenceCollection  0 instance 000000040002bbe0 _referenceCollection
000007fee3ec6408  40008c2       40         System.Int32  1 instance                0 _pooledCount
000007fee3ec4f30  40008c3       46       System.Boolean  1 instance                0 _connectionIsDoomed
000007fee3ec4f30  40008c4       47       System.Boolean  1 instance                0 _cannotBePooled
000007fee3ec4f30  40008c5       48       System.Boolean  1 instance                0 _isInStasis
000007fee3ec0980  40008c6       50      System.DateTime  1 instance 0000000400005d80 _createTime
000007fef08f39d8  40008c7       28 ...tions.Transaction  0 instance 0000000000000000 _enlistedTransaction
000007fef08f39d8  40008c8       30 ...tions.Transaction  0 instance 0000000000000000 _enlistedTransactionOriginal
000007fee3ec6408  40008b7      6e0         System.Int32  1   shared           static _objectTypeCount
                                 >> Domain:Value  0000000001770530:NotInit  000000000fe9e740:353 <<
000007fee4f9f5d8  40008b9      618 ...teChangeEventArgs  0   shared           static StateChangeClosed
                                 >> Domain:Value  0000000001770530:NotInit  000000000fe9e740:000000047ff62d98 <<
000007fee4f9f5d8  40008ba      620 ...teChangeEventArgs  0   shared           static StateChangeOpen
                                 >> Domain:Value  0000000001770530:NotInit  000000000fe9e740:000000047ff62db0 <<
000007fee4f9ae10  4001258       58 ...lConnectionString  0 instance 000000023fdf8250 _connectionOptions
000007fee3ec4f30  4001259       88       System.Boolean  1 instance                0 _isEnlistedInTransaction
000007fee3ec77d0  400125a       60        System.Byte[]  0 instance 0000000000000000 _promotedDTCToken
000007fee3ec77d0  400125b       68        System.Byte[]  0 instance 0000000000000000 _whereAbouts
000007fee3ec3658  400125c       70        System.String  0 instance 0000000400027c20 <CurrentDatabase>k__BackingField
000007fee3ec3658  400125d       78        System.String  0 instance 000000023fdf8368 <CurrentDataSource>k__BackingField
000007fee5552358  400125e       80 ...egatedTransaction  0 instance 0000000000000000 <DelegatedTransaction>k__BackingField
000007fee4f9ccc0  400126e       90 ...GroupProviderInfo  0 instance 000000023fdf85c0 _poolGroupProviderInfo
000007fee4f99ad8  400126f       98 ...lClient.TdsParser  0 instance 0000000400005fb0 _parser
000007fee4f9dbb0  4001270       a0 ...lient.SqlLoginAck  0 instance 0000000400027ca8 _loginAck
000007fee4f998b0  4001271       a8 ...ent.SqlCredential  0 instance 0000000000000000 _credential
[...]

Now let's check the TDS Parser (TDS = Tabular Data Stream) - to get the actual data:

0:108> !do 0000000400005fb0
Name:        System.Data.SqlClient.TdsParser
MethodTable: 000007fee4f99ad8
EEClass:     000007fee4dff318
Size:        160(0xa0) bytes
File:        D:WindowsMicrosoft.NetassemblyGAC_64System.Datav4.0_4.0.0.0__b77a5c561934e089System.Data.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fee3ec6408  40014d6       70         System.Int32  1 instance              275 _objectID
000007fee4f9dcc0  40014d8        8 ...ParserStateObject  0 instance 0000000400006050 _physicalStateObj
000007fee4f9dcc0  40014d9       10 ...ParserStateObject  0 instance 0000000000000000 _pMarsPhysicalConObj
000007fee4fb9758  40014da       74         System.Int32  1 instance                2 _state
000007fee3ec3658  40014db       18        System.String  0 instance 000000023fdf8368 _server
000007fee3ec4f30  40014dc       88       System.Boolean  1 instance                0 _fResetConnection
000007fee3ec4f30  40014dd       89       System.Boolean  1 instance                0 _fPreserveTransaction
[...]

We then have to look at the TDS Parser State Object:

0:108> !do 0000000400006050
Name:        System.Data.SqlClient.TdsParserStateObject
MethodTable: 000007fee4f9dcc0
EEClass:     000007fee4e009b8
Size:        424(0x1a8) bytes
File:        D:WindowsMicrosoft.NetassemblyGAC_64System.Datav4.0_4.0.0.0__b77a5c561934e089System.Data.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fee3ec6408  40015a4      128         System.Int32  1 instance              275 _objectID
000007fee4f99ad8  40015a5        8 ...lClient.TdsParser  0 instance 0000000400005fb0 _parser
000007fee4f99528  40015a6       10 ...lClient.SNIHandle  0 instance 0000000400027ab8 _sessionHandle
000007fee3ec0850  40015a7       18 System.WeakReference  0 instance 00000004000061f8 _owner
000007fee3ec6408  40015a8      12c         System.Int32  1 instance                0 _activateCount
000007fee3ec6408  40015a9      130         System.Int32  1 instance                8 _inputHeaderLen
000007fee3ec6408  40015aa      134         System.Int32  1 instance                8 _outputHeaderLen
000007fee3ec77d0  40015ab       20        System.Byte[]  0 instance 0000000400029c68 _outBuff
000007fee3ec6408  40015ac      138         System.Int32  1 instance                8 _outBytesUsed
000007fee3ec77d0  40015ad       28        System.Byte[]  0 instance 0000000400027d10 _inBuff
000007fee3ec6408  40015ae      13c         System.Int32  1 instance             5084 _inBytesUsed
000007fee3ec6408  40015af      140         System.Int32  1 instance             5084 _inBytesRead
000007fee3ec6408  40015b0      144         System.Int32  1 instance                0 _inBytesPacket
000007fee3ec5ab8  40015b1      170          System.Byte  1 instance                3 _outputMessageType
000007fee3ec5ab8  40015b2      171          System.Byte  1 instance                1 _messageStatus
000007fee3ec5ab8  40015b3      172          System.Byte  1 instance                1 _outputPacketNumber
000007fee3ec4f30  40015b4      173       System.Boolean  1 instance                0 _pendingData

[...]

_outBuff is our output buffer, i.e. what we sent to the server, and it has our command text... Now I know who's leaking!

0:108> !do 0000000400029c68
Name:        System.Byte[]
MethodTable: 000007fee3ec77d0
EEClass:     000007fee38ca4b8
Size:        8024(0x1f58) bytes
Array:       Rank 1, Number of elements 8000, Type Byte
Content:     ................................

]]>