Open the trace using the following command line or File –> Open and load the manifest from the Framework folder or launch the following from the command line. 

SvcPerf.exe wcf.etl %windir%\Microsoft.Net\Framework\v4.0.30319\Microsoft.Windows.ApplicationServer.Applications.45.man

Activity correlation can be obtained by clicking on the activity cell of the operation you are interested in as shown below and Ctrl+Shift+C should add a filter for the column you have chosen.

For more filters see – Filter in SvcPerf

In 4.5 we introduced ETW tracing for WCF. In the past couple of months we have been trying to establish some common tools that can be used to debug and analyze ETW traces from WCF. We have a large number of tools that allow ETW analysis and the one aspect we wanted to do was to allow activity correlation which WCF uses extensively. Folks who have already used ETW in its full capacity already know about ETW activities and how they are propagated. Those of you who don't, this FAQ: Common Questions for ETW and Windows Event Log should be a good start point. Others who just want to jump in, hop over to the tool page here -https://svcperf.codeplex.com/

 Pros of ETW

1. Ability to control it dynamically compared to System.Diagnostics
2. One of the fastest tracing mechanisms in the Operating System
3. Well understood formats of producing and consuming traces. 
4. Inherent concepts of Activities and correlation. 
5. Real-time analysis capabilities, even in production environments.

Some of the not so great features are

1. Lot of tools available which causes confusion in the tool to be used. 
2. Cannot be easily read with text analysis tools due to binary format of the file
3. File format can be better optimized.
4. Collection is machine wide and can impact the system if there are large number of WCF services on the box.

Despite these issues ETW offers one of the most robust tracing infrastructures on the platform. With that in mind we build our End-to-End(E2E) tracing for WCF. We needed to make multiple layers understand and consistent with our E2E model. We also needed to be able to correlate a large number of scenarios from our transports all the way to our dispatchers. We have large number of pumps in multiple layers and being extremely asynchronous means that you lose the ability to depend on things like TLS and ExecutionContext etc. With these in mind we choose a model where we explicitly flow our correlation across layers and use it when stamping our events. This also means that we need to be able to stitch causality chains for request analysis.

• Note: This does not deprecate Diagnostics Tracing and neither is it guaranteed to provide feature parity. Instead we consider this as new capability which we can use for production analysis cases where we would not like process restarts or configuration changes to running applications. Also for 4.5 we do not have message logging which still requires configuration and can be done only through Diagnostics Tracing.

Why do we need another tool?

The other more popular ETW tools like Xperf and WPA have a good amount of analysis capability already built in them but they are not tuned for statistical analysis of data. For e.g. given a trace how do you find out which are your slow requests or how to obtain a distribution of your request processing times. These kinds of question were better answered with SQL like queries. That said there are log parsers that have this capability but we wanted to work on raw ETL and this is where SvcPerf comes into play. Besides being a quick ETL viewer it also is a query engine built on top of Tx (LINQ over Traces).

In the above screen shot you can see that Activities are correlated for you and the ETW fields like ActivityID and RelatedActivityID are can be directly queried.

If you want to just give the tool a try you can get it from here. SvcPerf.exe (It is self-signed and will give you a warning but if you want you can try to compile the source as well provided you remove the strongname.).

To collect the trace use logman with the WCF provider and you also need the manifest so that events can be decoded. Refer - https://svcperf.codeplex.com/wikipage?title=How%20do%20I%20collect%20an%20ETL%20trace%20for%20WF%2fWCF%3f&referringTitle=FAQs

In the coming posts I will try to take up some simple scenarios where this tools might help in analysis.

1. Request Duration analysis and Drill Downs
2. Custom Queries and Graphs
3. Synthetic Performance Counters
4. Template Queries

https://kennyw.com/indigo/51

I’m going to talk about the internals of WCF Bufferpooling as it stands today. Kenny and Andrew already have a posts on bufferpooling that cover them at a higher level. Here I dive into how the bufferPool was optimized further for working with the large object heap. This didn’t make it into .NET 4.0 RTM but we created a hotfix that customers can download if they hit this specific issue of fragmentation and garbage collection - https://support.microsoft.com/kb/983182

This doesn’t impact most customers and since under steady state and when the application has been running for a while they generally won’t observer any issue at all.

The Problem

For services which were transferring a large amount of data per operation, we noticed a large number of un-rooted buffers in the large object heap (LOH - https://msdn.microsoft.com/en-us/magazine/cc534993.aspx). The expectation is that the buffer manager would reuse and hold on to the buffers that were allocated. But there were a large number of objects in the large object heap that did not have any valid root indicating that there were allocated and dropped. The number of such non-rooted objects would spike sometimes and cause the LOH to grow to about 1.8GB until a full GC occurs. This is generally not an issue once the application reaches a stable state but until it does there can be very large GC churn.

Here is a snapshot from the debugger which shows the LOH and we see only 2 objects rooted to the buffer pool

You can see that the memory has a large number of 8MB chunks of which only a few are rooted to the bufferPool and the rest are actually available for garbage collection which can take a while.  Such a spike in memory might be of concern for memory sensitive app. Remember these will be reclaimed once a full GC happens. For most application this would not be an issue since over time the pool would be trained automatically and you would see the applications usage reflect the allocation patterns.

Analysis

Internally we have a class called the PooledBufferManager and to understand this issue we need to have an idea as to how the PooledBufferManager uses the InternalBufferPool where the actual implementation exists. The InternalBufferPool uses a type called SynchronizedPool<byte[]>. The main purpose of the SynchronizedPool is the lock free nature and training heuristics it provides. It has a capability to build thread affinity for buffers and tune quota depending on usage. This is one highly tuned piece of engineering. The PooledBufferManager holds a number of BufferPools of whose sizes increase by power of 2 and one pool who size is equals the MaxBufferSize.

The Synchronized pool itself contains a locked global pool and it starts off with space for just a single entry. As the pool gets used more and more the number of takes and returns are used to compute whether the number of items in a BufferPool should grow which is done by PooledBufferManager. The side effect is that till the size grows large enough, buffers can’t be returned and all the newly allocated buffers would be dropped and GC would have to collect them. Concurrent requests would obviously cause a number of such buffers in flight but only a few can be returned till the buffer pool grows enough to hold all the returns. The training does not allow very fast growth and the size of the pools grow only one at a time.

This is not a problem for object lesser than 85000 since they can be collected by ephemeral GC mostly in gen0. The issue with objects that are larger than 85000 is that they would be allocated on the large object heap which does not get collected until a full GC actually happens. So during training large buffers would be dropped but not collected and would wait for a full GC till they are reclaimed.

The second part of this issue occurs when a particular buffer builds an affinity with a thread. Once a buffer is associated with a thread, it is moved from its global pool into the per thread pool. This also means that that a buffer can be taken and returned only by that particular thread until enough number of misses occurs to promote another thread. If another thread tries to return the buffer but the total number of buffers/entries in the SynchronizedPool is already maxed out then these would also be dropped and requests by other threads would also cause allocation.

Due to these two major code flow paths of the Synchronized pool we noticed very large number of drops and allocations.

Solution

To avoid hitting this code path for large objects we use a simpler pool that does not have any training and only a synchronized stack like the GlobalPool in the SynchronizedPool<T>. This would cause quota tuning to occur much faster and also never drop a buffer if there is available capacity and since there is no thread affinity there would not be any drops during return if there is an available slot on the stack.

Hence we need the BufferPool to support two different kinds of behaviors.

1. For buffers < 85000 we need the default behavior of the buffer pool since scenarios with very small amounts of data are very sensitive to changes.

2. For buffers >=85000 we use a new simpler Pool that doesn’t use the SynchronizedPool<T> and will save GC cost.

This means that the buffer pool would hold onto LOH objects much more aggressively. The result is a POOL with larger buffers will be formed more quickly and hence the reduce GC since there would be lesser large buffer allocations.

The term used is is very subjective. For a developer it would mean that you probably never created an instance or called a method on it. But this does not cover the whole story. You can instead consider what are the reasons for an assembly load occurring. Suzanne’s blog on Assembly loading Failures would give you a good understanding of failures if that is what you are interested in. This post focuses on how to identify what exactly is causing an assembly to load. 

We in the WCF team are very cautious on introducing assembly dependencies and how how our code paths can cause assembly loads since this impacts the reference set of your process. Images that get loaded during a WCF call can become the cause of slow start up since every assembly is a potential disk look up and larger the number the higher the impact to startup.  As a guidance for quick app startup is that you can eliminate a lot of the unnecessary assemblies from being loaded to speed up application startup if you refactor types properly.

To demonstrate the example here is  a simple application with 2 dependencies – For a simple program shown below will “b.dll” be loaded?

namespace App
{
    using a;
    using b;

    class Program
    {
        static void Main(string[] args)
        {
            Invoke();
        }

        private static void Invoke()
        {
            TestClass.Helper();
        }
    }

    class TestClass
    {
        public static FromB testInstance = null;

        public static void Helper() {}
        public static void Helper(FromA b){}
        public static void Helper(FromB b) { }
    }
}

YES!! Even though TestClass doesn’t actually have an type from b.dll  instantiated. There is a type defined which would cause JIT to resolve the type in TestClass when the static method is invoked on that. However a.dll is not loaded because JIT doesn’t need to compile all static methods on the type. Here is the stack when b.dll is loaded. You can see that the field is being resolved which will cause a module load. You can set a break point on module load using sxeld <dllName>

0:000> sxeld b.dll
0:000> g
(1a18.1a64): Unknown exception - code 04242420 (first chance)
ModLoad: 5bb00000 5bb08000   b.dll
eax=00000000 ebx=00000000 ecx=00000002 edx=00000000 esi=766e4cc8 edi=0046b65c
eip=77a9fc02 esp=0039a668 ebp=0039a6a0 iopl=0         nv up ei pl nz na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206
ntdll!ZwMapViewOfSection+0x12:
77a9fc02 83c404          add     esp,4

0:000> k
ChildEBP RetAddr
0039a668 76b6dfae ntdll!ZwMapViewOfSection+0x12
0039a6a0 6e366ffc KERNELBASE!MapViewOfFileEx+0x81
0039a6e8 6e36703e clr!CLRMapViewOfFileEx+0x23
0039a708 6e37366a clr!CLRMapViewOfFile+0x19
0039a970 6e373745 clr!MappedImageLayout::MappedImageLayout+0x1ce
0039a9ac 6e3737b2 clr!PEImageLayout::Map+0x2d
0039a9e0 6e366467 clr!PEImage::GetLayoutInternal+0x7c
0039aa28 6e3641a8 clr!PEImage::GetLayout+0xdd
0039aac0 6e363ce2 clr!RuntimeOpenImageInternal+0x131
0039ab04 6e363d6c clr!GetAssemblyMDInternalImportEx+0x9c
0039ab1c 6e47dfda clr!CreateMetaDataImport+0x16
0039ab58 6e47e08c clr!CAssemblyManifestImport::InitAndLoadMetaData+0x4f
0039ab88 6e47e610 clr!CreateAssemblyManifestImport+0x67
0039b24c 6e47f5fa clr!CAsmDownloadMgr::CreateAssembly+0x208
0039b5a4 6e47eb8c clr!CAsmDownloadMgr::DoSetupRFS+0xb7
0039b828 6e47e973 clr!CAsmDownloadMgr::DoSetup+0x22d
0039b888 6e47cd09 clr!CAssemblyDownload::DoSetup+0xae
0039b8bc 6e47ce9e clr!CAssemblyDownload::DownloadComplete+0xb6
0039bb24 6e47cf03 clr!CAssemblyDownload::KickOffDownload+0x35e
0039bbb4 6e36ae31 clr!CAssemblyName::BindToObject+0x8be
0039bc48 6e36aff2 clr!FusionBind::RemoteLoad+0x229
0039bcd4 6e36b180 clr!FusionBind::LoadAssembly+0x116
0039bf84 6e369e3b clr!AssemblySpec::FindAssemblyFile+0xf4
0039ccb4 6e367d35 clr!AppDomain::BindAssemblySpec+0x969
0039cd58 6e367bc1 clr!PEFile::LoadAssembly+0xbf
0039ce10 6e367e63 clr!Module::LoadAssembly+0x137
0039d0a4 6e321f02 clr!Assembly::FindModuleByTypeRef+0x256
0039d0f4 6e30f8fc clr!ClassLoader::LoadTypeDefOrRefThrowing+0xfb
0039d218 6e330a2a clr!SigPointer::GetTypeHandleThrowing+0x960
0039d26c 6e330b7d clr!CEEInfo::getFieldTypeInternal+0x8b
0039d310 6f864b13 clr!CEEInfo::getFieldInfo+0x35c
0039da30 6f85277d clrjit!Compiler::impImportBlockCode+0x3641
0039da94 6f85291e clrjit!Compiler::impImportBlock+0x7a
0039daac 6f85296a clrjit!Compiler::impImport+0x1bf
0039dab8 6f8529a7 clrjit!Compiler::fgImport+0x20
0039dac8 6f853b02 clrjit!Compiler::compCompile+0x45
0039db04 6f853c1a clrjit!Compiler::compCompileHelper+0x2dd
0039db78 6f853d54 clrjit!Compiler::compCompile+0x1e2
0039dc58 6f8545d9 clrjit!jitNativeCode+0x154
0039dc7c 6e3334b6 clrjit!CILJit::compileMethod+0x25
0039dce0 6e333535 clr!invokeCompileMethodHelper+0x5c
0039dd28 6e33357b clr!invokeCompileMethod+0x31
0039dd90 6e3337e1 clr!CallCompileMethodWithSEHWrapper+0x2e
0039e0f8 6e33f817 clr!UnsafeJitFunction+0x3eb
0039e1d0 6e33f97f clr!MethodDesc::MakeJitWorker+0x288
0039e240 6e3264f6 clr!MethodDesc::DoPrestub+0x49d
0039e2b8 6e302dbf clr!PreStubWorker+0x134
0039e2e8 6e30297e clr!ThePreStub+0x16
0039e2f8 6e323bc9 clr!CallDescrWorker+0x34
0039e374 6e324dfb clr!CallDescrWorkerWithHandler+0x8d
0039e400 6e324e64 clr!DispatchCallDebuggerWrapper+0x6f
0039e43c 6e328198 clr!DispatchCallSimple+0x4a
0039e4b4 6e3280cc clr!MethodTable::RunClassInitEx+0xdb
0039ee00 6e3ad21d clr!MethodTable::DoRunClassInitThrowing+0x4e5
0039ee68 6e3264f6 clr!MethodDesc::DoPrestub+0x11b
0039eee0 6e302dbf clr!PreStubWorker+0x134
0039ef10 005900c8 clr!ThePreStub+0x16
0039ef18 0059008c ConsoleApplication1!App.Program.Invoke()+0x18
0039ef24 6e30297e ConsoleApplication1!App.Program.Main(System.String[])+0x1c
0039ef34 6e323bc9 clr!CallDescrWorker+0x34
0039efb0 6e3246ec clr!CallDescrWorkerWithHandler+0x8d
0039f0f4 6e32471f clr!MethodDesc::CallDescr+0x18c
0039f110 6e32473d clr!MethodDesc::CallTargetWorker+0x1f
0039f128 6e4a863d clr!MethodDescCallSite::Call_RetArgSlot+0x1a
0039f290 6e4a8582 clr!ClassLoader::RunMain+0x24c
0039f4f8 6e4a8b98 clr!Assembly::ExecuteMainMethod+0xbf
0039fa00 6e4a8c24 clr!SystemDomain::ExecuteMainMethod+0x568
0039fa58 6e4a8d5c clr!ExecuteEXE+0x58
0039faa4 6e452e85 clr!_CorExeMainInternal+0x19a
0039fadc 716ba791 clr!_CorExeMain+0x4d
0039faec 71727f16 mscoreei!_CorExeMain+0x4a
0039fafc 71724de3 MSCOREE!ShellShim__CorExeMain+0x99
0039fb04 766e3677 MSCOREE!_CorExeMain_Exported+0x8
0039fb10 77ab9d72 KERNEL32!BaseThreadInitThunk+0xe
0039fb50 77ab9d45 ntdll!__RtlUserThreadStart+0x70
0039fb68 00000000 ntdll!_RtlUserThreadStart+0x1b
0:000>

c:\> logman -ets

Data Collector Set                      Type                          Status
-------------------------------------------------------------------------------
AITEventLog                             Trace                         Running
Audio                                   Trace                         Running
DiagLog                                 Trace                         Running
EventLog-Application                    Trace                         Running
EventLog-System                         Trace                         Running
NtfsLog                                 Trace                         Running
SQMLogger                               Trace                         Running
UBPM                                    Trace                         Running
WdiContextLog                           Trace                         Running
MpWppTracing                            Trace                         Running
FSysAgentTrace                          Trace                         Running
MSMQ                                    Trace                         Running
MSDTC_TRACE_SESSION                     Trace                         Running
test_trace                              Trace                         Running

The command completed successfully.


c:\> logman test_trace -ets

Name:                 test_trace
Status:               Running
Root Path:            C:\
Segment:              Off
Schedules:            On
Segment Max Size:     500 MB

Name:                 test_trace\test_trace
Type:                 Trace
Output Location:      C:\09_19_44.etl
Append:               Off
Circular:             On
Overwrite:            Off
Buffer Size:          8
Buffers Lost:         0
Buffers Written:      1
Buffer Flush Timer:   0
Clock Type:           Performance
File Mode:            File

Provider:
Name:                 Microsoft-Windows-Application Server-Applications
Provider Guid:        {C651F5F6-1C0D-492E-8AE1-B4EFD7C9D503}
Level:                5
KeywordsAll:          0x0
KeywordsAny:          0xffffffff
Properties:           0
Filter Type:          0

The command completed successfully.

Here is something for you performance guys. This was a hard one to crack but. Let me know what you think of this and how you would like to see this evolve. It will be a while before this is actually available.

Thanks to Wenlong for driving this all the way and for being our custodian for WCF/WF performance.