But there are problems with performance counters. Problems that should be fixed but seem to receive no attention. Here is my list.

1. Performance Monitor shows different data on different machines

Performance counter data can be recorded to files using logman. The resulting files have a .blg extension. This is great for performance testing since I can keep all the perf counter data in a compact log file for each test. The problem comes when I try to view these blg files. Here is a file that was produced on a Windows Server 2008 R2 VM opened in performance monitor on a different VM with the same OS:

Now here is the same blg file opened from a Windows 8 machine:

This is not consistent either. Most of the time I can open BLG files on my Windows 8 machine without a problem. The trouble is I don't know if it's limited to just Windows 8.

The good thing is that this is just a viewer. The blg file still has the correct data. So how about using a different viewer?

2. There are no libraries to read BLG files

Microsoft has never published the official specifications for the BLG format. There are no libraries available to read the format either. It is possible to reverse-engineer the file format to create a library, but you can never be 100% certain that it works in all situations. Given the inconsistencies in the viewers, I wonder if Microsoft itself has lost the specifications for this file format.

The good thing is that you can convert BLG to CSV format using relog.

3.  The first line of a converted CSV has empty values

When you convert a BLG file to a CSV, the first line below the header contains some empty values. In relog you can specify a -b parameter to begin at a different timestamp. No matter what you set this to, the first line of the CSV has empty values. Here is a look at a converted file:

The first line has empty spaces in it for some of the values but not all. If you move to 5:22:42 (since the -b parameter is only good up to seconds), you'll get this instead:

While this isn't a big deal, it does mean that if you want counters from a specific time period, you have to factor in that this first line tends to be incomplete.

4. Combining several BLGs into one is limited to 32 files and has poor error messages 

Let's say I set up a rolling log in logman to generate a new BLG file every 10 minutes. I want to examine a period of several hours so I take the BLG files generated from that time period. Using relog, I can combine these files into one BLG so that I don't have to open lots of windows and try to understand what happened in what order. In my case, I have 37 BLG files. When I run the command to combine the files, I get a strange error:

The error "The specified log file type has not been installed on this computer" appears to be a catch-all error for relog. One would think that maybe one of the files is corrupt but I can combine into two different groups and combine the results of that without an issue:

C:\Temp\2013-08-08\origin0>relog perfcounters_08080*.blg -o combined0.blg

Input
----------------
File(s):
     perfcounters_08080451.blg (Binary)
     perfcounters_08080501.blg (Binary)
     perfcounters_08080511.blg (Binary)
     perfcounters_08080521.blg (Binary)
     perfcounters_08080531.blg (Binary)
     perfcounters_08080541.blg (Binary)
     perfcounters_08080551.blg (Binary)
     perfcounters_08080601.blg (Binary)
     perfcounters_08080611.blg (Binary)
     perfcounters_08080621.blg (Binary)
     perfcounters_08080631.blg (Binary)
     perfcounters_08080641.blg (Binary)
     perfcounters_08080651.blg (Binary)
     perfcounters_08080701.blg (Binary)
     perfcounters_08080711.blg (Binary)
     perfcounters_08080721.blg (Binary)
     perfcounters_08080731.blg (Binary)
     perfcounters_08080741.blg (Binary)
     perfcounters_08080751.blg (Binary)
     perfcounters_08080801.blg (Binary)
     perfcounters_08080811.blg (Binary)
     perfcounters_08080821.blg (Binary)
     perfcounters_08080831.blg (Binary)
     perfcounters_08080841.blg (Binary)
     perfcounters_08080851.blg (Binary)
     perfcounters_08080901.blg (Binary)
     perfcounters_08080911.blg (Binary)
     perfcounters_08080921.blg (Binary)
     perfcounters_08080931.blg (Binary)
     perfcounters_08080941.blg (Binary)
     perfcounters_08080951.blg (Binary)

Begin:    8/8/2013 4:51:02
End:      8/8/2013 10:01:21
Samples:  18623

100.00%

Output
----------------
File:     combined0.blg

Begin:    8/8/2013 4:51:02
End:      8/8/2013 10:01:21
Samples:  18623

The command completed successfully.

C:\Temp\2013-08-08\origin0>relog perfcounters_08081*.blg -o combined1.blg

Input
----------------
File(s):
     perfcounters_08081001.blg (Binary)
     perfcounters_08081011.blg (Binary)
     perfcounters_08081021.blg (Binary)
     perfcounters_08081031.blg (Binary)
     perfcounters_08081041.blg (Binary)
     perfcounters_08081051.blg (Binary)

Begin:    8/8/2013 10:01:22
End:      8/8/2013 11:01:24
Samples:  3605

100.00%

Output
----------------
File:     combined1.blg

Begin:    8/8/2013 10:01:22
End:      8/8/2013 11:01:24
Samples:  3605

The command completed successfully.

C:\Temp\2013-08-08\origin0>relog combined*.blg -o combined.blg

Input
----------------
File(s):
     combined0.blg (Binary)
     combined1.blg (Binary)

Begin:    8/8/2013 4:51:02
End:      8/8/2013 11:01:24
Samples:  22228

100.00%

Output
----------------
File:     combined.blg

Begin:    8/8/2013 4:51:02
End:      8/8/2013 11:01:24
Samples:  22228

The command completed successfully.

So perhaps the issue is the number of files. By removing files one at a time, I found that relog will handle 32 files maximum. This is not documented nor is the error message able to communicate this clearly.

5. Combining BLG files from multiple machines does not work well

While this may be stretching what relog was intended to do, I think there is a perfectly reasonable case for wanting to do this. If you have a cluster of machines performing the same function, you may want to get an average CPU or a total throughput for the whole cluster. I do this all the time in Azure. If you collect the performance counters from each machine, then it should be straightforward to combine these data. Unfortunately it's not. The counters in the BLG have the machine name included. So when you combine the counters don't mix. You have to do this by hand:

1. Convert each BLG file to a CSV
2. In each CSV replace the machine name in the perf counter with a generic name like "server"
3. Read the data from each CSV simultaneously using a rolling window and combine the perf counters in the time window

In other words, forget about it and pay for a tool that will do it for you. 

6. Relog has trouble reading dates from the command line

Look at the following output:

c:\test>relog perf1.blg -f csv -o perf1.csv -b "8/31/2013 0:52:14" -e "8/31/2013 1:02:41"

Input
----------------
File(s):
     perf1.blg (Binary)

Begin:    8/31/2013 0:52:16
End:      8/31/2013 1:02:41
Samples:  626

100.00%

Output
----------------
File:     blah.csv

Begin:    8/31/2013 0:52:16
End:      8/31/2013 1:02:41
Samples:  625

The command completed successfully.

c:\test>relog perf1.blg -f csv -o perf1.csv -b "8/31/2013 12:52:14AM" -e "8/31/2013 1:02:41AM"

Input
----------------
File(s):
     perf1.blg (Binary)

Begin:    8/31/2013 0:52:16
End:      8/31/2013 1:02:41
Samples:  626

Error: The time range specified has no overlap with the input logs.

The relog documentation says you can use timestamps with AM/PM but does not explicitly say you can use 24 hour time. Also, when it shows begin and end, those appear to be the same in both commands. But one works and one does not.

Kripke and Sheldon were working on the same project and the university required them to work together. Regular viewers of the show know that Sheldon is very difficult to work with. After reluctantly exchanging notes on their theories, Sheldon was stunned to find out Kripke's theories were better. What I really liked was how Kripke handled the situation. If he gloated, Sheldon would be resistant and the project would suffer.

Kripke, whether intentionally or not, gave Sheldon a way to save face by attributing his less-than-stellar performance to his preoccupation with his girlfriend. Then Kripke followed up with a compliment by telling Sheldon that he had some great ideas. He then challenged Sheldon to work harder.

In other words, Kripke applied Dale Carnegie principles to turn a difficult person into a willing team member. I have no idea if that was what the show's writers we're going for but I enjoyed the episode.

I'm curious what other people use to write code snippets. I'd be glad to hear any suggestions.

The profiler of choice is PerfView. If you search Channel 9 you'll find all kinds of great videos on how to use PerfView for analysis of performance problems. PerfView's command line options allow you to start and stop profiles with separate calls. There is also built-in support to stop collection on performance counters, slow HTTP requests, long GC collects, and events in the Application event log matching a regular expression pattern. However, let's say you are getting an occasional WCF exception in production. Rather than write a message to the Application event log, you could just listen for the event directly.

I put together a C# solution for this using the EventTraceWatcher helper class which is an excellent tool put together by Daniel Vasquez Lopez. The main code for this is very straightforward:

using Samples.Eventing;
using System;
using System.Diagnostics;
using System.Threading;

namespace CircularTrace
{
    class Program
    {
        private static ManualResetEvent waitForEvent = new ManualResetEvent(false);
        static void Main(string[] args)
        {
            try
            {
                RunProcess("PerfView.exe", "start -AcceptEULA -NoGui -Circular:250 -logfile:perfview_start.log -ThreadTime");

                // Wait for a WCF exception to occur
                Guid RewriteProviderId = new Guid("c651f5f6-1c0d-492e-8ae1-b4efd7c9d503");

                using (EventTraceWatcher watcher = new EventTraceWatcher("WcfErrors", RewriteProviderId))
                {
                    watcher.EventArrived += delegate(object sender, EventArrivedEventArgs e)
                    {
                        waitForEvent.Set();
                    };

                    // Start listening
                    watcher.Start();
                    waitForEvent.WaitOne();
                }

                RunProcess("PerfView.exe", "stop -AcceptEULA -NoGui -logfile:perfview_stop.log");
            }
            catch (Exception exc)
            {
                Console.WriteLine(exc);
            }
        }

        private static void RunProcess(string executable, string arguments)
        {
            Process process;
            ProcessStartInfo startInfo;

            process = new Process();
            startInfo = new ProcessStartInfo(executable, arguments);
            startInfo.UseShellExecute = false;
            process.StartInfo = startInfo;
            process.Start();
            process.WaitForExit();
        }
    }
}

This starts PerfView with a 250MB circular buffer. The AcceptEULA and NoGui parameters are probably not necessary with the logfile parameter specified. In the middle the wait is implemented by watching the WCF ETW provider on an existing session named WcfErrors. That session must be setup prior to running this tool.

Here are instructions on how to create the session, which are the same as Daniel included in his blog:

1. Open the Performance Monitor tool (should be under Administrative Tools)
2. Locate Data Collector Sets -> Event Trace Sessions in the tree on the left
3. Right-click on Event Trace Sessions and picked New -> Data Collector Set
4. Name the set “WcfErrors”, use manual setup, and hit next
5. Next to Providers, click Add and find the provider "Microsoft-Windows-Application Server-Applications"
6. Under Properties select Level and hit Edit
7. Pick win:Error and hit OK
8. Click Next twice
9. Select "Open properties for this data collector set" and hit Finish
10. Go to the Trace Session tab
11. Change the Stream mode to Realtime
12. Hit OK
13. Start the WcfErrors session

With the session created you should be able to run the tool. When it finishes, PerfView will create a zip file and a summary file. If necessary PerfView will generate the NGEN PDBs and store them in a symbols folder in the zip file.

The only filter I apply above is on the level, which I set to Error. But you can be more specific by filtering to a certain channel and set of keywords. EventTraceWatcher unfortunately does not give you the whole event, just a dictionary of properties for the event. So if you wanted to filter in your code, then you would have to edit the EventTraceWatcher code to expose the rest of the event's properties.

 $changesets = get-tfsitemhistory . -recurse -Version "D09/01/2012~D12/31/2099"
foreach ($changeset in $changesets)
{
  "------------------------------------------------------------------------------------------------------"
  "Changeset: {0} " -f $changeset.ChangeSetId
  "Owner: {0}" -f $changeset.Owner
  "Date: {0}" -f $changeset.CreationDate
  ""
  "Comments: {0}" -f $changeset.Comment
  ""
  Get-TfsChangeset $changeset.changesetid | select changesetid -exp changes | format-table -auto
  "Work Items:"
  Get-TfsChangeset $changeset.changesetid | select changesetid -exp workitems | format-table id,title 
  ""
}
 

After upgrading to VS2012, the script stopped working and would only report the following errors:

Unable to determine the source control server.
Unable to determine the workspace.

The first error message was coming from the get-tfsitemhistory command. I have more than one TFS enlistment on my machine so I figured it could be that. The command has a -server parameter so I attempted to use that by calling get-tfsserver. Get-tfsserver works just fine and is able to find the server. But get-tfsitemhistory fails with the "unable to determine the workspace" error. I've seen many possible solutions online about re-creating the workspace and connecting visual studio to the workspace but none of those worked.

My problem turned out to be VS2012. When you install VS2012, you need to update your TFS Power Tools. Here are the steps I followed:

1. Open Add/Remove programs and uninstall Microsoft Team Foundation Server 2010 Power Tools
2. Go to: https://visualstudiogallery.msdn.microsoft.com/b1ef7eb2-e084-4cb8-9bc7-06c3bad9148f
3. Click on download and open the msi
4. Choose custom installation and make sure that powershell cmdlets are installed

Environment

This setup is pretty common. A set of load-balanced web servers is talking to a load-balanced WCF middle tier. The middle tier interacts with the data source.

In this case, there are two WCF services: one streamed and one buffered. Both services use the net.tcp transport with binary encoding.

The Problem

One of the symptoms that I dread the most when diagnosing WCF issues is the CommunicationException. It's a generic catch-all that gives no information as to what actually happened. You'll usually see something like this in the diagnostics logs.

The error message usually looks like this:

<Exception>
  <ExceptionType>System.ServiceModel.CommunicationException, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
  <Message>The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:00:59.7348017'.</Message>
  <StackTrace>...</StackTrace>
  <ExceptionString>
    System.ServiceModel.CommunicationException: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:00:59.7348017'. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
    at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
    at System.ServiceModel.Channels.SocketConnection.ReadCore(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout, Boolean closing)
    --- End of inner exception stack trace ---
  </ExceptionString>
  <InnerException>
    <ExceptionType>System.Net.Sockets.SocketException, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
    <Message>An existing connection was forcibly closed by the remote host</Message>
    <StackTrace>
      at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
      at System.ServiceModel.Channels.SocketConnection.ReadCore(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout, Boolean closing)
    </StackTrace>
    <ExceptionString>
      System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host
      at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
      at System.ServiceModel.Channels.SocketConnection.ReadCore(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout, Boolean closing)
    </ExceptionString>
    <NativeErrorCode>2746</NativeErrorCode>
  </InnerException>
</Exception>

This exception can happen for a lot of reasons. So it pays to dig around the log and look for other signs of trouble. There could be other kinds of exceptions buried in there or warnings about hitting throttling limits. In our case, there was another kind of exception mixed in that WCF eats by default.

You can see there is a sequence of events to this exception. After the exception is thrown, it is handled and a message is logged indicating that an attempt to reuse a pooled connection failed. The socket connection is aborted and then a retry is attempted. These exceptions are being eaten by WCF since you can see that after a couple attempts, the rest of the request continues. Note that this is coming from the client side. The exception itself looks like this:

<Exception>
  <ExceptionType>System.ServiceModel.ProtocolException, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
  <Message>The server at net.tcp://someserver/MyService.svc rejected the session-establishment request.</Message>
  <StackTrace>
  at System.ServiceModel.Channels.ConnectionUpgradeHelper.ValidatePreambleResponse(Byte[] buffer, Int32 count, ClientFramingDecoder decoder, Uri via)
  at System.ServiceModel.Channels.StreamedFramingRequestChannel.SendPreamble(IConnection connection, TimeoutHelper& timeoutHelper, ClientFramingDecoder decoder, SecurityMessageProperty& remoteSecurity)
  at System.ServiceModel.Channels.StreamedFramingRequestChannel.StreamedConnectionPoolHelper.AcceptPooledConnection(IConnection connection, TimeoutHelper& timeoutHelper)
  at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
  at System.ServiceModel.Channels.StreamedFramingRequestChannel.StreamedFramingRequest.SendRequest(Message message, TimeSpan timeout)
  at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
  at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
  at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
  at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
  at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
  ...
  at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
  at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
  at System.Web.HttpApplication.ApplicationStepManager.ResumeSteps(Exception error)
  at System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(HttpContext context, AsyncCallback cb, Object extraData)
  at System.Web.HttpRuntime.ProcessRequestInternal(HttpWorkerRequest wr)
  at System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr ecb, Int32 iWRType)
  </StackTrace>
  <ExceptionString>System.ServiceModel.ProtocolException: The server at net.tcp://someserver/MyService.svc rejected the session-establishment request. ---> System.ServiceModel.ProtocolException: Error while reading message framing format at position 0 of stream (state: ReadingUpgradeRecord) ---> System.IO.InvalidDataException: More data was expected, but EOF was reached.
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---</ExceptionString>
  <InnerException>
    <ExceptionType>System.ServiceModel.ProtocolException, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
    <Message>Error while reading message framing format at position 0 of stream (state: ReadingUpgradeRecord)</Message>
    <StackTrace>
    at System.ServiceModel.Channels.ConnectionUpgradeHelper.ValidatePreambleResponse(Byte[] buffer, Int32 count, ClientFramingDecoder decoder, Uri via)
    at System.ServiceModel.Channels.StreamedFramingRequestChannel.SendPreamble(IConnection connection, TimeoutHelper& timeoutHelper, ClientFramingDecoder decoder, SecurityMessageProperty& remoteSecurity)
    at System.ServiceModel.Channels.StreamedFramingRequestChannel.StreamedConnectionPoolHelper.AcceptPooledConnection(IConnection connection, TimeoutHelper& timeoutHelper)
    at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
    at System.ServiceModel.Channels.StreamedFramingRequestChannel.StreamedFramingRequest.SendRequest(Message message, TimeSpan timeout)
    at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
    at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
    at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
    at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
    at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
    ...
    at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
    at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
    at System.Web.HttpApplication.ApplicationStepManager.ResumeSteps(Exception error)
    at System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(HttpContext context, AsyncCallback cb, Object extraData)
    at System.Web.HttpRuntime.ProcessRequestInternal(HttpWorkerRequest wr)
    at System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr ecb, Int32 iWRType)
    </StackTrace>
    <ExceptionString>System.ServiceModel.ProtocolException: Error while reading message framing format at position 0 of stream (state: ReadingUpgradeRecord) ---> System.IO.InvalidDataException: More data was expected, but EOF was reached.
     --- End of inner exception stack trace ---</ExceptionString>
    <InnerException>
      <ExceptionType>System.IO.InvalidDataException, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
      <Message>More data was expected, but EOF was reached.</Message>
      <StackTrace>
      at System.ServiceModel.Channels.ConnectionUpgradeHelper.ValidatePreambleResponse(Byte[] buffer, Int32 count, ClientFramingDecoder decoder, Uri via)
      at System.ServiceModel.Channels.StreamedFramingRequestChannel.SendPreamble(IConnection connection, TimeoutHelper& timeoutHelper, ClientFramingDecoder decoder, SecurityMessageProperty& remoteSecurity)
      at System.ServiceModel.Channels.StreamedFramingRequestChannel.StreamedConnectionPoolHelper.AcceptPooledConnection(IConnection connection, TimeoutHelper& timeoutHelper)
      at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)
      at System.ServiceModel.Channels.StreamedFramingRequestChannel.StreamedFramingRequest.SendRequest(Message message, TimeSpan timeout)
      at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
      at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
      at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
      at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
      at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
      ...
      at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
      at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
      at System.Web.HttpApplication.ApplicationStepManager.ResumeSteps(Exception error)
      at System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(HttpContext context, AsyncCallback cb, Object extraData)
      at System.Web.HttpRuntime.ProcessRequestInternal(HttpWorkerRequest wr)
      at System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr ecb, Int32 iWRType)
      </StackTrace>
      <ExceptionString>System.IO.InvalidDataException: More data was expected, but EOF was reached.</ExceptionString>
    </InnerException>
  </InnerException>
</Exception>

ProtocolException

When sending a message over a connection, WCF will first establish the protocol by sending a few meaningful bytes. In this case, the WCF server has both a buffered service and a streamed service. These two transfer modes use different protocols. So if a connection was created by a buffered service call, added to the connection pool upon the buffered service call's completion, and then used by a streamed service call, that could explain the ProtocolException. We thought this could occur because the client side WCF connection pool for TCP is based on IP address. Since the two services were on the same server, we believed the clients were sharing the connection pool. To test our theory, we asked the client to set the connectionGroupName attribute in their client side configuration. Setting the group name allows you to isolate the connection pools. The client tried this but observed no change in behavior. The reason for this is streamed services don't use connection pools. Something else was causing the ProtocolException.

Creating a Repro

As your possible solutions become more difficult to test, the best thing to do is create a small reproduction of the issue. This can be tricky to get right. Most of the WCF repros I write take into account the following characteristics of the original service:

• Hosting - IIS, port sharing, self-host
• Binding - TCP, HTTP
• Types of services - quick responses, I/O intensive, CPU intensive
• Sync or async
• Transfer modes - buffered, streamed

In this case, there were both buffered and streamed, synchronous, I/O-bound (calling into a database) services using a TCP binding with IIS hosting (TCP and IIS implies port sharing).

This is the buffered service. The sleep is to simulate holding the thread while contacting the database. Usually database calls are quicker than this but sleeps like 1 or 2 seconds can help make problems easier to recreate and analyze.

And this is the streamed service:

Here is the configuration for the services. The custom binding used is equivalent to a nettcp binding. Using a custom binding will come in handy for trying different settings (such as the connection pool group name mentioned earlier).

On the client side I attempt to simulate load on the service by running many parallel requests:

The CorrelationManager.ActivityId is useful for grouping the events for a given request under unique ids. Otherwise when you look at the trace in svctraceviewer you'll see everything is grouped under the 00000 id.

The full solution is available at the bottom of the post.

When I tried running the code I initially had no luck reproducing the problem despite tweaking thread pool settings, the number of simultaneous requests, and the throttle settings on the services. So I tried a different approach and ran many client processes simultaneously using the following command line trick:

for /l %i in (1,1,5) do start TestClient.exe

The number 5 indicates that I want the for loop to execute 5 times. When I ran this I was able to reproduce the issue and get the exact same ProtocolException in the client side logs.

The Solution

With the repro it is now possible to tweak settings and quickly find out what's wrong. For instance, if you just run the buffered service by itself, the issue still repros meaning that it's not the combination of the buffered and streamed service. The real difference was that multiple client processes were talking to the same service. This is entirely possible when you're dealing with an environment where multiple web servers are calling into a WCF middle tier. The problem ended up being with the connection pool settings on the services and the fix is easy.

The service is keeping a pool of connections to the clients calling it. Also each client is keeping a pool of connections to the buffered service. By default, these two pools will be the same size. So one client and one server sending multiple simultaneous requests should work just fine. But when there are many clients, the service runs out of connections and starts closing down inactive ones. The client doesn't know the service closed the connection and only finds out when it gets the ProtocolException. That's why WCF eats the exception. The real problem was then that eventually all the connections would be active and after a client exhausted all the connections in the pool it would try to establish a new connection but the server would reject it, resulting in the CommunicationException. Bumping the maxOutboundConnectionsPerEndpoint fixed the issue.

ConnectionPool.zip

C:\Users\Dustin>powercfg -lastwake
Wake History Count - 1
Wake History [0]
  Wake Source Count - 1
  Wake Source [0]
    Type: Wake Timer
    Owner: [PROCESS] \Device\HarddiskVolume1\Windows\System32\services.exe
    Owner Supplied Reason: Windows will execute 'NT TASK\Microsoft\Windows\Media Center\mcupdate_scheduled' scheduled task that requested waking the computer.

So apparently I have a scheduled task called mcupdate_scheduled that woke my computer up. Next step, go to the start screen and type "scheduled" and you'll see an application called Scheduled Tasks. The path to the task is listed in the message above: Microsoft -> Windows -> Media Center. Navigate to that in the tree on the left and look for the mcupdate_scheduled task. If you look at the task and go to the Conditions tab you'll see that the "Wake the computer to run this task" checkbox is checked.

Seriously? Who in their infinite wisdom decided that Media Center update was so important that my computer needed to be woken up for it? There's even a KB article for it: https://support.microsoft.com/kb/979878

First, I think I should explain why I did this.

In performance analysis, the more information you can capture, the better. Profiles, process dumps, logs, etc. are all useful pieces of information essential to finding and fixing performance problems. In a typical application performance analysis happens at the same time as the other development work, which makes it very important to automate as much as possible. For instance, the infrastructure used to performance test WCF instruments for memory, throughput, and latency and captures profiles all on a daily basis. When a regression is detected, we can instantly pull up the logs and/or profiles and do a comparison to find out what happened. I wanted to be able to achieve the same thing in my new role working on Azure applications.

The WCF performance lab has a private network with dedicated network hardware. We can use a tool like psexec to run commands. There is a network share available to copy logs and other files. A typical test might go something like this:

1. Copy bits to server and client machines
2. Setup profilers and/or log collection agents on server(s)
3. Start services on the server(s)
4. Start client(s)
5. Wait for test execution to complete
6. Close profilers/log collection agents
7. Copy files from client and server machines to network share
8. Analyze results and upload to a database/website for reporting

Azure gives us a new challenge in several areas: copying bits on and off the machines, starting/stopping profilers or agents, and controlling clients. When we deploy the cspkg to Azure, that is how we get bits onto the server machines. The clients are specialized roles that will download test bits from blob storage and have communication ports open to accept commands for running tests. But on the server instances we need a way to get files on and off the machines as well as send commands without changing the code that was deployed in the cspkg. We want to test that code as it would be deployed in production.

The solution to this starts with a previous post on how to use Azure tables as a transport mechanism. This allows me to create a WCF service that does not need to open a port. The minimal impact I could think of was to add a startup task to the csdef and a few binaries to the cspkg that would start a Windows service on the Azure instance.

While I could create a WCF service that accepts specific commands like "iisreset" or "start test", it's not a very flexible solution. I chose to do PowerShell not only because it is very powerful but also because it allows for custom hosting.

In my code, I started with the sample linked above and made a few changes. The sample is meant to give you a custom shell that runs in a console window. It does a lot of things like change the foreground/background colors, get/set cursor positions, and gets the dimensions of the console window. All of that has to be adjusted. Also, there is an interface called IHostUISupportsMultipleChoiceSelection. If you were executing a command that asks for verification, such as for overwriting a file, the custom host can decide not to support that. In all of my Prompt methods I throw NotImplementedException. It makes the HostUserInterface class pretty simple:

As you can see, the only thing that's really implemented here is the Write method, which just writes to a StringBuilder. I use this to capture all the output from a command. When a command is done executing, I empty the contents of the StringBuilder and pass it back in the response.

PSListenerConsoleSample is pretty much verbatim copied from the example. It starts the UI. In my case, I don't call the Run method, only the Execute method. The WCF service inherits from PSListenerConsoleSample:

The DumpOutput method just gets the current string from the StringBuilder and then empties it. SendCommand takes the string in the request, executes it, and then returns the output as a response.

One interesting thing I noticed is that in the example, the output of a command will typically go to the console window that is running the PowerShell host. Since mine is running as a Windows service I thought I would have to find a way to redirect that output. Much to my happiness, PowerShell's code is already redirecting it and I didn't have to do anything.

The RawUserInterface is the last thing to make changes to. You have to replace the data it's trying to get from the Console. Here is my version:

The buffer size I chose is 1000 wide by 9999 high. If you connect to this PowerShell with a console client, be sure to change the buffer size on the console to at least the same width.

The rest of the code should be pretty easy to understand if you've been following the posts. There is a console client and server and a Windows service. In subsequent posts, I'll cover how to add this to an Azure role and how to get profiles.

The code is available here: https://github.com/dmetzgar/azure-remote-powershell

The full code is available here: https://github.com/dmetzgar/azure-table-transport

The genesis of this transport channel came about while I was trying to find a suitable way to communicate with individual Azure instances that did not require an open port and did not go through the load balancer. There are a number of approaches that could work but the technique of communicating through Azure tables was something another group in Microsoft was doing and had success with.

From the service side, here is the design:

• The service runs on an Azure role instance
• Deployment id, role name, and instance name are acquired from RoleEnvironment and concatenated to create a partition key for the Azure table
• The service polls the Azure table with the partition key looking for messages
• When a message is received, it is executed and the response is sent to another Azure table

The client works in a similar way where it writes a request message to an Azure table and polls another table for a response. The difference is that it has to be pointed to a particular Azure role instance.

Turning this into a WCF transport channel seems a piece of cake when modifying the existing custom transport channel code that uses the file system.

The code below is using Azure SDK 1.8. The first thing I created was a TableEntity to store the SOAP message. This works for both the request and response.

The URI for the service endpoint should serve as the name of the Azure table. Since I have two tables, I append Request and Reply to them. The ChannelListener seems a good place to create the tables if they do not already exist.

The lynchpin of the whole system is the code to poll for messages. As with the file transport channel, both the request and reply channel classes inherit from a common base class. This is where I've added the WaitForMessage method.

The partitionKey is stored as a member variable. You can see where it's applied as a filter condition when querying the table. The sleep value is passed in to the method. You could make this configurable if you want.

Writing a message to a table is the same for both request and reply.

The code here uses the encoder (which is set to the text encoder) to write the message to a buffer created by the buffer manager. Then it's encoded as a UTF8 string. If you were to open the table in a tool like Azure Storage Explorer you would see the text SOAP message. This can help for debugging purposes.

Reading the message is split into two methods. This is because there are two code paths for getting the message. In the reply channel, WaitForRequest only returns a Boolean indicating whether or not a request was received within the timeout. The other path is TryReceiveRequest in the reply channel and the Request method in the request channel that waits for the reply message. This path will use the message read from the WaitForMessage method instead of reading again from the table.

Notice also that the table entity is deleted in the finally. This is to keep us from reading the same messages over and over. You could modify this to update a flag indicating the message was read so you could keep an archive of the messages.

The rest of the code is pretty similar to the file transport channel. One improvement is that there are several properties added to the BindingConfigurationElement that are used in the app.config. That code is fairly straightforward as well.

To test this you should have the Azure SDK 1.8 installed. Start up the Storage Emulator first. Then set multiple startup projects on the solution and start the service above the client. It should work the exact same way as the file transport channel did in previous posts.

Click here to see all the posts in this series. All the code is now available on github here: https://github.com/dmetzgar/custom-transport-channel

Adding a configuration element is relatively simple. The file transport is easier to work with because it has no configuration options. Handling custom configuration is handled in a later post. Add a new file to the library project called FileTransportBindingConfigurationElement, this will hold two classes: the configuration element and the collection element. The collection element is just one line, so I prefer to just put it in the same class file an the configuration element.

Notice that this really doesn't do much except identify the FileTransportBinding as the BindingElementType.

The next step is to update the FileTransportBinding to read from a configuration section. To do this, we add another constructor that takes a string and reads the FileTransportBindingCollectionElement out of the configuration section.

Starting from the service side, add an app.config and put in the following serviceModel section.

The bindingExtensions collection is what allows WCF to load the custom binding element. Since there are no configuration options, the binding just needs a name and to be assigned to an endpoint. The address is specified on the endpoint so it no longer needs to be in the main method. That simplifies the code for Program.cs down to this:

On the client side, we have to add another constructor to the ReverseClient proxy that can read from config.

Then add an app.config to the client with the following.

The same bindingExtension is added here on the client side. We do have some timeouts we can configure that are universal. This simplifies the main method of the client down to this:

And that's it!