Performance Counter Testing

If at all applicable, a well-written application should include one or more performance counters, which will enable operations staff (or maintainance developers) to monitor and troubleshoot the application. Since the goal of a performance counter is to provide an accurate snapshot of what's going on inside the application, it is essential that you have a suite of automated tests to ensure that those metrics are correct.

Whether you will call such tests integration tests or unit tests is mostly of semantic importance - personally, I think this would constitute integration tests, since you are testing how your application integrates with the operating system.

The thing about using performance counters is that you need to define them in Windows before you can use them. According to my integration testing principles, that should be part of the automatic setup for the test suite. While you could definitely use some custom code utilizing PerformanceCounterCategory.Create, this would entail a doubling of effort in terms of creating and maintaining the definition of your counters. If you change the definition of a counter in your application, you would also need to change the definition of the counter in the test suite.

For security reasons, you should create the performance counter category as part of the application installation, since this will allow the application to run with least privilege (creating performance counters requires elevated privileges). The easiest way to do this is to configure a PerformanceCounterInstaller within a custom application Installer. This would look something like this:

 [RunInstaller(true)]

 public class MyApplicationInstaller : Installer

 {

     private PerformanceCounterInstaller counterInstaller_;

     public MyApplicationInstaller()

     {

         this.counterInstaller_ = 

             new PerformanceCounterInstaller();

         this.counterInstaller_.CategoryName = "MyCategory";

         this.counterInstaller_.CategoryType =

             PerformanceCounterCategoryType.SingleInstance;

         CounterCreationData ccd = new CounterCreationData();

         ccd.CounterName = "MyCounter";

         ccd.CounterType = PerformanceCounterType.CounterDelta32;

         this.counterInstaller_.Counters.Add(ccd);

         this.Installers.Add(this.counterInstaller_);

     }

     public PerformanceCounterInstaller PerformanceCounterInstaller

     {

         get { return this.counterInstaller_; }

     }

 }

When you include such a class in your application, you can easily install the application using InstallUtil.exe. In this example, I've added the extra feature of exposing the PerformanceCounterInstaller as a publich property. This isn't at all necessary from a pure application installation point of view, but allows me to reuse the performance counter definition when setting up an integration test suite. From an API design perspective, this is quite acceptable, since the PerformanceCounterInstaller instance is already publicly available via the Installer's Installers collection, so the public property is only a convenience, and it's now easy to reuse the performance counter definition to automate the initialization of a test suite:

 [ClassInitialize]

 public static void InitializeClass(TestContext context)

 {

     MyApplicationInstaller appInstaller =

         new MyApplicationInstaller();

     PerformanceCounterInstaller counterInstaller =

         appInstaller.PerformanceCounterInstaller;

     TransactedInstaller ti = new TransactedInstaller();

     ti.Installers.Add(counterInstaller);

     ti.Install(new Hashtable());

 }

While not strictly necessary, I've chosen to wrap the PerformanceCounterInstaller within a TransactedInstaller. As an alternative, I could also have called Install followed by Commit directly on the counterInstaller instance, but then I would have needed (in theory at least) to test whether the Install phase succeeded before deciding on either a Commit or Rollback. Using a TransactedInstaller instance, this happens automatically.

You should be aware that installing performance counters takes a lot of time. On my machine it takes about one and a half minute, which means that setting up the test suite is going to take this long. That's not particularly conducive to TDD, so in its pure form, performance counter testing may fit more naturally into the category of Build Verification Tests.

On the other hand, you could employ some workarounds, like testing whether the performance counter category already exists and not install it if that's the case. You could then modify the test suite's clean-up method so that it doesn't uninstall the category. This would mean that you'd only need to wait for the performance counters to be installed during the first run of your test suite, and subsequent runs would be much faster. You just have to remember to re-enable the clean-up method when you are done working with the tests. I'll leave the specifics on this as an exercise for the interested reader.

Cleaning up the test suite is quite similar to initialization:

 [ClassCleanup]

 public static void CleanupClass()

 {

     MyApplicationInstaller appInstaller =

         new MyApplicationInstaller();

     PerformanceCounterInstaller counterInstaller =

         appInstaller.PerformanceCounterInstaller;

     TransactedInstaller ti = new TransactedInstaller();

     ti.Installers.Add(counterInstaller);

     ti.Uninstall(null);

 }

The only real difference is that if you can't provide an instance of IDictionary representing the saved installation state, you need to pass null to the Uninstall method, which is quite all right in this case. Contrary to installation, uninstallation is done in a matter of seconds.

While setting up the performance counters as a prelude to the actual testing is a necessary step, it obviously doesn't accomplish anything in itself. You still need to test that the counters are correctly updated by your application.

Imagine that you want to test this simple class:

 public class MyClass

 {

     private readonly static PerformanceCounter myCounter_ =

         new PerformanceCounter("MyCategory", "MyCounter", false);

     public MyClass() { }

     public void Act()

     {

         // Do something important here...

         MyClass.myCounter_.Increment();

     }

 }

MyClass uses the performance counter defined in the MyApplicationInstaller class. Since the performance counter in question represents the difference between two samples of the counter (CounterDelta32) it requires two samples to yield a result. Here's one way to test that the Act method increments the counter:

 [TestMethod]

 public void CountUsingCalculatedValue()

 {

     PerformanceCounter actionCounter =

         new PerformanceCounter("MyCategory", "MyCounter", true);

     actionCounter.NextValue();

     MyClass mc = new MyClass();

     mc.Act();

     float finalCounter = actionCounter.NextValue();

     Assert.AreEqual<float>(1, finalCounter);

 }

If you look closely at this test code, you may wonder about the first call to actionCounter.NextValue. Since the result isn't assigned to any variable, this method call seems redundant. However, if you remove it, the test will fail, since the value of finalCounter will then be zero. This is because the first sample provides the basis for calculating the value when the next sample occurs (for CounterDelta32, the first value is simply subtracted from the second), so by definition, the calculated value of the first sample is zero.

Since the above unit test doesn't clearly communicate its intent, I prefer a more explicit alternative:

 [TestMethod]

 public void CountSingleAction()

 {

     PerformanceCounter actionCounter = 

         new PerformanceCounter("MyCategory", "MyCounter", true);

     CounterSample initialSample = actionCounter.NextSample();

     MyClass mc = new MyClass();

     mc.Act();

     CounterSample finalSample = actionCounter.NextSample();

     float counterResult = 

         CounterSampleCalculator.ComputeCounterValue(

         initialSample, finalSample);

     Assert.AreEqual<float>(1, counterResult);

 }

With this approach, I use CounterSampleCalculator to calculate the result. Since I'm using the overload of ComputeCounterValue that takes two instances of CounterSample, it's a bit more apparent what's going on, and why I'm taking the initial sample.

As you can see, testing performance counters is pretty simple, and to a large extent, you can reuse the installation logic that you ought to have in place regardless.

Comments

• Anonymous
  September 16, 2007
  Great post. Just started looking into custom performance counters. Never thought of putting performance counters into tests but this is of cause a good idea. Thanks

• Anonymous
  September 16, 2007
  Hi Anders Once you start doing TDD, you soon start wanting to put everything in test :)

• Anonymous
  November 28, 2007
  It's no secret that I prefer unit tests over integration tests. Whenever it's possible to replace an

• Anonymous
  November 28, 2007
  It's no secret that I prefer unit tests over integration tests. Whenever it's possible to replace

• Anonymous
  March 15, 2010
  This was a big help for architecting the install and testing of my custom performance counters.  This sample should be part of the MSDN docs on performance counters.

• Anonymous
  December 03, 2013
  Great help.