Framework for intergrated FxCop rule testing

Roy Osherove has put together a Framework for testing FxCop rules. It's an interesting approach, firing up the FxCop engine for each rule that he wants to test.

There are few things that I would like to see it support:

1. The ability to specify positive tests (code that rules should not fire when run over) on top of the negative (code that rules do fire when run over) tests that he supports
2. The ability to separate the actual test from the methods you want to run your rules over. Unfortunately, this approach will not scale when you are testing 3 difference languages (C#, Visual Basic and C++) for 200+ rules that we ship.

Unfortunately, his approach is not going to work in the next version of FxCop/Visual Studio for a couple of reasons. 1) We did some work in Orcas to reduce the visibility of API's that did not need to be public for actually writing custom rules (for example, everything in FxCopCommon.dll is now internal) and 2) We also removed the Reflection 'bridge', that is, the ability to create CCI objects from their Reflection counterparts and vice versa.

Apart from the things mentioned above, it's a great start and its got us thinking, how many users actually automate the testing of custom rules? Is this something you see as important? Would you like to see how we do this internally?

Comments

• Anonymous
  February 25, 2007
  If you don't know what FxCop is, go look . You'll like it. In response to my release of FxCopUnit which

• Anonymous
  February 25, 2007
  David, I think this is a big mistake - to remove any shred of testability still left in the product. See my full thoughts on this post  here: http://weblogs.asp.net/rosherove/archive/2007/02/25/fxcop-is-heading-in-the-wrong-direction-no-testability-whatsoever.aspx

• Anonymous
  February 25, 2007
  The typical user of FxCop/CA, I would expect, is one that utilizes things like unit testing frameworks.  I would think someone going to all the trouble of writing FxCop rules would also be very interested in unit testing those rules.

• Anonymous
  February 26, 2007
  David,  I think you have your question backwards.  You should not be asking how many user currently write tests, but rather why aren't more people writing tests.  When you have a framework specifically designed for writing plug-ins, and yet a vibrant community of plug-ins does not seem to materialize, you should probably be asking yourself what you are doing wrong that is hindering creation of custom rules by more developers.  You should be asking your users how can you make custom rule creation easier and more robust.

• Anonymous
  February 26, 2007
  I also automate rule testing, but I do so by analyzing FxCop reports (including both positive and negative tests).  It's an approach that should be reasonably resitant to changes in FxCop itself since, as long as I can transform an FxCop report via XSLT, I can generate the test input data in whatever form I like, but I'll probably wait until at least beta 2 before I start counting any chickens... ;)

• Anonymous
  February 27, 2007
  Roy Osherove has noted a discrepancy between pure object-oriented design and testable design principles,