Sneak Preview: Code Digger — The New Pex Experience

NEW: Code Digger now available for Visual Studio 2012

Download it now from the Visual Studio Gallery. Read my latest blog post on how to use Code Digger for Visual Studio 2012.

Note: Code Digger for Visual Studio 2012 is somewhat different from Code Digger for VS2008 and VS2010 that is described below. In particular, you can't save tests.

About Code Digger for VS2008 and VS2010

Do you control your code?

Did you ever write some new code, change some old code, or simply look at existing code, and you were not quite sure what it was capable of? Which inputs would trigger a successful "happy" path, and which ones would be rejected? Maybe you were thinking: "If only I had a test suite..."

Well, be our guest, or rather Pex' guest: You can have that test suite! (And you won't have to sweat for it.)

Update: You can now download Code Digger as part of the latest Pex release. And just as a reminder: Register for PDC, go to our talk and watch the latest demo at the Microsoft Research booth !

If you want to read more, we have just completed the full Code Digger tutorial. It already refers to VS2010, but the Microsoft Research download also works with VS2008.

Code Digging with Pex

Ok, back to your code. Let's say you are looking at a method that capitalizes words in a string. With horror, you’ve just realized that you totally forgot to write tests for it. In fact, you even don't remember why the method has so many if-statements that treat special cases. It seems that the situations is hopeless, but wait... All you have to do is right-click on your code in the Visual Studio editor, and select "Run Pex Explorations"!

That will trigger a deep code analysis using the Pex engine.

And let us repeat the main point: You start the analysis from the code, not from an already existing Unit Test, or even Parameterized Unit Test! No test suite needed. No test project needed. The only reason we are using the word "test" so often because we want to stress that you don't need any.

So, what do we get? A table with inputs and outputs that cover all the successful and failing corner cases. To this end, Pex analyzed all the branches in the code to figure out what interested values are. The code we were "digging" here only distinguished lower-case characters, upper-case characters, and punctuation characters. (This is the same kind of table that Pex produces when you analyze Parameterized Unit Tests.)

Okay, that's nice. You can look at all the cases and check whether they make sense to you. You can use all the already existing Pex features. For example, with two mouse clicks, you could let Pex fix your code to throw an ArgumentNullException instead of the NullReferenceException.

But there is more. Remember, we promised you a test suite...

Code Digging for Gold: Test Suites for Free!

You can select one, several, or simply all rows, and click on the "Save..." button.

Pex will create an entire test project, and generate C# code for all selected rows as unit test.

Integration with Visual Studio Team Test

And what's best: The generated tests integrate perfectly with the Visual Studio Team Test unit test framework, showing up in Visual Studio's Test View window.

Now you can leverage all unit testing features that come with Visual Studio. For example, you can let Visual Studio collect code coverage information. Or, if you wonder how a line in the code can be triggered, you can set a breakpoint in that line, and run the test suite under the debugger. It's that easy. With the Pex Code Digger, you don't have to write a test driver yourself!

Regression Test Suite

You might wonder: Besides debugging issues right away, what is this generated test suite actually good for? We thought about that too. Each of the generated tests contains a line that asserts the behavior that Pex observed today, for example:

  1: [TestMethod]

  2: [PexGeneratedBy(typeof(StringExtensionsTest))]

  3: public void Capitalize11()

  4: {

  5:     string s = this.Capitalize("p:p");

  6:     Assert.AreEqual<string>("P_P", s);

  7: }

After saving today's tests, when you ever change the code again in the future, you will have an excellent regression test suite that will identify the breaking changes.

Does this replace traditional testing?

Not really. With Code Digger, you have to manually inspect each generated test case to check whether your code does what you want it do to. In contrast, Unit Tests, or even better, Parameterized Unit Test, are an excellent way to describe programmatically what your code is supposed to do. Code Digger and (Parameterized) Unit Testing are complementary, and one doesn't make the other obsolete.

The end

That's all for this post; keep watching for further announcements, or read the full Code Digger tutorial.

Comments

• Anonymous
  October 21, 2008
  PingBack from http://blog.a-foton.ru/index.php/2008/10/21/sneak-preview-code-digger-%e2%80%94-the-new-pex-experience/

• Anonymous
  October 21, 2008
   Integration with Visual Studio Team Suite Test Edition seems a bit of an odd feature when the licence for Pex prohibits "using the Software in the creation or use of commercial products" - are there many people who have Team Suite for non-commercial use?  Is this a sign that Pex is going to become a Team Suite feature, or that the licence agreement is going to be relaxed a bit?

• Anonymous
  October 21, 2008
  You are doing an amazing job! I'm looking forward to the next update of Pex :-) Regards, Daniel Lidström, Stockholm, Sweden

• Anonymous
  October 22, 2008
  We just released the most exciting iteration of Pex to this date: version 0.8. The Microsoft Research

• Anonymous
  October 22, 2008
  Hi Marklam, the Pex team's goal is indeed to make Pex available under a different license eventually. Since you asked: There are indeed some non-commercial projects going on at universities, where researchers study based on Pex what the limits of dynamic test generation are. In any case, at this time the Pex team is looking for feedback to improve Pex.

• Anonymous
  October 22, 2008
  Hi Nikolai, It's good to hear that the licence may not always prevent use by commercial developers. I didn't mean to suggest that there wasn't any non-commercial use for Pex. I just thought that for non-commercial use, integration with Team Suite was a less obvious choice than with an open-source test framework.