Pex 0.13 Released: Unit Tests as Inputs, Fuzzing and Visual Studio 2010 Beta 1, F# Beta 1 support

We just released an update of Pex that brings a very cool new feature, Unit Tests as Inputs and updates for Visual Studio 2010 Beta 1, Code Contracts and F# Beta 1. As usual, this release also contains a number of bug fixes and small feature improvements, most of which address your suggestions from the Pex forum.

Seeding the Exploration Part 1: Unit Tests as Inputs

Before using heavy-weight constraint solving to explore hard-to-reach execution paths, Pex can now leverage already existing unit tests that call a parameterized unit tests: Pex scans their body to extract the parameter values, and then Pex uses these values to seed the exploration. (In the past, Pex would have seeded the exploration by simply using the default values for all parameters, and nothing else.)

Here is an example, where Pex starts the exploration of the parameterized unit test Multiply by reusing the values mentioned in the unit test MultiplySeed, as you can see in the first row of the table of generated tests.

As a (intended) side effect, Pex also reuses the tests that Pex might have generated during earlier explorations. In effect, Pex never has to start from scratch again, but instead can reuse knowledge from earlier explorations.

There is a big limitation in this first version of the new feature: Pex can only use values of primitive types, and arrays of primitive types as seeds, and the values must be passed as parameters, not via PexChoose. The unit tests must be in the same class as the parameterized unit test, marked as a unit test, and have a simple structure, i.e., they must just create the input data, and the first method call must be to a parameterized unit test.

Seeding the Exploration Part 2: [PexArgument(…)] to Fuzz Inputs and Help Pex

Pex can not only extract values from existing unit tests, but there is in fact a general API through which seed values can be given. In particular, you can use the new PexArgumentAttribute (and a bunch of related new attributes) to provide seed values.

Here is an example:

During the subsequent exploration, Pex will often reuse parts of the seeded values, which may make constraint solving easier for Pex. Also, when Pex cannot solve all constraints to leap beyond a difficult branch, then you can use this feature to help Pex by providing the values yourself.

Object Creation with Invariants

Code Contracts enable the specification of class invariants by writing an “invariant method” decorated with the ContractInvariantMethod attribute. For example, for a typical ArrayList implementation, this invariant method might read as follows:

     public class ArrayList {
        private Object[] _items;
        private int _size;
        ...
        [ContractInvariantMethod] // attribute comes with Code Contracts, not Pex
        protected void Invariant() {
            Contract.Invariant(this._items != null);
            Contract.Invariant(this._size >= 0);
            Contract.Invariant(this._items.Length >= this._size);
        }

If you have such an invariant, and if you correctly configure runtime checking of contracts, then Pex will create instances of this type by first creating a raw instance, setting all private fields directly, and then making sure that the invariant holds. A generated test case for the Add method of the ArrayList class might then read as follows:

 [TestMethod]
public void Add01() {
    object[] os = new object[0];
    ArrayList arrayList = PexInvariant.CreateInstance<ArrayList>();   // creates raw instance
    PexInvariant.SetField<object[]>(arrayList, "_items", os); // sets private field via reflection
    PexInvariant.SetField<int>(arrayList, "_size", 0);        // same
    PexInvariant.CheckInvariant(arrayList);                   // invokes invariant method via reflection
    arrayList.Add(null);                                      // call to method-under-test
}

A limitation of the current implementation is that the type itself must be public, and the types of all fields must be public (but not the fields themselves.)

Surviving unhandled exceptions in Finalizers

Finalizers are especially problematic for test runners: there is no guarantee from the runtime when they will be executed, and when they throw an exception, the process terminates. This is not an uncommon scenario when you apply Pex to (buggy) code with Finalizers: It seems as if Pex suddenly crashes.

In this version, we have improved the robustness of Pex with respect to those failures. Pex effectively rewrites all finalizers of instrumented types with a catch handler that swallows all exceptions and sends it to Pex’ logging infrastructure.

Visual Studio 2010 Beta 1 support

We’ve updated Pex for Visual Studio 2010 Beta 1, the latest Code Contracts version and F# Beta 1.

Send Feedback at your fingertips

We made it even easier for you to ask questions about Pex, or to just tell us what you think, right from Visual Studio:

Bug fixes and other small features

• The Stubs framework generates stubs for types containing obsolete members. This behavior can be disabled.
• The wizard can generate tests for inherited members by adding the /gim flag or from the wizard settings/options.
• The default test assembly name suffix string, i.e., the suffix used to generate the test assembly name, can be specified on the wizard command line, e.g. /tans:.PexTests, or in the Visual Studio options (Tools –> Options –> Pex –> Wizard –> Test Assembly Name Format String).
• int ChooseIndexValue<T>(string, T[]) was added to the choice API to choose valid index values.
• We added new overloads in PexAssume/PexAssert that support IEnumerable<T>, i.e. for AreElementsEqual etc…
• Better understanding of _.GetType(), and conditions involving System.Type values.
• The Stubs framework can assign values to ref parameters.