Running tests in Team Build - 1

Why would I want to run tests as part of build process?

During the building of any given product, you may want to run tests after each build in order to scout the basic quality of the new build. Or maybe developers writing different parts of the product would like to run unit tests against their components after each build so that component wide regressions are prevented. Testers can check in their aggregated test automation code into the team project and run all tests each time after compilation so that product wise regressions are caught during build process itself.

How do I write tests?

With the Visual Studio Team Edition for Testers, you can author test projects with different kinds of tests. These test projects can be written in C#, VB.NET, or C++. Each test project can contain unit tests, generic tests, load tests, ordered tests, ASP.NET unit tests or Web tests. Consider a company using Team Foundation Build has a Web site to maintain. The source code consists of an ASP.NET Web site, a Web service that connects to an external source and a couple of class libraries where the logic of the entire product is stored.

• Unit tests are primarily used to write developer regression tests, or DRTs, that run on each component’s code. These verify basic functionality of the component and prevent regressions due to new code. In the above instance, it can be a test to check if a given function in the library is returning the correct output given a set of known input values.
• Generic tests are used to wrap existing test code or test scripts so that any pre-existing code can still be used in the test framework within Visual Studio. These tests pass or fail based on the error code returned. In the instance above, it is a wrapper over a batch file that a user had already built in order to check for some particular environment variable set.
• ASP.NET unit tests can be used to run unit tests against the code residing in the AppCode folder. These are just like normal unit tests except that they run on code in the AppCode folder in Web sites or services. In the instance above, these tests can be written to check if a given function in a class in the AppCode folder is returning the correct output given a set of known input values.
• Web tests are used to check functionality of already deployed Web sites by navigating to the Web site directly. In the illustration above, the Web site can be deployed using Web development server or IIS and then Web tests can be run on the deployed site to test if links in the site navigate to the expected location.
• Load tests are used to stress the given product and can be customized to apply load on the system. In the instance above, a user can have a load test simulating 100 people accessing the Web site at once and checking performance of the test. Usually these will not be run as part of build process.
• Ordered tests are a sequence of any of the above tests arranged in an order. These tests will execute the specified number of times as well as in the order mentioned. In the instance above, a user might just add an ordered test to run a couple of library unit tests and an ASP.NET unit test for the Web service that consumes the libraries and stipulate that the ASP.NET unit test need not run in case of failure in any of the other unit tests.

A user creates the desired tests and adds them to appropriate test lists. A test list is an aggregate of tests that can be defined by the user. Note that this test project is part of the Visual Studio solution that contains the projects that this test project is testing. The corresponding solution items – the vsmdi file which is the test metadata file and the run configuration file(s) that contains the test run settings, are all also checked in as part of the solution.

How do I configure my build to run tests each time?

A build type can be configured to run tests after compilation each time a build is kicked off using that build type. In the Options page of the build type wizard:

1. Check the "Run Test" check box.
2. In the "Test metadata file" drop-down that is enabled , choose the appropriate vsmdi file from the solution you are building.
3. In the "Test list to run" tree structure panel, check the test lists that you would like to run along with the build.

Now, your build type is ready to run tests when builds are kicked off with this build type.

Okay – I ran my tests – now how do I see results?

Once a build is kicked off with tests enabled, the test lists specified are run as part of the build process. In order to view the test results, open the build report and expand the test results tab.

Here, you can see a link to the test results file, the number of tests that have passed, failed, and have a different result. Tests that have a different result other than passed or failed might be load tests that have a status of "Completed" if they ran successfully or tests that might have been "Inconclusive" due to insufficient data being obtained while running the test. When you click on the hyperlink pointing to the test results file, a dialog box opens, prompting you to choose a location where the test results can be downloaded. Upon specifying a location, the test results are downloaded at that location. This is a .trx file and a folder that contains the deployed items and other test binaries. After download, the test results are displayed in the Results window in the Visual Studio client.

Clicking the test run hyperlink takes you to a tabbed window containing the test run details. Double-clicking each of the tests opens tabbed windows each containing the details of that particular test.

We will look at more detailed scenarios around running tests in Team Build in the next part. Meanwhile, please feel free to send any comments/feedback...