Testing Concepts and Phases

This section provides a conceptual overview of the most common approaches to code testing. It introduces some of the key terminology and identifies the scenarios in which each type of test may be appropriate.

Unit Testing

Unit tests are automated procedures that verify whether an isolated piece of code behaves as expected in response to a specific input. Unit tests are usually created by developers and are typically written against public methods and interfaces. Each unit test should focus on testing a single aspect of the code under test; therefore, it should generally not contain any branching logic. In test-driven development scenarios, developers create unit tests before they code a particular method. The developer can run the unit tests repeatedly as they add code to the method. The developer's task is complete when their code passes all of its unit tests.

A unit test isolates the code under test from all external dependencies, such as external APIs, systems, and services. There are various patterns and tools you can use to ensure that your classes and methods can be isolated in this way—these are discussed later in this section.

Unit tests should verify that the code under test responds as expected to both normal and exceptional conditions. Unit tests can also provide a way to test responses to error conditions that are hard to generate on demand in real systems, such as hardware failures and out-of-memory exceptions. Because unit tests are isolated from external dependencies, they run very quickly—it is typical for a large suite consisting of hundreds of unit tests to run in a matter of seconds. The speed of execution is critical when you are using an iterative approach to development, because the developer should run the test suite on a regular basis during the development process.

Unit tests make it easier to exercise all code paths in branching logic. They do this by simulating conditions that are difficult to produce on real systems in order to drive all paths through the code. This leads to fewer production bugs, which are often costly to the business in terms of the resulting downtime, instability, and the effort required to create, test, and apply production patches.

Integration Testing

While unit tests verify the functionality of a piece of code in isolation, integration tests verify the functionality of a piece of code against a target system or platform. Just like unit tests, integration tests are automated procedures that run within a testing framework. Although comprehensive unit testing verifies that your code behaves as expected in isolation, you still need to ensure that your code behaves as expected in its target environment, and that the external systems on which your code depends behave as anticipated. That is where integration testing comes in.

Unlike a unit test, an integration test executes all code in the call path for each method under test—regardless of whether that code is within the class you are testing or is part of an external API. Because of this, it takes much longer to set up the test conditions for an integration test. For example, you may need to create users and groups or add lists and list items. Integration tests also take considerably longer to run. However, unlike unit tests, integration tests do not rely on assumptions about the behavior of external systems and services. As a result, integration tests may detect bugs that are missed by unit tests.

Developers often use integration tests to verify that external dependencies, such as Web services, behave as expected, or to test code with a heavy reliance on external dependencies that cannot be factored out. Testers often also develop and use integration tests for more diverse scenarios, such as security testing and stress testing.

In many cases, organizations do not distinguish between integration and unit testing, because both types of tests are typically driven by unit testing frameworks such as nUnit, xUnit, and Visual Studio Unit Test. Typically, organizations that use agile development practices make this distinction, because the two types of tests have different purposes within the agile process.

Note

In the Visual Studio 2010 release, there is a limitation that prevents you from testing a SharePoint assembly using Visual Studio Unit Test. Unit tests created for Visual Studio Unit Test must be developed using .NET Framework 4.0 in Visual Studio 2010; whereas, SharePoint 2010 assemblies are based on .NET Framework 3.5. In many cases, this is not an issue—because, generally, .NET Framework 4.0 assemblies are compatible with .NET Framework 3.5 assemblies, so you can run a .NET Framework 4.0 test against a .NET Framework 3.5 assembly. However, the way in which SharePoint loads the .NET common language runtime (CLR) prevents the runtime from properly loading and running the tests within Visual Studio Unit Test.
This limitation prevents you from running integration tests with SharePoint within Visual Studio Unit Test. Integration tests execute real SharePoint API logic instead of substituting the logic with a test implementation. Two isolation tools discussed in the following sections, TypeMock and Moles, will continue to work because they intercept calls to the SharePoint API before the actual SharePoint logic is invoked. You can execute integration tests using a third-party framework such as xUnit or nUnit. Coded user interface (UI) tests against SharePoint applications will run without any issues from within Visual Studio 2010.

Continuous Integration Testing

Continuous integration (CI) is a process that provides a continual verification of code as it is checked into the source repository. This process ensures that the quality of checked in code is always high, because developers do not want to be responsible for breaking the team build. It also ensures that any problems are quickly identified and addressed—in many agile teams, development stops if the CI server is "red" until the issue is resolved.

Typically, development teams run CI in response to a check-in event when code is added or changed, although it may also run periodically at a regular interval such as every couple of hours. The CI process builds the code and runs all of the unit tests. The CI process can also run additional checks, such as static analysis. For example, when you work with SharePoint solutions, a recommended practice is to run the SPDisposeCheck utility to check for leaky disposal of SharePoint objects.

Typically, CI servers use a commercial tool, such as Team Foundation Build, or an open source tool, such as Cruise Control, to help automate the build and test process. These tools simplify the setup and execution of the CI process and provide reporting on build and test results. For information on setting up Team Foundation Build for CI with SharePoint 2010 see How to Build SharePoint Projects with TFS Team Build.

Web Testing

Web testing simulates the interaction between a user and a Web-based user interface. The Web test sends HTTP requests to your solution and verifies that the HTTP response it receives is as you expect. Even with sophisticated tools, writing a robust, repeatable Web test can be challenging and time consuming for complex user interfaces. Within Visual Studio, Web tests are known as coded UI tests.

Stress Testing

Stress tests run an isolated component under excessive load conditions. The purpose of a stress test is to drive the component beyond its normal operating conditions to ensure that it degrades gracefully. Usually, you will use integration tests to conduct stress testing, although you can also use coded UI tests. Stress tests are a useful way to detect certain classes of problems, including memory leaks due to improper disposal and threading-related issues, such as deadlocks or resource contention. When you conduct stress testing, you need to make sure that you stay within the limits of the underlying hardware and operating system, because, inevitably, failures will arise as you exceed the capacity of the infrastructure.

Functional Testing

Functional testing refers to any procedure that tests the functionality of an application from the perspective of a user. Functional tests can include manual tests, Web tests, and integration tests. Integration tests are included in functional testing because systems often expose APIs for extensibility or for programmatic use. In this case, the target user is a developer.

Build Verification Testing

Build verification tests (BVTs) work in a similar way to continuous integration, and typically use the same tools. However, while continuous integration ensures that code builds successfully and passes unit tests, BVTs are used to determine whether code satisfies a representative subset of the functionality expected by end users. Typically, BVTs use a combination of integration tests and coded UI tests. A BVT process builds, installs, deploys, and tests an application on a regular basis. BVTs must often perform extensive scripted configuration of the deployment environment before running intensive test processes; because of this, they can take tens of minutes to complete.

BVTs provide a baseline measure of confidence in the quality of a build against a real system before it is deployed more widely into other testing environments. BVTs should be conducted in addition to rather than instead of unit testing, because unit tests do not catch bugs related to the behavior of a system at run time. A build can be "green" on the CI server but still may not function in the production environment.

Load or Scale Testing

Load or scale testing measures the performance of a solution against a specific set of resources. Ideally, you should run load or scale testing on a test farm that replicates the conditions of your production environment. The idea is to ensure that your system behaves well under normal high-end load conditions and to understand how your application scales as load increases. Load or scale testing uses coded UI tests, often with multiple computers running client test agents to simulate requests and measure responses. Preparing and running load or scale tests is a time-consuming and resource-intensive process.

User Acceptance Testing

User acceptance testing is any process that tests your solution from the user's perspective, such as load testing and functional testing procedures. In many agile development methodologies, the business owner for the system is also required to test the solution to ensure that business needs are being met. Functional testing by business owners is considered to be a part of user acceptance testing.