Running Tests as Part of a Build (Chris O'Brien)

In the previous post in this series, we showed how to deploy WSPs as part of an automated build process. This knowledge is useful for supporting deployment testing, but most people agree that the real point of Continuous Integration is to support automated testing. Automated testing can take the following forms:

• Unit tests
• Integration tests (usually defined as unit tests that call into the core product/platform – SharePoint in this case)
• User Interface (UI) tests

When executed as part of an automated build, such tests are often referred to as Build Verification Tests. In most scenarios, using a combination of automated test types yields the most benefit. It’s also worth remembering that a test strategy which relies exclusively on automated tests of any description is unlikely to be fool proof – manual testing still has a crucial part to play. The skill of an experienced tester is often hugely valuable in uncovering issues before users do. The idea of automated testing is to increase the efficiency of the testing process, since it is inherently a time-consuming process when performed only by humans.

Overview of testing in Visual Studio Team Foundation Server

Visual Studio and Team Foundation Server (TFS) provide a great toolset for implementing automated tests. Teams can achieve great efficiency when using a product like TFS to manage their software development process. Tests can be associated with work items (e.g. a bug or other task for a developer), and a range of tools exist to suit the needs and tasks of different stakeholders in the process (e.g. developers, testers, project managers). As an example, Microsoft Test Manager is suited to testers who may wish to run both automated and manual tests against the application.

For executing automated tests, Microsoft provides the command-line utility MSTest.exe as a “test engine.” Although other tools such as Visual Studio and Microsoft Test Manager provide UI entry points, MSTest provides a consistent execution model across unit/integration tests and UI tests. This consistency is useful when running tests from the command-line. Fortunately, it’s rarely necessary to go to direct to the command-line level when running tests as part of an automated build, but this is what happens beneath the covers: TFS Build calls into MSTest at the command-line level. In any case, it’s useful to know that this is an option.

Basics of adding tests to a build

In this article we’ll focus on using UI tests to uncover issues within the build process. UI tests can be very powerful as they test the application in exactly the same way an end-user would. The only difference between UI testing and manual testing is that the test framework drives the controls on the web page, not a human. UI tests, introduced in Visual Studio 2010, are known as “Coded UI Tests”, reflecting the fact that these UI tests are actually made up of .NET code. Usually this code is generated by recording the test steps in a browser, but as we’ll see later, it is designed to be modified if needed.

The following prerequisites must be met to use Coded UI tests in an automated build:

• Visual Studio 2010 Premium or Visual Studio 2010 Ultimate (for recording the tests)
• Installation of the Visual Studio 2010 Test Controller and Test Agent
  • Note that several topology options exist here, but it is common to install the Test Controller on the build server and the Test Agent on the SharePoint test server (where the build output is deployed to). See Installing and Configuring Visual Studio Agents and Test and Build Controllers
• Coded UI Tests which are created have properties that match what the build process is looking for
  • This includes meeting a naming convention, ensuring test assemblies are being deployed and the files can be found. Importantly, for Coded UI Tests certain changes must be made to the build workflow XAML (detailed later).

We’ll delve into these details later, but let’s first look at the process of recording a Coded UI Test.

Testing SharePoint applications with Coded UI Tests

Perform the following steps in the Visual Studio 2010 development environment.

1. Create a new Visual Studio project in which to store Coded UI Tests. Visual Studio provides a project template named Test Project that adds necessary references to the Microsoft.VisualStudio.QualityTools assemblies. Note also that this project should be a .NET Framework 4 project.

2. Delete the default unit test that is automatically added to the project. It’s recommended that you store unit tests in a different project to Coded UI Tests, since if they call into SharePoint 2010 (i.e. they are technically integration tests) they must use NET 3.5 code.

3. On the Add menu, select Coded UI Test… to add a new Coded UI Test to the project.

4. In the dialog that appears, select the first option to record the actions for this test.

5. The Coded UI Test Builder dialog box appears (along with an instructions panel) - this floating dialog controls the process of recording the test.

6. When you’re ready to begin recording steps to test within the application, click the red Record button on the dialog.

7. A new browser window must now be opened to record the test steps. (A good practice in UI tests is to always open a new browser instance at the beginning of the test, and close it at the end.) An appropriate way to open the browser is to use Start > Run > iexplore.exe, rather than using a shortcut which may not be present in all circumstances (e.g. for all user profiles).

8. Once the browser window has opened, navigate to the SharePoint site you wish to test.

9. Now, perform the steps in your application you wish to test. For this example, we’ll test a ribbon customization called “Subscribe” that updates a custom SQL database.

10. If the SQL update is successful, the user interface should respond with a success message (on a green background) on the SharePoint status bar. At this point, recording should be stopped as we now have an element in the UI we wish to test. To do this, click the Generate Code button, the rightmost button in the Coded UI Test Builder panel. A dialog appears prompting for a name for these steps.

11. In the background, Visual Studio generates .NET code representing the steps it followed. The crosshair button in the panel becomes enabled, and we will now use this button to test the UI. The crosshair can be dragged to any element on the page. As the crosshair is moved around, it detects elements of the page’s structure (the Document Object Model) and highlights the current element with a blue border. In this case, it’s the status bar message we wish to test, so drop the crosshair there.

12. When the element is selected, a property pane appears, listing the HTML and CSS properties of the DOM element. In this case, the DisplayText property is an appropriate property to test. Right-click DisplayText and then select Add Assertion on the context menu.

13. For this example, enter “Contains” in the Comparator box to test the value. (Many other comparators also exist.) Note that the element’s current value for the DisplayText property is automatically added to the Comparison Value box. Once the test looks satisfactory, click the OK button to add the assertion.

14. After the assertion has been added, click the Generate Code button in the Coded UI Test Builder panel. Provide a name for this test method when prompted.

15. Once the assertion has been added, we can continue testing the UI by adding more steps and assertions – simply press the red ‘Record’ button to continue recording. Something to consider here is that all the steps/assertions within one test recording constitute a single test within the test framework. It may be preferable to maintain only one assertion per test (in line with accepted unit testing guidelines), in which case recording should stop here and any other assertions should be added to separate Coded UI Tests.
When a test is complete, closing the Coded UI Test Builder panel generates .NET code for the test. The following image shows an example:

16. Once the code has been generated, you should rename the generated class and methods. In particular, the core method (decorated with the “TestMethod” attribute) should be renamed as this is used to reference the test. Give it a descriptive name that represents what is being tested in the application.

17. If appropriate, uncomment the “MyTestInitialize” and “MyTestCleanup” methods and then add any startup/teardown code that should run at the beginning and end of each test respectively. This may be necessary to manage state (e.g. to delete a database record added by a test).

18. Now that the test has been created, a good practice is to “test the test” immediately. To do this, use the Test List Editor window in Visual Studio. (If this window isn’t visible, you can enable it by selecting Windows on the Test menu.) To run a test, right-click it and select Run Checked Tests on the context menu.

19. The test framework should take control of the machine, open a browser window and run the test steps. Do not use the mouse or keyboard at this time. Once complete, the Test Results window should appear with an item specifying the test results.

20. If the test run is successful, the test can be included in the automated build. One prerequisite for being included in an automated build is that the test project must checked into TFS source control, so go ahead and check the test in now.

Considerations for using UI tests

What’s interesting about the test in the previous scenario is just how much code is exercised in the test. Although the specifics depend on the test/application, in the case of this “subscriptions” functionality, we know that the following code should be executed during this test:

• The declarative XML used to provision the SharePoint ribbon customization
• jQuery code that executes when the ribbon button is clicked
• A custom HTTP handler that is called using jQuery AJAX
• A method in a custom SharePoint service application that provides the ‘subscriptions’ functionality
• A method in the data access layer for the subscriptions data store
• The SQL stored procedure that writes the subscription record to the database

Clearly that’s a reasonable amount of code to test, and it would take some time to create unit tests providing the same coverage. Of course, the “coarseness” of a UI test means that care is needed when designing tests – if the UI was to respond with a false positive, the functionality may not actually be working as expected. Strategies for protecting against this might include using several UI tests for a given piece of functionality; combining UI tests with unit tests; and/or supplementing the generated UI test code with additional assertions.

Coded UI Tests, while powerful, have inherent challenges that developers must be aware of. They are fragile by nature and can be broken by frequent UI changes. They are also slower to run than unit tests or mocked tests. Developers should evaluate whether UI tests are appropriate – this may not be the case early in the development cycle when application’s user interface is changing frequently, since test maintenance could become a burden and this has an associated cost. A better strategy is usually to wait until the application has reached some level of maturity and stability.

Still, when used appropriately it’s easy to see why UI testing is appealing, given the fact it most closely reflects how end-users will use the application.

Integrating Coded UI Tests into the build process

Once Coded UI Tests are created, the good news is that running them as part of a TFS automated build is relatively simple. This is because the build workflows that ship with TFS 2010, such as DefaultTemplate.xaml, already have an activity that calls MSTest.exe, and this runs whatever tests have been specified in other parameters of the build definition. This is great as it means there is less setup work to do. However, while it works fine for unit tests, there is one crucial adjustment which must be made for Coded UI Tests, and this relates to the sequence of events during in the build process. Unit tests, of course, simply execute against the assemblies which have been built, but Coded UI Tests execute against the application itself. This means the assemblies must have been deployed to the application (via WSP deployment in SharePoint’s case), and any other deployment steps performed, such as application pool recycles, and so forth. Consequently, to use Coded UI Tests in an automated SharePoint build, the “Run Tests” activity in the workflow must be moved to a later sequence in the build process. By default, it runs immediately after compilation, which is inappropriate for Coded UI Tests against SharePoint.

To illustrate this, we can see in DefaultTemplate.xaml that the tests run in the Compile and Test for Configuration sequence.

However, in a workflow modified for SharePoint projects, the tests should run after WSP deployment.

Note: The customized build workflow supplied in the SharePoint 2010 Continuous Integration Starter Pack supports Coded UI Testing. In other words, the tests happen in an appropriate place in the build process.

In addition to this change to the build workflow, the configuration of the build must also match in several other crucial ways for any tests to run properly. (This applies to unit tests or Coded UI Tests.) If a customized version of DefaultTemplate.xaml is used for the build workflow (remember this usually represents the best starting point, and is the base for workflows in the SharePoint 2010 Continuous Integration Starter Pack), then this workflow exposes a parameter named Automated Tests. This property (and its sub-properties) define what happens when the workflow hits the activity that calls MSTest.exe.

The following values are especially important:

• Test Assembly Filespec – this property specifies the naming pattern used to find assemblies containing tests. The default value is “**\*test*.dll”, meaning that any assembly with “test” anywhere in the name is used. This is a good convention to follow.
• TestSettings file– this property value should contain the path in source control to a .testsettings file. This file defines many settings relevant to the test run. The following points summarize some key requirements for these settings:
  • Test execution method – for Coded UI Tests, this value should be Remote if the topology uses different machines for the Test Controller and Test Agent (e.g. build server and SharePoint deployment server). Note however, that when the tests are triggered manually from Visual Studio, it is necessary to use a value of Local. For this reason, multiple .testsettings files are usually created.
  • Data and diagnostics – here the diagnostic adapters which should collect data while the tests are running are specified. When a test fails, this data is critical to understanding the cause of the failure. Data from the test machine’s Windows event log can be captured, and it is even possible to configure video recording of the execution of the Coded UI Test – very powerful!
  • Hosts – A value of Run in default host should be selected, and Coded UI Tests should use the value Force tests to run in 32-bit process.

Some final things to consider are:

• Visual Studio projects containing tests must also be checked-in to source control, and included in the projects/solutions being built and deployed
• Coded UI Tests in Visual Studio 2010 require a project configured for .NET Framework 4 and x86 compilation. One consequence of this is that they cannot be in the same project as SharePoint integration tests (unit tests that call the SharePoint API)
• Running Coded UI Tests and SharePoint integration tests in the same build process requires the use of multiple test runs. This is because a test run using Coded UI Tests should run under a 32-bit host, whereas SharePoint integration tests require a 64-bit host to match the SharePoint installation. This can be managed by creating two .testsettings files (one for each execution), then adding a second test run in the build definition – clicking the ellipses next to the ‘Automated Tests’ parameter presents a dialog box to perform this configuration

For more information on some of these points see Use Your Build System to Work with Tests.

Getting the payback – finding the cause of a failed build

So, if we had an automated build that runs UI tests, how can that help us find out when a bug is checked in? Of course, the answer really depends upon the specifics of the bug and the tests you have in place, but let’s work through an example. (You may recall that this was covered in the first article of the series, SP2010 Continuous Integration – pt 1: Benefits, but for the sake of completeness, let’s look at it here too.) Let’s say there’s an issue in the data access layer in the example used above. We could imagine that there’s a mismatch of stored procedure parameters for example, but let’s use a slightly simpler case: a developer inadvertently adds some code that causes a DivideByZeroException:

int result = 0;

int i = 10 / result;

So, our fictional developer checks in because everything builds/packages fine and he/she has been testing throughout, and, of course, that tiny last change couldn’t possibly have broken anything! The build runs. If you’re in the office at this time (you might not be if the build is configured to run nightly), then team members have the option of seeing a system tray notification pop-up whenever a build succeeds or fails:

For detailed investigation, the main entry point is the build report – this gives us a view onto any build warnings/errors and a summary of test results.

There is also a more detailed log, and if UI tests are included then you’ll see the result of these – note the ‘Error Message’ section below shows that one of the assertions failed in a test.

That information’s useful, but in most cases we’ll need more detail to understand why we have a bug. Depending on which diagnostic adapters are enabled in the .testsettings file used by the build, links to some files should appear at the bottom of the build report:

Let’s walk through these files in turn:

1. A .png file – this is a screenshot of what the UI looked like at the time of the failing test. This is incredibly useful, and because we can see that the UI did not show the green ‘success’ bar , we get an understanding of how the bug manifested itself in the UI.

2. An XML file – this is data collected from the Windows event log at the time of the failing test. In many cases, this is how you’ll find the actual bug. In this case, we can clearly see the class/method and stack trace of where the problem occurred.

Note that this requires that you report exceptions to the event log in your code, such as by using the SharePoint Patterns and Practices logging framework. Crucially, we now have a screenshot of the failed test and event log data. This is typically the information needed to locate the offending bug.

3. A video file – yes, it’s actually possible to get Visual Studio to capture a video recording of the Coded UI Test run (this requires Visual Studio Service Pack 1 and Expression Encoder). This is extremely powerful as it allows anyone investigating the failed build to get a clear understanding of exactly how the UI responded during the test, even if it ran overnight.

Summary

Once testing is added to the build process, implementing continuous integration starts to give huge gains. Successful software development involves finding bugs as early as possible, and it’s easy to see how automatically building and testing the latest code several times a day (or at least overnight) can enable a team to identify issues sooner. Coded UI Tests provide a powerful approach to testing the application. Even without automated builds, testers can spend their time recording repeatable UI tests rather than only running manual tests. When run as part of an automated build, however, the benefits are even bigger as the tests can execute very regularly with little human intervention.

Comments

• Anonymous
  April 12, 2012
  How do you comapre this type of UI test with the similar feature in the Visual Studio test manager in the Ultimate and Test MSDN versions?

• Anonymous
  December 18, 2013
  I am facing issue while running my coded ui test as part of build. My coded ui test have DDA attached to it as part of CSV files. While build succeeded, coded ui test failed with the following error message. "The unit test adapter failed to connect to the data source or to read the data. For more information on troubleshooting this error, see "Troubleshooting Data-Driven Unit Tests" (go.microsoft.com/fwlink) in the MSDN Library.Error details: The Microsoft Jet database engine could not find the object 'DDAAccounts.csv'.  Make sure the object exists and that you spell its name and the path name correctly". Coded UI test are running successfully as part of the build, but the test which are having CSV files attached as part of DDA are only failing. Can anyone suggest a possible solution.