How to: Configure and Run Scheduled Tests After Building and Deploying Your Application

You can use Visual Studio Lab Management to check the quality of builds for your application. You can use a specific lab template for a build definition to build your application, deploy the application to a virtual environment, and then run automated tests for that build. This process enables you to make sure that the tests are run in a clean environment by using a known state for your virtual environment. If you have test failures, you reduce the probability that this is caused by an incorrectly configured or corrupted environment, because you can use the same clean one every time.

You run the tests by selecting a test suite from your test plan. For example, you might create a test suite called Build Smoke Tests. You can associate automated tests that test the basic functionality of your application with test cases, and then add these test cases to your suite. For example, the tests might be coded UI tests that test basic operations in your application.

Every time that the build workflow is completed for this build definition, the test results will be saved. You can use these results to view how stable your build is and use that information to decide whether this is the build that the testing team should now start to use. Also, you can check daily to see how frequently these tests passed for your builds.

Use the following procedures to create the build workflow to build, deploy and test your application, investigate any issues and view or analyze your results:

• Check the Prerequisites

• Create Your Build Definition and Start Your Build

• Connect to the Environment from Your Build Results

• View and Analyze the Test Results For Your Build

Web and other complex applications might require additional steps to deploy them when you use Visual Studio 2010. For example, if you use Visual Studio 2010 to deploy a Web application to an IIS server, the additional steps are described on this Microsoft Web page.

Note

You can only use the lab template for a build definition with Manual, Scheduled, or Rolling build triggers. Rolling build triggers are not recommended, because a test failure still allows the next rolling build to start, or they stop the entire build system. Gated check-in and Continuous integration triggers are not supported.

Prerequisites

Before you can set up your build workflow to build, deploy and test your application, use this list to verify that you have completed the following tasks:

Prerequisite Tasks

1. Configure Lab Management including a build controller and a test controller: Configuring Lab Management for the First Time.

2. Create virtual machines for your environment and add the agents to these virtual machines. Then store the virtual machines into your library share: How to: Create and Store Virtual Machines and Templates Ready for Lab Management.

3. Import the virtual machines or templates into your team project from SCVMM: How to: Import a Virtual Machine or a Template from SCVMM.

4. Create an environment that uses these virtual machines in the required roles and select to run tests and use a workflow for this environment and start your environment: How to: Create an Environment from Virtual Machines or Templates.

   Note

   You must install and configure any software that is required for each virtual machine for the role that it has been selected for. Selecting a role for a virtual machine does not install any software.

5. To run tests as part of your scheduled tests that interact with your desktop, such as coded UI tests, you must follow these steps:

   1. Set up your test agent to run as an interactive process instead of a service: How to: Set Up Your Test Agent to Run Tests that Interact with the Desktop.

      You will now see the test agent process running in the notification area with a status of online.

   2. If the build definition for your workflow reverts to a specific snapshot, the virtual machine cannot be locked when you try to run the tests. You must connect to the machine by using a host based connection or using a console session before you take the snapshot to use for your workflow. For more information about this, see How to: Connect to a Virtual Environment.

6. (Recommended) Take a snapshot of your environment to use as a clean state for your workflow: How to: Save the Current State of Your Environment. Before you take this snapshot, follow these steps:

   1. Make sure that the virtual machines in your environment have the latest updates for their operating systems.

   2. Run the command gpupdate /force for any virtual machine in the environment that is connected to a domain to make sure that any changes to user policies have been updated. If you do not run this command, your deployment scripts might not work correctly or your tests might not run correctly. Make sure that your code projects and test projects for your application are checked into source control: Placing Files under Version Control.

   3. Check that the state for the environment is "Running" and that the state of both the workflow and testing capabilities is "Ready".

      Note

      If the virtual machines in this snapshot are joined to a domain and the snapshot is used for longer than the password expiry period for the domain controller, the virtual machines may no longer be able to join the domain. For more information, see How to: Save the Current State of Your Environment.

7. Create a build definition for your application that you can use to build your application, or to select a specific build, when you create the build workflow using the lab template: Create a Basic Build Definition.

8. Associate your automated tests with a test case, and add the test case to a test suite in your test plan for the team project. For more information, see the following Help topics:

   1. How to: Associate an Automated Test with a Test Case

   2. How to: Add a Test Case to a Test Suite

   3. How to: Create a Test Plan.

9. Create test settings to run automated tests that use the roles in the virtual environment that you have created: Create Test Settings for Automated Tests as Part of a Test Plan.

Create your Build, Deploy and Test Workflow Using the Lab Template for Your Build Definition

To create your build, deploy and test workflow, you must follow these steps:

Step	Action
1	Create a basic build definition
2	Create another build definition for your build, deploy, and test workflow
3	Add the details for your workflow
4	Queue the build definition for your workflow

Create a Basic Build Definition

You must first create a build definition for the code in the application that you want to deploy. If you plan to build your application every time, disable tests in this definition because you will run the tests from your workflow by using the Lab template.

To create a build definition for your application

1. On the Build menu, click New Build Definition.

2. On the General tab, in the Build definition name box, specify a name and in the Description box add an appropriate description.

3. Follow the steps as described in the topic Create a Basic Build Definition.

Create the Build Definition for Your Workflow

Next you must create another build definition for your build, deploy, and test workflow, as shown in the following illustration:

You must select the LabDefaultTemplate file to create your workflow by selecting to show the details for the build process template, as shown in the following illustration:

To create a build definition for your workflow

1. On the Build menu, click New Build Definition.

2. On the General tab, in the Build definition name box, specify a name and in the Description box add an appropriate description.

3. Choose settings on the Trigger and Workspace, Build Defaults, and Retention Policy tabs as described in the topic Create a Basic Build Definition.

   Note

   You do not have to enter a build drop path on the Build Defaults tab for this build workflow because you do not create build output when you use the lab template. Clear My builds copy outputs and no drop folder is required.

4. To be able to select the Lab Template for the build definition, on the Process tab, under Build process template, click Show details.

   A drop-down list appears.

5. Select a template. This is the build process file that defines your workflow.

6. To create a workflow for your build definition to deploy your application to a virtual environment, select LabDefaultTemplate.xaml from the drop-down list for Build process file.

Add the Details for your Workflow

Now you can add the details for the workflow process, as shown in the following illustration.

The Lab Workflow Parameters wizard takes you through the information that you must provide.

You can now queue this build to run your workflow and view the progress of your build workflow.

To add the details for your workflow

1. To enter the data for your workflow, under Build process parameters, click Lab process settings and then click the ellipses (…).

   This opens the Lab Workflow Parameters wizard where you enter the information for the workflow.

2. On the Environment tab, select the virtual environment to which you want to deploy your application.

   Note

   This environment must be active. If you are using an environment that is stored in the library, you must deploy the environment to make it active. We also recommend that this environment be created specifically for your workflow and should not be used by other users. This will prevent issues in which the environment is currently being used and the build workflow reverts the environment to a specific snapshot, or deployment scripts are run on the environment when another user runs a test.

3. (Recommended) If you want to have the lab build definition revert the environment to a known state, select Revert to a specific snapshot of the environment and then click the ellipses (…) to select a specific snapshot.

   The Select environment snapshot dialog box is displayed. Select the snapshot and then click OK.

   Important

   It is recommended that you revert to a snapshot to make sure that you consistently run your tests every time that you build from a known state for your environment. This reduces uncertainty in determining the cause of test failures. For example, another user could have changed the current environment by adding software that could cause the tests to fail.

4. Click Next.

5. If you want to use this workflow definition to build your application every time that you queue this workflow definition, follow these steps:

   1. Select Use a Team Foundation build, and select the definition that you created earlier.

   2. Select Queue a new build.

6. If you want this workflow definition to use an existing build and not rebuild your application, do the following:

   1. Select Use a Team Foundation build, and select the definition that you created earlier.

   2. Select Select an existing build. Then select a build from the drop-down list. The existing builds created by the build definition that you selected are displayed in the list.

   3. Select a build configuration from Select build configuration.

      Note

      The build configurations are specified when you create your build definition for your application. If there is more than one build configuration, you can select one from this list.

7. If you want to define the location of a build, select Use a build from a specified location and then specify the UNC path of the existing build.

8. Click Next.

9. To deploy the application as part of your workflow, from the Deploy tab select Deploy the build.

10. To add the scripts or commands required to deploy your application, click Add. Select the virtual machine that you want to add the script or command for.

    You can now add scripts or commands for each virtual machine in your environment. For example, if you have a Windows client as part of your application, you might have a script that copies the executable to the location that your coded UI test will use to start the tests on your virtual machine. If you have a Web server then you will have to run the script or command to deploy that part of your application.

    The following variables are available for you to use with your scripts:

    • $(BuildLocation): This is the location of the build. If you specified to use the build from a shared location, then this variable represents that path. For the other options, this is the full path for the build based on the configuration that you selected to build and the build drop location in the build definition. If you build your application as part of your workflow, you can use this to access the latest files that were created by that build.

    • $(InternalComputerName_<VM Name>): This is used to obtain the computer name for a virtual machine that is part of a virtual environment. You might know the virtual machine name, but not the computer name. If you have a deployment script to set up a Web server that requires the computer name, you can pass this as an argument to the script. For example, if the virtual machine name for the Web server was VM1 and the computer name was MyWebServer, you would type $(InternalComputerName_VM1) as the argument for your script and this would pass the value MyWebServer to your script.

    • $(ComputerName_<VM Name>): This is the fully qualified domain name of the virtual machine. This can be used to access the computer even from outside the virtual environment. You might want to pass this as an argument to set up a Web server. For example, if the virtual machine name for the Web server was VM1, you would type $(ComputerName_VM1) as the argument for your script to pass the fully qualified domain name of the virtual machine.

    If you are using network isolation for your environment the value of $(InternalComputerName_<VM Name>) will be the same for the instance of a virtual machine in each copy of this environment, but the $(ComputerName_<VM Name>) is different. For example, the computer name for a virtual machine could be MyWebServer in each copy of the environment but the fully qualified domain name would be unique: VM_<unique identifier>.domain_name.com.

    Important

    If you want to add a command that is run from a windows prompt, such as mkdir or running a batch file, you must begin the command by using cmd /c. For example, cmd /c $(BuildLocation)\copyexe $(BuildLocation) where copyexe is a batch file copyexe.bat that copies an executable to a local directory on the virtual machine.

    If your script or command requires a specific working directory, you can type the directory in Working directory.

    Note

    Verify that you can run your tests based on the location of files after you deploy your application. For example, if your coded UI test starts a Windows client application, verify that the executable is in the correct directory for the tests to run.

    You might also want to verify that the names of the machines in your environment are correct for your application. For example, you might have to verify that the virtual machine for the Web server role is configured to access a database server instance on the virtual machine for the role of database server.

11. (Recommended) To take a snapshot of your environment after the application has been deployed, but before you run any tests, you must do the following:

    1. Select After deploying the build, take a snapshot of the environment.

       Important

       If you run this build definition as part of your nightly workflow process, each virtual machine in the environment will soon have many snapshots associated with it. This deteriorates the performance of the virtual machine. In addition, there is a limit of 50 snapshots that can be stored for each virtual environment. Therefore, you must delete the old snapshots regularly.

    2. In Enter the snapshot name, type a name for this snapshot.

    Note

    You can use this snapshot to connect to the environment and rerun a test if you want to investigate an issue. Or, another member of your team could do this. It can frequently be useful to have this snapshot to rerun a test on a clean system with the application installed to determine what occurred, or even to verify that the application was installed correctly.

12. Click Next.

13. To run automated tests after you deploy your application, you must follow these steps:

    1. Select Run these tests in the environment.

    2. Under Select the test plan, select the test plan that you want to use. The test results will be saved as part of this test plan.

    3. Under Select the test suites click the ellipses (…), and in the Select test suites dialog box, select the test suites you want to run.

       Note

       By default, the root test suite is selected. If you do not want to run tests in this test suite, you must clear this field.

    4. Under Select the test configuration, select the configuration that you plan to use to run your tests.

       Note

       The test results for each test case in each selected test suite will be saved as a pairing of each test case in the suite and the test configuration that you selected. For more information about test configurations, see Defining Your Test Matrix Using Test Configurations.

    5. Under Select the automated test settings, select the test settings that you created earlier to match the roles in your virtual environment in step 9 of the overall process for this topic. For more information about test settings, see Create Test Settings for Automated Tests as Part of a Test Plan.

14. Click Finish.

15. Click Save to save your build definition.

    The created build definition appears in the Builds folder in Team Explorer.

Queue the build definition for your workflow

You can now queue this build to run your workflow and view the progress of your build workflow.

To queue the build definition for your workflow

1. To start the build definition to build, deploy, and test your application, right-click the lab build definition in the Builds folder and click Queue New Build.

   The Queue Build dialog box is displayed.

2. Verify the information for your build workflow and then click Queue.

   The Build Explorer view is displayed.

3. To see the Build Summary view as the build progresses, double-click your build.

   You can see the status as the build progresses.

4. (Optional) If you want to view the environment as the build progresses, open Microsoft Test Manager, locate the Lab Center, click Lab, and then click your environment in the list. You can view the progress of the build reflected in the image for your environment and in the environment details above this image as follows:

   • The snapshot is restored if you selected this option.

   • The post-deployment snapshot is taken if you selected this option.

   • The status of the capabilities (a green arrow is displayed when a capability is ready).

   • The tests as they run, if the tests interact with the user interface.

   If your build workflow is completed successfully, you will see a green check mark . If there are errors, you can click View Log to see details.

Connect to the Environment from Your Build Results

You might want to connect to your environment to investigate an issue if a test fails during the build workflow process. You can connect either to the post deployment snapshot, if you selected this option in your build workflow, or to the environment in its current state, as shown in the following illustration.

To connect to the environment from your build results

1. From the Builds folder in Team Explorer, right-click your build workflow definition and point to View Builds.

   The Build Explorer view is displayed.

2. To view your completed build, click the Completed tab.

3. Double-click the build that you want to view.

   The Build Summary view is displayed.

4. Click the link next to View environment snapshot <Build name and number>.

   The Connect to environment dialog box is displayed.

5. If you want to connect to the snapshot that was taken after the application was deployed, click Connect to the snapshot in this environment.

   Note

   By connecting to this snapshot, any changes that were made after this post-deployment snapshot will be discarded. If you want to keep any changes, connect to the environment in its current state and take a snapshot first, before reverting to the post-deployment snapshot. For information about how to take a snapshot, see How to: Save the Current State of Your Environment.

6. If you want to connect to the environment in its current state after you have run any tests from your workflow, click Connect to the environment in its current state.

7. Click Connect.

   The Microsoft Environment Viewer is displayed and you are connected to the environment. You can now investigate any issues.

View and Analyze the Test Results for Your Build Workflow

You can view the test results summary in your build workflow summary. However, you can also view and analyze the test results by using Microsoft Test Manager because the results are stored as part of your test plan. This is shown in the following illustration. For more information about reporting on test results for a test plan, see Reporting on Testing Progress for Test Plans.

To view and analyze the test results from Microsoft Test Manager

1. Open Microsoft Test Manager.

   Note

   To display the Microsoft Test Manager window, click Start, and then click All Programs. Point to Microsoft Visual Studio 2010 and then click Microsoft Test Manager.

2. To view the test results, click the down-arrow on the center group switcher and then click Testing Center.

3. On the center group menu bar, click Test and then select one of the test suites from the test suite hierarchy that you used in your build workflow.

   You can see the results for the tests for the configuration that you selected in your build workflow.

4. If you want to analyze the complete test run, click Analyze Test Runs.

   The Analyze Test Runs activity is displayed. It shows any test runs for this test plan.

   Note

   The run title will reflect the name of your build definition. The run ID is displayed in the build summary page for your build to help you identify the run.

5. Double-click a test run to open it and view the details. The test run details are displayed.

6. (Optional) To update the title of your test run to be more meaningful, type the new name in Title.

7. (Optional) If your test failed, you can update the reason for the failure. Click Resolution and select the reason for the failure from the list.

8. (Optional) To add comments to the test result, click the Comments icon. Type your comments and then click Save comments.

9. (Optional) To view the details of the individual test, double-click the test.

   The test result is displayed. It shows the details from the test run, the attachments for data that was collected for this test result, and the test results history for that test. You can close this view to return to the test run.

   Note

   If you determine that there is a bug, you can create a bug from this view.

10. To save any changes for the test run, click Save in the toolbar.

See Also

Concepts

Testing Using Virtual Environments

Other Resources

Troubleshooting Running Tests on Your Virtual Environment