How to: Run Manual Tests and Create Reproducible Bugs With Virtual Environments
You can use virtual environments when you run your manual tests, if you have Visual Studio Lab Management. You create the virtual machines or templates for the roles that are required for your application and select this set of roles from your test settings. For more information about environments, test settings and roles, see Setting Up Test Machines to Run Tests or Collect Data and Setting Up Machines and Collecting Diagnostic Information Using Test Settings.
When you set up your virtual environment to run manual tests, you create virtual machines or templates for the backend roles of your application. For example, you could create a virtual environment that includes virtual machines or templates for the Web server and database server roles in your environment. Then you could run the manual tests from a desktop computer that is not part of the environment. Or, you might run a browser to connect to your Web server from this computer, as shown in the following illustration.
Note
You must install Visual Studio Test Professional 2010 or Visual Studio Ultimate on your desktop computer to run the manual tests if you use Test Runner.
.png "Run Manual Tests from a Virtual Environment")
Note
If you want to use this same environment to run automated tests, you might want to add a role for the desktop client and use a virtual machine or template for that role. You cannot use this role to run your manual tests. However, you can use this role when you run automated tests. For more information about how to run automated tests with virtual environments, see How to: Run Automated Tests Using Virtual Environments.
If you find a bug in your application when you run your manual tests, you can take a snapshot of your virtual environment that developers can use to connect to the actual state of your servers in your environment when you found the bug. This helps reduce the number of bugs that cannot be reproduced by developers. This state of the environment is saved for developers to use to investigate whenever it is convenient, and you can continue to run more tests on this same environment. When developers want to use the environment, they can connect to the snapshot by using the link in the bug.
Note
This snapshot is a snapshot of each virtual machine in the environment. These snapshots are stored on the physical host that is used for the virtual machine. The snapshots do not use physical space on Team Foundation Server. In the bug or the test results, only the link to the snapshot is added.
If the environment might be deleted before the developer checks the bug, the tester can store a copy of the environment in the library share when they have completed testing with it. The link in the bug will allow the developer to create a copy of the stored environment and use this.
Note
If you want to create a copy of the environment using the link in the bug, the environment must be a network isolated environment.
If you want to build and deploy the backend tiers of your application to your virtual environment to easily set up to run your manual tests, you can create a build definition to do this. You will require a build controller for the workflow integration capabilities. For more information, see How to: Deploy an Application on a Virtual Environment.
Prerequisites
Before you can run manual tests using your environment you must follow these steps:
Create a virtual machine or template for each role that is required in your virtual environment.
Install a lab agent on each virtual machine.
Install a test agent on each virtual machine. This enables you to use the virtual machines in your environment to collect data and diagnostics when you run your tests, and to create a snapshot of your environment when you run your tests and find a bug.
If you want to deploy your application by using the workflow integration capability, you must also install build agents on the virtual machines.
Install Visual Studio Test Professional 2010 or Visual Studio Ultimate on any of the machines that you want to use to run your manual tests.
Create your manual test cases by using Microsoft Test Manager and add them to a test suite in a test plan. For more information, see Defining Your Testing Effort Using Test Plans.
Procedures
Use the following procedures to enable you to run your manual tests and create reproducible bugs. If you find an issue, you can connect to the virtual machines in your environment to verify information, or take a snapshot of the environment and save it in a bug so that a developer can use the environment to help reproduce the bug.
Run manual tests using virtual environments
Connect to the virtual machines in your environment
Create a snapshot of your virtual environment
Connect to a snapshot of a virtual environment
Run Manual Tests Using Virtual Environments
To run manual tests using virtual environments
Create a virtual environment and add the necessary virtual machines or templates for each role. For more information, see Virtual Environments Concepts and Guidelines and How to: Create an Environment from Virtual Machines or Templates.
Important
To collect data and diagnostics on the virtual machines in your environment when you run your manual tests, you must enable the testing capability in the virtual environment.
To do this, you must have a test controller that you can use for your environment. For more information, see How to: Add Test Controllers to Use for Testing with Lab Management.
To start the virtual environment, select the environment from the Environments view and click Start.
Verify that the machines in your environment have started and that the test capability is enabled if you selected this capability for your environment. In the Environments view, the status of the testing capability is displayed next to Testing in the details for the environment.
Note
If the environment that you want to use has been stored in the library, you must deploy the environment to create the virtual machines from the stored images before you can start the environment.
(Recommended) To prevent other users from accidentally connecting to your environment when you run your tests, click the drop-down arrow next to Mark 'In Use' in the toolbar. Add a comment about who is using the environment and click Mark 'In Use'.
If another user tries to connect to the environment that is being used, a warning with the comments that you entered is displayed in the Connect to environment dialog box. However, this user can still connect to the environment and ignore the warning.
Note
You can also use the Microsoft Environment Viewer to mark which environment is being used.
Install the backend tiers of your application on the virtual machines in your environment. To connect to the backend tiers, you might also have to install the local client for your application on your desktop computer. You can use the workflow integration capability to install the appropriate backend tiers of your application automatically by using a build definition. For more information about how to do this, see How to: Deploy an Application on a Virtual Environment. Or, you can install the application manually.
Note
We recommend that you take a snapshot of your environment after you install your application so that you can easily return to this clean state with the application installed.
Create test settings that contain the set of roles for your virtual environment by using Microsoft Test Manager and select the data and diagnostics that you want to collect for each role. For more information about how to create test settings for manual tests, see Create Test Settings for Manual Tests.
To assign the test settings and your virtual environment that you created to your test plan and run your tests, follow these steps:
Using Microsoft Test Manager from the machine where you want to run your tests, click the down-arrow on the center group switcher and then click Testing Center.
On the center group menu bar, click Plan and then click Properties.
The properties for the currently selected test plan are displayed.
Click the drop-down arrow for Test settings under Manual runs and select the test setting.
Click the drop-down arrow for Test Environment under Manual runs and select the environment.
Click Save and Close.
To run your tests, click Test, click Run Tests. Select the tests that you want to run, and click Run.
To run your tests using the test settings that you created for this specific test run only, follow these steps:
Using Microsoft Test Manager from the machine where you want to run your tests, click the down-arrow on the center group switcher and then click Testing Center.
To run your tests, click Test, click Run Tests. Right-click the test suite or tests that you want to run, and click Run with options.
The Run Options dialog box is displayed.
Select the test settings that you created from Test Settings under Manual test runs.
Select the virtual environment from Environment under Manual test runs and click Run.
Test Runner is now displayed with the first test selected.
To start the first test, click Start Test. For more information about how to run manual tests, see How to: Run Manual Tests.
You can now begin to run all the tests that you selected.
To run the steps in your test, you must start your application under test. To connect to the part of the application that is installed on your environment, you use the client for your application. This could be a desktop client or a Web browser that is installed on the computer where you have installed Microsoft Test Manager.
If you are using an environment that is network-isolated, see the following topic for more information about connecting to your application, see the section about Communicating Between Machines in Network-Isolated Environments and Computers in the External Network in the following topic: How to: Create and Use a Network Isolated Environment.
To save your test results, click Save and Close from Test Runner.
The Run Tests view is displayed.
Connect to the Virtual Machines in the Environment
If you want to verify some information about a machine in your virtual environment when you run your tests, you can connect to any of the virtual machines. For example, you might want to verify that the value of a registry entry is correct.
To connect to the virtual machines in the environment
To view the state of a machine in your virtual environment when you run a test, you can click Connect to environment in the test steps toolbar for Test Runner.
The Microsoft Environment Viewer is displayed. You can select the virtual machine that you want to view. You can also run commands on the virtual machine.
For more information about how to connect to machines in your environment, see How to: Connect to a Virtual Environment.
If you are using an environment with network isolation, see How to: Create and Use a Network Isolated Environment.
Note
You must not restore to a snapshot when you connect to an environment while you are running tests. If you restore to a snapshot while tests are running, the test agents will be restarted and the data and diagnostics cannot be collected for the test run. If you want to restore a snapshot, you must stop the test run and restore to a snapshot before you start to run tests.
Create a Snapshot of Your Virtual Environment
To create a snapshot of your virtual environment
If you find an issue when you run your test, click the Take Environment Snapshot icon at the far right of the toolbar for the test steps panel. Now when you create a bug, the link to this snapshot of the virtual environment is added to the bug.
Note
This takes a snapshot of each virtual machine in the environment. These snapshots are stored on the physical host that is used for the virtual machine. The snapshots do not use physical space on Team Foundation Server. In the bug or the test results, only the link to the snapshot is added.
In the Details tab for the bug, you can see the snapshot with the following name: TC<test case id>snapshot<snapshot number>.lvr. You can also see any data or diagnostics that was collected on the virtual machines for your environment in the details for the bug.
Note
You can also view this link to the snapshot in the test results.
To save your test results, click Save and Close from Test Runner.
The Run Tests view is displayed.
Connect to a Snapshot of a Virtual Environment
To connect to a snapshot of a virtual environment
To connect to the virtual environment snapshot from your test results, from the Run Tests view in Microsoft Test Manager, right-click a test and point to View Results.
Click the link to the snapshot of the environment displayed in the test step when you created the snapshot.
Click Open to open the file.
The Connect to environment dialog box is displayed.
From this dialog box, select one of the following options:
To restore the exact state of the virtual environment at which the snapshot was taken, click Connect to the saved snapshot in this environment.
Note
If you select this option, you might disconnect any user who is currently using this environment and this could lead to loss of their work. You can verify that there is no warning message that the environment is in use.
To connect to the environment in its current state and not revert to the snapshot that was taken when the test was run, select Connect to the environment in its current state and click Connect.
Note
You might still disconnect another user who is connected to this environment, but you will not change the state of the environment.
If there is a stored copy of the environment in the library share, you can create a copy of the environment and connect to that. You might want to do that if the environment is currently in use, or you do not want to prevent the testers from using the environment. To create a copy of the environment and connect to that copy, select Connect to a new instance of this environment.
Note
When you create a copy of the environment, this can take a long time and also uses significant disk space on the project library share. If this copy of the environment might be started at the same time as the tester's copy of the environment, you have to create a network isolated environment. For more information about environments, see Creating Virtual Environments.
To connect to the virtual environment snapshot from your bug, you can open the bug from the Verify Bugs view using Microsoft Test Manager, or you can open the bug from the Team Explorer view in Microsoft Visual Studio 2010. Then you can click the link to the snapshot of the environment in the Details tab for the bug.
The Connect to Environment dialog box is displayed. Select the appropriate option as in the previous step in this procedure. If the environment has been marked in use, a warning message appears in the Connect to environment dialog box
Note
Developers can connect to the virtual environment from the bug to re-create the issue in the bug and help them resolve the issue more quickly. When you restore an environment to a snapshot, the network connections between the machines in the snapshot might stop working and any transactions that were in process at the time that the snapshot was taken might fail. If reproducing a bug depends on TCP connections that are open and active when the snapshot was taken, then you might not be able to reproduce the bug.