Test configurations: specifying test platforms
Your users will probably install or run your app on a wide variety of configurations, such as different operating systems, web browsers, and other variations. You will want to run at least some of your tests in environments that have those different configurations. Use your test plan to decide which tests you want to run on which configurations. You have to make sure that when you run your tests that you have set up your environments for the configurations that you need.
You might draw up a schematic matrix of the combinations that you want to test:
.png "Configuration variables and configurations")
Use Microsoft Test Manager to specify test configurations. But you can still run the tests either with Team Web Access or with Microsoft Test Manager.
Requirements
- Visual Studio Ultimate, Visual Studio Premium, Visual Studio Test Professional
Planning tests with configurations
Connect Microsoft Test Manager to your test project and open your test plan. Open your test plan by opening Testing Center, Plan, Contents.
Select one or more tests, then choose Configurations.
.png "Assign configurations to the test points.")
Set the configurations you want to run the tests on.
.png "Assign configurations to the test points.")
- Don’t see the configurations you want? Choose All configurations. If you still don’t see what you need, learn how to define your own configurations.
I have a test case that appears in several test plans and test suites. Do I have to set the configurations for each of these test points?
Yes. The same test case can have different configuration settings in different test suites and test plans.
Running tests with configurations
When you want to run a test that has multiple configurations, you'll see that it appears more than once in the run list.
.png "Required configurations in the lists of tests.")
Set up a testing platform for a particular configuration, and then sort the list to show the tests to run on that configuration.
.png "Sort test points by configuration")
When you run a test, a reminder of the required configuration appears on the Test Runner window.
Team Web Access |
Microsoft Test Manager |
|---|---|
.png "Configuration name in Test Runner.")
|
.png "The required configuration appears in Test Runner.")
|
Test Runner doesn't verify that you're actually running on the specified configuration. However, if you use Microsoft Test Manager, system information is stored in the test log.
Create new configurations for your team project
A few configurations are already defined for you, but you'll probably want to add your own.
A test configuration is a combination of configuration variable values. Your configuration variables could be, for example, operating system, browser, CPU type, database. A configuration might be "Windows 7 + 32-bit CPU" or "Windows 8 + 64-bit CPU."
Choose Testing Center, Organize, Test Configuration Manager.
To add your own configuration variables and values, choose Manage configuration variables:
.png "Define configuration variables and values.")
Create new configurations that your tests can use:
.png "Define configurations as combinations of values.")
Q and A
Q: Are different test data a good use of a test configuration variable?
A: It’s better to use parameters when you want a test to be run with different test data, because it’s easy to set different parameters for different test cases. Test configurations are better for variations in the hardware or software platform on which the application under test is installed.Q: Repeating tests on different configurations is boring.
A: And slow and error-prone. But there is a way to speed things up. Record your actions on one configuration, and then play them back on another.If you play back on a different browser, choose the Change browser for playback option under the play menu in test runner.
However, be aware that record/playback doesn’t work for all browsers and applications. In some cases you might have to play back some steps manually.