Guidance on Microsoft Test Manager usage
TFS 2017 | TFS 2015
Visual Studio 2017 | Visual Studio 2015
Azure DevOps and Team Foundation Server (TFS) offer a web-based solution for manual testing. Azure Test Plans or the Test hub in TFS is a web-based manual testing solution, which works across all platforms and with all browsers. It provides experiences across Plan, Author, Execute and Track stages of Manual testing.
In the past, we have also shipped a client based solution as Microsoft Test Manager. The Test Center in Microsoft Test Manager client is a desktop-based manual testing solution, which has traditionally been used by testers for their manual testing needs (see Run manual tests with Microsoft Test Manager).
Note
Microsoft Test Manager has been deprecated. It has been replaced with Azure Test Plans, a fully featured test management solution. Microsoft Test Manager 2017, which shipped with Microsoft Visual Studio 2017, is the last version available. This topic will help you understand why Azure Test Plans or the Test hub in TFS is a more comprehensive solution for manual testing compared to Microsoft Test Manager.
Supported combinations
| Client | TFS 2015 | TFS 2017 | TFS 2018 | Azure DevOps Server 2019 | Azure DevOps Server vNext | Azure DevOps Services |
|---|---|---|---|---|---|---|
| Microsoft Test Manager 2015 | Yes | Yes | Yes | Yes | No | Up to Jan 1, 2020 |
| Microsoft Test Manager 2017 | No | Yes | Yes | Yes | No | Up to Jan 1, 2020 |
Manual Testing
Azure Test Plans or the Test hub in TFS is a fully-featured test management solution spanning all stages of the testing lifecycle. It works on all platforms (such as Linux, macOS, Windows, and others) and all browsers (such as Edge, Chrome, Firefox, and others). You can easily get started using manual testing features right from your Kanban board, and use it for more advanced manual testing capabilities. This topic describes the capabilities included in Azure Test Plans or the Test hub in TFS.
Test planning
Create and manage test plans and test suites for your teams with ease. Export and share the test plans and test suites with your team or assign multiple testers to execute tests. See the comparison matrix below for more information about these features.
Comparison of test planning with Azure Test Plans or the Test hub in TFS and test planning with Microsoft Test Manager:
| Test planning capability | Web-based test features | Client-based Microsoft Test Manager |
|---|---|---|
| Create test plan | ✔️ | ✔️ |
| Create/Manage suites | ✔️ | ✔️ |
| Add/remove tests from test suite | ✔️ | ✔️ |
| Assign individual testers to test plan/test suite | ✔️ | ✔️ |
| Create/edit/assign configurations | ✔️ | ✔️ |
| Copy/clone test plan/test suite | ✔️ | ✔️ |
| Order manual tests within suites (RBS, QBS, Static) | ✔️ | ❌ |
| Export test plans and test suites | ✔️ | ❌ |
| View test case references across test suites | ✔️ | ❌ |
| Assign multiple testers to test plans and test suites for user acceptance testing | ✔️ | ❌ |
The copy/clone capability is surfaced via the following APIs: Copy/Clone a Test Plan and Copy/Clone a Test Suite. We plan to expose this capability through the out-of-box UI in a future release.
Test authoring
You can easily get started creating test cases right from your Kanban board in the Work hub. Easily add, view, interact with, and execute tests from your Kanban cards, and create multiple test cases using a grid in Azure Test Plans or the Test hub in TFS. Create shared parameters and use them in multiple test cases for data driven testing.
Comparison of test authoring with Azure Test Plans or the Test hub in TFS and test authoring with Microsoft Test Manager:
| Test authoring capability | Web-based test features | Client-based Microsoft Test Manager |
|---|---|---|
| Create and edit test cases using WIT form | ✔️ | ✔️ |
| Create and edit shared steps | ✔️ | ✔️ |
| Bulk author and edit test cases | ✔️ | ❌ |
| Inline tests on Kanban board | ✔️ | ❌ |
| Create and edit shared parameters | ✔️ | ❌ |
Test execution
Test your web apps and your desktop apps.
Azure Test Plans or the Test hub in TFS provides a browser based Test Runner and a client based Test Runner which you can use to test your web and desktop apps. Using either runner, you can mark test steps and test outcomes as pass or fail, and collect diagnostic information such as system information, image action logs, screen recordings and screen captures during your tests. Alternatively, you can use the Microsoft Test Runner client that is part of Microsoft Test Manager.
Comparison of test execution with web based Test Runner and test execution with Microsoft Test Runner desktop client:
| Test execution capability | Web-based test features | Client-based Microsoft Test Manager |
|---|---|---|
| Bulk mark tests without opening in Test Runner | ✔️ | ✔️ |
| Pass or fail tests or test steps using Test Runner | ✔️ | ✔️ |
| Inline changes to tests during execution | ✔️ | ✔️ |
| Pause and resume tests | ✔️ | ✔️ |
| File bugs during test execution | ✔️ | ✔️ |
| Capture screenshots, image action log, and screen recording during test execution | ✔️ | ✔️ |
| Update existing bugs during test execution | ✔️ | ✔️ |
| Verify bugs | ✔️ | ✔️ |
| Assign a Build for the test run | ✔️ | ✔️ |
| Assign test settings | ✔️ | ✔️ |
The test runners offered through the web-based hub does not support Action Recording (fast-forward test steps), Advanced Data collection (code coverage, IntelliTrace, and test impact) during your tests and Connecting to a machine in an environment. These 3 capabilities were not leveraged by our customers much and hence not being taken forward. You may however use the Microsoft Test Runner client, launched from Azure Test Plans or the Test hub in TFS, for these requirements if really required.
Test tracking
You can easily track your manual testing results using your chosen light-weight chart types, and pin them to your dashboard to quickly analyze the test results. View test case result history across test suites and test plans easily by using the right-hand pane in the Test plans page of Azure Test Plans or the Test hub in TFS. You can also select a retention policy to specify how long you want to keep your manual testing results.
| Test tracking capability | Web-based test features | Client-based Microsoft Test Manager |
|---|---|---|
| Test run and results analysis | ✔️ | ✔️ |
| Create, configure, and pin light-weight test result charts | ✔️ | ❌ |
| Test run and results retention policy | ✔️ | ❌ |
| View test results history across test suites and test plans | ✔️ | ❌ |
Exploratory Testing
Use the lightweight Test & Feedback browser extension to perform exploratory testing on your web applications. You can collect rich diagnostic data such as screen captures, screen recording, and image action logs using this extension. The extension also has the capability to capture page load data for your web applications. In Azure Test Plans or the Test hub in TFS you can view completed exploratory testing sessions and derive meaningful insights at team or individual level, and for a specific period.
To explore your desktop applications, you can use the Exploratory Test Runner client in Microsoft Test Manager by launching it from Azure Test Plans or the Test hub in TFS.
Comparison of exploratory testing with the Test & Feedback extension and exploratory testing with Exploratory runner (client):
| Exploratory testing capability | Web-based extension for web apps | Client-based Exploratory runner for desktop apps |
|---|---|---|
| Explore user stories | ✔️ | ✔️ |
| File bugs using screen capture and recording, and image action log | ✔️ | ✔️ |
| Create test cases and tasks | ✔️ | ✔️ |
| Exploratory testing session insights | ✔️ | ✔️ |
| Capture page load performance data | ✔️ | ❌ |
Conclusion
Azure Test Plans or the Test hub in TFS is a richer, faster, and easier-to-use solution for manual testing as compared to the Test Center in Microsoft Test Manager. It works on all platforms and all browsers, and has a rich and modern web UI that improves your testing experience across all stages of manual testing.
All the test plans, test suites, test cases, results and other test management data you create using Microsoft Test Manager are stored in Azure DevOps or TFS. Hence, existing Microsoft Test Manager users can easily get started using Azure Test Plans or the Test hub in TFS without any data loss.
Help and support
Report any problems on Developer Community, get advice on Stack Overflow, and get support via our Support page.