FAQs for Test Case Management reporting

Makale
01/18/2019

TFS 2017 | TFS 2015

Visual Studio 2019 | Visual Studio 2017 | Visual Studio 2015

I am using Microsoft Test Manager (MTM) for my testing activities - what type of reports can I see within MTM?

You can view basic reporting information within Microsoft Test Manager:

Test Run status at a Test Plan level:

This provides a status of all the tests that have been run, along with their current outcomes, within that specific test plan. This data is aggregated across all suites within the Test Plan. You can view this from the Test Plan Properties page.
Test Run status at a Test Suite Level:

This provides a status of all the tests that have been run, along with their current outcomes, within that specific test suite. This information can be seen on both the Test Activity Center (as shown below) or Plan Activity Center.

Note

The data shown at a suite level includes the tests that are explicitly within that suite node only - it does not include all the child suites within that suite.

I would like to create reports in Excel using Work Item Queries, but I cannot do so for all the Test artifacts such as Test Suites, Test Runs, etc.? Why?

Create reports in Microsoft Excel by using Work Item Queries

You can use the above method to query for any TFS "Work Items". So, you should be able to create a report for "Test Case", "Bug", "Task" work items.

However, all the other test artifacts, such as test configurations, test points, or test runs, etc., are not "work items". Hence, you cannot use the work item queries for them.

clip_image003

All the test artifacts which are enclosed within the black box (in above diagram) are not work items but they are stored directly within the Test Case Management store.

The reporting information shown within MTM is basic in nature - are there any other Test reports that ship out-of-the-box that I can use?

Most of the out-of-the-box TFS reports, including Test-related reports, can be accessed from within Team Explorer.

There are two types of reports that can be viewed from Team Explorer - Excel Reports and SQL Reporting Services Reports.

Excel Reports: These reports use Excel to connect to the TFS warehouse or Analysis cube to display the reporting data. The figure below shows the list of all Excel reports that ship out of the box. Depending upon the process template (Agile, CMMI) you used at the time of creation of the project, the respective reports are created at project creation time.

All the test-specific reports are highlighted in the above diagram.

Details of each of the Test-related reports can be found below:
Reporting Services based Reports: The figure below shows the list of all Reporting Services based reports that ship out of the box. Depending upon the process template (Agile, CMMI) you used at the time of creation of the project, the respective reports are created at project creation time.

All the test-specific reports are highlighted in the above diagram.

Details of each of the Test-related reports can be found below:

I am connected to my Team Project within Team Explorer, but I don't see any of the Reporting Services based Reports listed above. Why?

If you want to use the reporting feature (Excel Reports or Reporting Services Reports) in Team Foundation Server, you must have an installation of SQL Server Reporting Services and SQL Server Analysis Services. These features can be installed on the same server or different servers. Team Foundation Server requires a report server that is dedicated only to its use. You cannot share the report server with any other application.

To learn more, see the following resources:

I am connected to my Team Project within Team Explorer, but I don't see any of the Excel Reports listed above. Why?

Team Foundation Server requires a report server that is dedicated only to its use. You cannot share the report server with any other application.

In addition, for Excel Reports, you also need to enable SharePoint on the Team Foundation Server because all the Excel reports, by default, get copied and stored in the SharePoint Document Library at the project creation time.

Links:

I would like to create custom test reports in Excel based off the Analysis Services cube. How should I go about it?

There are two ways of doing this, start from scratch or use one of the existing out-of-the-box Excel reports as a starting point.

Start from Scratch

You can always start creating a new Excel report by adding a new data source, as mentioned below:
1. From "Data" tab, click on "From Other Sources->From Analysis Services" option
2. Provide the name of your TFS server.
3. This should connect you to the TFS_Analysis database. If your Azure DevOps data warehouse uses SQL Server Enterprise Edition, the list of cubes will include Team System and a set of perspectives. The perspectives provide a focused view of the data so that you do not have to scroll through all of the dimensions and measure groups that are defined for the whole Team System cube. Choose the CUBE or the TEST perspective to create test-related reports.
4. Save the connection information locally or it can also be saved on a SharePoint.
5. Select where you want to create the PivotTable Report or PivotChart
6. Drag and drop various dimensions and measures from Pivot table field list to construct your custom report.
Use existing out-of-the-box Excel reports as a starting point

You could also start with any of the existing out-of-the-box Excel reports and then customize it to suit your needs by dragging and dropping the different measures or dimensions.

The simplicity of Excel reports (of just using drag and drop of dimensions or measures) is one of the main reasons why you will see a lot more out-of-the-box reports for Excel rather than Reporting Services based reports.

All the test-related measures present in the Analysis Services cube are shown below:

clip_image016

Description for each of above measures can be found here: Analyze and report on test results using the test perspective

A list of all the perspectives and measure groups present in Team Foundation Server can be found here: Perspectives and measure groups

All the dimensions present in the Analysis Services cube are shown below:

clip_image017

Description for all the Test-specific dimensions can be found here:

Analyze and report on test results using the test perspective

Can I control what information makes it into the warehouse or cube? If so, how?

Here is a quick primer on the different components of data warehouse and how data moves from the Operation store to Warehouse to Cube using the warehouse adapters: Components of the data warehouse for Team Foundation.

Yes, you can control what goes into the warehouse - refer to the following articles for details around work item specific fields:

Adding and Modifying Work Item Fields to Support Reporting

Note that you can use the above to control Test Case Work item field customizations. However, since the other test artifacts (Test Suites, Test Runs, Test Results etc.) are not work items, it is not possible to use the above approach.

I don't see any way to report on what steps were passed or failed during manual testing using the warehouse or cube. How can I achieve this?

That is correct - Test Steps along with expected results and the passed or failed information is not present in the warehouse or cube. The entire Test Steps (including Step and Expected Result) is a HTML based custom control, hence this information is not pushed into the warehouse. As a result you cannot create reports for the same using the cube.

However, there are couple of ways of getting this Step related data:

TFS APIs: You could get this information using the Microsoft.TeamFoundation.TestManagement.Client Namespace Refer to a sample Test Steps Report, which was created using the TFS APIs.

http://geekswithblogs.net/TarunArora/archive/2011/10/02/mtm-testing-scorecard-using-tfs-api.aspx

Since this sample also has source code, you could customize it to suit your organization needs.
Test Scribe: You could also use Test Scribe tool to generate a document containing Test Steps. There are two modes of generating the document using this tool -
- "Test Plan Summary" view which you can use to review or sign-off the Test Plan and its associated test cases along with individual test steps or expected results or iteration values.
- "Test Run Summary" view which you can use to get detailed Run level information - however it does not provide a breakdown of individual test results within a particular test run.