During our first release of this new workflow in Team Foundation Server 2015 and Visual Studio Team Services we consciously left out some functionality that wasn’t complete for the release. One of these was the support for ordered tests. While we are working hard to get this scenario lit up for the next release we have heard enough requests to provide a workaround.

So what is the workaround?

You can copy the ordered test adapter for vstest to the right directory on your test agent and the scenario will light up.

If you are using the BDT build template then add a remote file copy task before your "Run Functional Tests" task.

In the copy files task set the source as <VSIntallDir>\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions\Microsoft.VisualStudio.TestPlatform.Extensions.OrderedTestAdapter.dll and destination as <TestAgentInstallDir>\Common7\IDE\CommonExtensions\Microsoft\TestWindow\Extensions\Microsoft.VisualStudio.TestPlatform.Extensions.OrderedTestAdapter.dll The <TestAgentInstallDir> would be in the %systemdrive%\Program Files (x86)\Microsoft Visual Studio 14.0 directory

In the Run Function Tests task you can now specify **\*orderedtest.dll as the test filter.

One of the issues that we noticed with this approach is that ordered test failures are not registered as failures in the test run. To overcome this

1. Enable oauth access to the agent in the build definition -> options
2. Place the ps1 script in a location accessible to the build agent ( https://github.com/Microsoft/customer-scripts/blob/master/test/build/orderedtest/throwif-trxfromtestrun-ofbuild-hasfailedoutcome.ps1 )
3. Add a powershell task after the run task and provide the path to the ps1 in this task
4. The ps1 will error out if the ordered test has failed

The script checks the latest trx file associated with the test run associated with the current build and throws an error if the trx specifies the test run has a failed outcome. Feel free to contribute back if you find any issues in the script.

Some points to note

1. Given that an ordered test is single unit of execution it wont be distributed across machines, if you specify multiple machines in the machines input.
2. The reporting shows one entry per ordered test. You can drill down further by downloading the .trx file which is attached to the test run
3. Ordered tests within a test suite are not supported

The Agents for Visual Studio suite containing the test controller and agents allowed you to execute your tests on a set of remote machines by providing the right configuration in your .testsettings file. Based on the feedback we have received we are providing a new and simpler way for you to do your remote automated testing starting Team Foundation Server (TFS) 2015 RC2.

In this blog post we will quickly setup a test rig for remote test execution using TFS 2015 RC2

Pre-requisites

1. TFS 2015 RC2 installation with a Build vNext agent configured on a machine with internet access.
2. A (virtual) machine to be used for testing with following requirements

Step-By-Step Guide

Head to your TFS homepage and navigate to the team project you want to work with. The below slides will guide you through this entire scenario.

NOTE: The "Run Tests using Test Agent" task is renamed to "Run Functional Tests"

So what has changed

There have been a lot of changes to get the stability and resiliency that you demanded in test execution. To summarize the top few improvements

1. The test controller has been removed as the intermediate hop and the agents now directly communicate with the TFS server using HTTP.
2. You can now run tests using 3^rd party test frameworks like nunit/xunit/cpp using the relevant unit test adapter, similar to Visual Studio.
3. Automated installation and configuration of your test agent on your test machines as a part of the Build workflow using our tasks in Build vNext.
4. Improved reporting for automated runs in Test Management.

What are our limitations

1. We do distribution on a per test container basis ie if you have 2 machines and 2 test dlls we will execute all tests from testdll1 on machine1 and the rest from testdll2 on machine2.
2. We don’t have support for workgroup/isolated/azure machines as yet. As long as you have http connectivity from the agent to TFS/Visual Studio Team Services and your build machine you will be good to go.
3. We do not support Azure VMs as machines in the machine group used for testing
4. Yes the scenario can be used in TFS/Visual Studio Team Services as well. However we do not support hosted build pools when using on-prem machines. For this case you would have to create a new build pool and register a new build agent with it.

 More up-to-date details @ https://aka.ms/remotevstest

FAQ

Why do we have a test agent and a build agent? Why can’t there just be a common agent?

The build agent does provide you with unit testing facilities but only after you install Visual Studio on your build machine. The test agent has no such prerequisite. This is the sole reason why we have two agents instead of one. We are definitely looking at ways in which we no longer need to make such a distinction.

Will I be able to run my java tests on my remote Linux/OSX machines using the test agent?

Today the cross platform functionality is available in Build vNext. We are looking for the right data points before we build the remote testing scenario for other platforms.

We hope that this post will allow you to get started with the new remote test execution workflow on your on-prem TFS installation. Try it out and let us know what you think devops_tools@microsoft.com.

Increase the timeout setting for Agent timeout:

[Controller machine] In QTController.exe.config: <drive letter:>\Program Files (x86)\Microsoft Visual Studio <Visual Studio Version>\Common7\IDE\.

    <add key="AgentConnectionTimeoutInSeconds" value="120"/>

    <add key=" AgentConnectionTimeoutInSeconds" value="120"/>

    <add key="AgentDeploymentTimeout" value="300"/>

    <add key="AgentSyncTimeoutInSeconds " value="300"/>

Increase the value to triple.

[Agent machine] In QTAgentService.exe.config <drive letter:>\Program Files (x86)\Microsoft Visual Studio <Visual Studio Version>\Common7\IDE

                    Set ControllerConnectionPeriodInSeconds to 90 (making this 3 times too).

Restart the services.

Apart from this increasing the bucket size will help reducing communication too.

https://www.peterprovost.org/blog/2012/04/15/visual-studio-11-fakes-part-1/

https://www.peterprovost.org/blog/2012/04/25/visual-studio-11-fakes-part-2/

https://www.peterprovost.org/blog/2012/11/29/visual-studio-2012-fakes-part-3/

Setup and Settings

If you are on the latest VS2012U1+ or VS2013 you can ignore step#1&2

1. Ensure that Visual Studio 2010 SP1 is installed on all machines

Dev10 SP1 - https://www.microsoft.com/en-us/download/details.aspx?id=23691

2. Next install the latest patch available

https://support.microsoft.com/kb/2801364

3. Open "<VSINSTALLDIR>\Common7\IDE\QTControllerService.exe.config" and add the following entry under

<add key="ControllerJobSpooling" value="false"/>

• Restart Controller
4. Agent setup
• Install the latest patch available
• Open "<VSINSTALLDIR>\Common7\IDE\QTAgentService.exe.config" and add the following

<add key="RestartTestExecutionProcessForEachRun" value="true"/>

• Restart Agent
5. Client Setup
• Install the latest patch available
• Open "<VSINSTALLDIR>\common7\ide\mstest.exe.config and add the following key

<add key="DeleteTestDeploymentFiles" value="yes" />

6. In your test settings
• Keep bucket size as 1000. This will result in a test run going to smaller set of agents.

<Execution>

    <Buckets size="1000"/>

7. Set test run timeouts to ensure no runs go beyond acceptable limits

<Execution>

<Timeouts runTimeout="7200000" />

7200000 = 2hrs specified in milliseconds

8. Run your tests in parallel if your agents are on a multiproc system

<Execution parallelTestCount=0>

https://blogs.msdn.com/b/vstsqualitytools/archive/2009/12/01/executing-unit-tests-in-parallel-on-a-multi-cpu-core-machine.aspx

Other Recommendations

1. Test Controller keeps all results for a run in memory. Therefore it is necessary that test runs be split into logical groups. We recommend you split tests into assemblies so that each test dll has <=5000 tests and/or run the TC in 64bit mode

testcontrollerconfig.exe configure <… configuration parameters… > /platform64

2. When any component of the Test Rig (any of the agents or controller) faces an issues (network disconnect, deployment failures, test crashes), the current test run is aborted. The probability of the test run getting aborted due to failure in agents reduces when we send one test run to a smaller set of agents. Keep a run limited to minimum number of test assemblies
3. To scale out you should be using multiple such rigs and breaking up your test runs amongst these rigs as required.

With Visual Studio 2013 you can now check the status of your agents on the controller itself using the commandline

Syntax-
TestControllerConfig.exe status [/testController:<testControllerUri>]

testController              URI of the test controller. Default is localhost:6901

Sample output

Microsoft (R) Visual Studio Test Controller Configuration Tool
Version 12.0…. for Microsoft Visual Studio 2013
Copyright (c) Microsoft Corporation.  All rights reserved.

Total number of agents         : 1

Status of agents :
Agent name                     : vstfs:///…
Agent status                   : Disconnected

Summary of all agents :
Ready                          : 0
Running tests                  : 0
Offline                        : 0
Deploying build                : 0
Disconnected                   : 1

The agent status correspond to the existing known states listed here

1. Visual Studio Test Controller 2012 Update3+ now supports backward compat with TFS servers. So you can connect your 2013 test controller to TFS 2012+
2. The test controller now works with hosted builds having server drop locations

​If you find yourself with a an empty .coverage file and see errors similar to the below in your event logs you most probably have a corrupt install

(info) .NET Runtime version 4.0.30319.17929 - The profiler has requested that the CLR instance not load the profiler into this process. Profiler CLSID: '{b19f184a-cc62-4137-9a6f-af0f91730165}'. Process ID (decimal): 12624. Message ID: [0x2516].

(Error) TraceLog Profiler failed in initialization due to a lack of instrumentation methods, process vstest.executionengine.x86.exe

Check

a) Environment variable VS110COMNTOOLS is set to <vsinstalldir>\common7\tools

b) Regkey HKLM\SOFTWARE\Microsoft\VisualStudio\11.0\InstallDir is set to your <vsinstalldir>\Common7\IDE\

c) covrun32.dll and covrun64.dll exist in "<vsinstalldir>\Team Tools\Dynamic Code Coverage"

At the end of the Code Coverage run you see code coverage within the Code Coverage results window. The total code coverage as well as code coverage for each binary are reported.

However in some cases you may end up with a .coverage file which shows 0% coverage in the Code Coverage Results window with an error similar to “Empty results generated: No binaries were instrumented …”

This means code coverage results were not obtained for any binary.

In this article we will attempt to list down the most common reasons for such a problem and provide you with resolutions for the same.

1. No tests were executed
2. PDBs (symbol files) are unavailable/missing
3. Using an instrumented or optimized binary
4. Code executed is not managed (.NET) or native (cpp) code
5. A custom .runsettings file with exclusions is being used

Further Analysis

To know why a specific DLL was not included by code coverage, you can analyze the .coverage file generated further using our command line utility, CodeCoverage.exe

Cd “<VSInstallDir>\Team Tools\Dynamic Code Coverage Tools”

CodeCoverage.exe analyze /include_skipped_modules my.coverage > analysis.xml

my.coverage is the .coverage file generated for your test run found in the TestResults folder.

Here /include_skipped_modules will provide information regarding each and every module that was inspected by the Code Coverage engine and provide a reason why it was skipped. These reason codes are to be interpreted as below:

If the output file is empty, see “No tests were executed”