VSTest@2 - Visual Studio Test v2 task

Use this task to run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test (VSTest) runner. You can run test frameworks that have a Visual Studio test adapter. Example frameworks are MSTest, xUnit, NUnit, Chutzpah (for JavaScript tests using QUnit, Mocha and Jasmine), etc. Tests can be distributed on multiple agents using this task.

Note

VSTest@3 is the newest version of the task and should be used in your pipelines.

Syntax

# Visual Studio Test v2
# Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test (VsTest) runner. Test frameworks that have a Visual Studio test adapter such as MsTest, xUnit, NUnit, Chutzpah (for JavaScript tests using QUnit, Mocha and Jasmine), etc. can be run. Tests can be distributed on multiple agents using this task (version 2).
- task: VSTest@2
  inputs:
  # Test selection
    testSelector: 'testAssemblies' # 'testAssemblies' | 'testPlan' | 'testRun'. Required. Select tests using. Default: testAssemblies.
    testAssemblyVer2: # string. Required when testSelector = testAssemblies. Test files. 
    #testPlan: # string. Required when testSelector = testPlan. Test plan. 
    #testSuite: # string. Required when testSelector = testPlan. Test suite. 
    #testConfiguration: # string. Required when testSelector = testPlan. Test configuration. 
    #tcmTestRun: '$(test.RunId)' # string. Optional. Use when testSelector = testRun. Test Run. Default: $(test.RunId).
    searchFolder: '$(System.DefaultWorkingDirectory)' # string. Required. Search folder. Default: $(System.DefaultWorkingDirectory).
    #resultsFolder: '$(Agent.TempDirectory)\TestResults' # string. Test results folder. Default: $(Agent.TempDirectory)\TestResults.
    #testFiltercriteria: # string. Optional. Use when testSelector = testAssemblies. Test filter criteria. 
    #runOnlyImpactedTests: False # boolean. Optional. Use when testSelector = testAssemblies. Run only impacted tests. Default: False.
    #runAllTestsAfterXBuilds: '50' # string. Optional. Use when testSelector = testAssemblies && runOnlyImpactedTests = true. Number of builds after which all tests should be run. Default: 50.
    #uiTests: false # boolean. Test mix contains UI tests. Default: false.
  # Execution options
    #vstestLocationMethod: 'version' # 'version' | 'location'. Select test platform using. Default: version.
    #vsTestVersion: 'latest' # 'latest' | '17.0' | '16.0' | '15.0' | '14.0' | 'toolsInstaller'. Optional. Use when vstestLocationMethod = version. Test platform version. Default: latest.
    #vstestLocation: # string. Optional. Use when vstestLocationMethod = location. Path to vstest.console.exe. 
    #runSettingsFile: # string. Settings file. 
    #overrideTestrunParameters: # string. Override test run parameters. 
    #pathtoCustomTestAdapters: # string. Path to custom test adapters. 
    #runInParallel: False # boolean. Run tests in parallel on multi-core machines. Default: False.
    #runTestsInIsolation: False # boolean. Run tests in isolation. Default: False.
    #codeCoverageEnabled: False # boolean. Code coverage enabled. Default: False.
    #otherConsoleOptions: # string. Other console options. 
    #diagnosticsEnabled: false # boolean. Collect advanced diagnostics in case of catastrophic failures. Default: false.
    #collectDumpOn: 'onAbortOnly' # 'onAbortOnly' | 'always' | 'never'. Optional. Use when diagnosticsEnabled = true. Collect process dump and attach to test run report. Default: onAbortOnly.
    #rerunFailedTests: False # boolean. Rerun failed tests. Default: False.
    #rerunType: 'basedOnTestFailurePercentage' # 'basedOnTestFailurePercentage' | 'basedOnTestFailureCount'. Optional. Use when rerunFailedTests = true. Do not rerun if test failures exceed specified threshold. Default: basedOnTestFailurePercentage.
    #rerunFailedThreshold: '30' # string. Optional. Use when rerunFailedTests = true && rerunType = basedOnTestFailurePercentage. % failure. Default: 30.
    #rerunFailedTestCasesMaxLimit: '5' # string. Optional. Use when rerunFailedTests = true && rerunType = basedOnTestFailureCount. # of failed tests. Default: 5.
    #rerunMaxAttempts: '3' # string. Optional. Use when rerunFailedTests = true. Maximum # of attempts. Default: 3.
  # Advanced execution options
    #distributionBatchType: 'basedOnTestCases' # 'basedOnTestCases' | 'basedOnExecutionTime' | 'basedOnAssembly'. Batch tests. Default: basedOnTestCases.
    #batchingBasedOnAgentsOption: 'autoBatchSize' # 'autoBatchSize' | 'customBatchSize'. Optional. Use when distributionBatchType = basedOnTestCases. Batch options. Default: autoBatchSize.
    #customBatchSizeValue: '10' # string. Required when distributionBatchType = basedOnTestCases && batchingBasedOnAgentsOption = customBatchSize. Number of tests per batch. Default: 10.
    #batchingBasedOnExecutionTimeOption: 'autoBatchSize' # 'autoBatchSize' | 'customTimeBatchSize'. Optional. Use when distributionBatchType = basedOnExecutionTime. Batch options. Default: autoBatchSize.
    #customRunTimePerBatchValue: '60' # string. Required when distributionBatchType = basedOnExecutionTime && batchingBasedOnExecutionTimeOption = customTimeBatchSize. Running time (sec) per batch. Default: 60.
    #dontDistribute: False # boolean. Replicate tests instead of distributing when multiple agents are used in the job. Default: False.
  # Reporting options
    #testRunTitle: # string. Test run title. 
    #platform: # string. Build platform. 
    #configuration: # string. Build configuration. 
    #publishRunAttachments: true # boolean. Upload test attachments. Default: true.
    #failOnMinTestsNotRun: False # boolean. Fail the task if a minimum number of tests are not run. Default: False.
    #minimumExpectedTests: '1' # string. Optional. Use when failOnMinTestsNotRun = true. Minimum # of tests. Default: 1.

Inputs

testSelector - Select tests using
string. Required. Allowed values: testAssemblies (Test assemblies), testPlan (Test plan), testRun (Test run). Default value: testAssemblies.

• Test assembly: Specifies one or more test assemblies that contain your tests. You can optionally specify a filter criteria to select only specific tests.
• Test plan: Runs tests from your test plan that have an automated test method associated with it. To learn more about how to associate tests with a test case work item, see Associate automated tests with test cases.
• Test run: Use this option when you are setting up an environment to run tests from test plans. This option should not be used when running tests in a continuous integration/continuous deployment (CI/CD) pipeline.

testAssemblyVer2 - Test files
string. Required when testSelector = testAssemblies. Default value: **\bin\**\*test.dll\n**\bin\**\*tests.dll.

Runs tests from the specified files. Ordered tests and webtests can be run by specifying the .orderedtest and .webtest files respectively. To run .webtest, Visual Studio 2017 Update 4 or higher is needed. The file paths are relative to the search folder. This input supports multiple lines of minimatch patterns.

# Example
- task: VSTest@2
  inputs:
    testSelector: 'testAssemblies'
    testAssemblyVer2: |
      **\*test*.dll
      !**\*TestAdapter.dll
      !**\obj\**

testPlan - Test plan
string. Required when testSelector = testPlan.

Specifies a test plan containing test suites with automated test cases.

testSuite - Test suite
string. Required when testSelector = testPlan.

Specifies one or more test suites containing automated test cases. Test case work items must be associated with an automated test method.

testConfiguration - Test configuration
string. Required when testSelector = testPlan.

Specifies the test configuration.

tcmTestRun - Test Run
string. Optional. Use when testSelector = testRun. Default value: $(test.RunId).

Specifies the test run-based selection that is used when triggering automated test runs from test plans. This option cannot be used for running tests in the CI/CD pipeline.

searchFolder - Search folder
string. Required. Default value: $(System.DefaultWorkingDirectory).

Specifies the folder to search for the test assemblies.

resultsFolder - Test results folder
string. Default value: $(Agent.TempDirectory)\TestResults.

Specifies the folder to store test results. When using the default directory, it is cleaned at the end of a pipeline run. The results directory will always be cleaned up at the start of the vstest task before the tests are run. The relative folder path, if provided, will be considered relative to $(Agent.TempDirectory).

testFiltercriteria - Test filter criteria
string. Optional. Use when testSelector = testAssemblies.

Specifies additional criteria to filter tests from test assemblies. For example: Priority=1|Name=MyTestMethod. Learn about command-line options.

runOnlyImpactedTests - Run only impacted tests
boolean. Optional. Use when testSelector = testAssemblies. Default value: False.

Automatically specifies and runs the tests needed to validate the code change. Learn about using Test Impact Analysis.

runAllTestsAfterXBuilds - Number of builds after which all tests should be run
string. Optional. Use when testSelector = testAssemblies && runOnlyImpactedTests = true. Default value: 50.

Specifies the number of builds to be executed before all tests are automatically run. Test Impact Analysis stores the mapping between test cases and source code. It is recommended to regenerate the mapping by running all tests on a regular basis.

uiTests - Test mix contains UI tests
boolean. Default value: false.

To run UI tests, ensure that the agent is set to run in interactive mode with Autologon enabled. Setting up an agent to run interactively must be done before queueing the build/release. Checking this box does not configure the agent in interactive mode automatically. This option serves as a reminder to configure the agent appropriately to avoid failures. Hosted Windows agents from the VS 2015 and 2017 pools can be used to run UI tests.

vstestLocationMethod - Select test platform using
string. Allowed values: version, location (Specific location). Default value: version.

Specifies which test platform to use.

vsTestVersion - Test platform version
string. Optional. Use when vstestLocationMethod = version. Allowed values: latest, 17.0 (Visual Studio 2022), 16.0 (Visual Studio 2019), 15.0 (Visual Studio 2017), 14.0 (Visual Studio 2015), toolsInstaller (Installed by Tools Installer). Default value: latest.

Specifies the version of Visual Studio Test to use. If latest is specified, this input chooses the latest version (from the list of allowed values) that is installed. To run tests without needing Visual Studio on the agent, use the Installed by tools installer option. Be sure to include the Visual Studio Test Platform Installer task to acquire the test platform from NuGet.

vstestLocation - Path to vstest.console.exe
string. Optional. Use when vstestLocationMethod = location.

Specifies the path to VSTest.

runSettingsFile - Settings file
string.

Specifies the path to a runsettings or testsettings file to use with the tests. For Visual Studio 15.7 and higher, use runsettings for all test types. Learn more about converting a .testsettings file to a .runsettings file.

overrideTestrunParameters - Override test run parameters
string.

Overrides the parameters defined in the TestRunParameters section of a runsettings file or the Properties section of a testsettings file. For example: -key1 value1 -key2 value2. Note: Properties specified in a testsettings file can be accessed via the TestContext using Visual Studio 2017 (update 4 or higher).

pathtoCustomTestAdapters - Path to custom test adapters
string.

Specifies the directory path to custom test adapters. Adapters residing in the same folder as the test assemblies are automatically discovered.

runInParallel - Run tests in parallel on multi-core machines
boolean. Default value: False.

If set to true, tests are run in parallel and leverage available cores of the machine. This will override the MaxCpuCount if specified in your runsettings file. Learn more about how tests are run in parallel.

runTestsInIsolation - Run tests in isolation
boolean. Default value: False.

Runs the tests in an isolated process. This likely leads to fewer errors in the vstest.console.exe test process, but tests might run slower. This option currently cannot be used when running with the multi-agent job setting.

codeCoverageEnabled - Code coverage enabled
boolean. Default value: False.

Collects code coverage information from the test run.

otherConsoleOptions - Other console options
string.

Other console options that can be passed to vstest.console.exe.

These options are not supported and will be ignored when running tests using the Multi-agent parallel setting of an agent job, when running tests using the Test plan or Test run option, or when a custom batching option is selected. The options can be specified using a settings file instead.

distributionBatchType - Batch tests
string. Allowed values: basedOnTestCases (Based on number of tests and agents), basedOnExecutionTime (Based on past running time of tests), basedOnAssembly (Based on test assemblies). Default value: basedOnTestCases.

A batch is a group of tests. A batch of tests runs its tests at the same time, and results are published for the batch. If the job in which the task runs is set to use multiple agents, each agent picks up any available batches of tests to run in parallel. A batch can be run:

based on the number of tests and agents. Simple batching based on the number of tests and agents participating in the test run.

based on the past running time of tests. This batching considers the past running time to create batches of tests where each batch has approximately equal running time.

based on test assemblies. Tests from an assembly are batched together.

batchingBasedOnAgentsOption - Batch options
string. Optional. Use when distributionBatchType = basedOnTestCases. Allowed values: autoBatchSize (Automatically determine the batch size), customBatchSize (Specify a batch size). Default value: autoBatchSize.

Specifies simple batching based on the number of tests and agents participating in the test run. When the batch size is automatically determined, each batch contains (total number of tests / number of agents) tests. If a batch size is specified, each batch will contain the specified number of tests.

customBatchSizeValue - Number of tests per batch
string. Required when distributionBatchType = basedOnTestCases && batchingBasedOnAgentsOption = customBatchSize. Default value: 10.

Specifies the batch size.

batchingBasedOnExecutionTimeOption - Batch options
string. Optional. Use when distributionBatchType = basedOnExecutionTime. Allowed values: autoBatchSize (Automatically determine the batch time), customTimeBatchSize (Specify running time per batch). Default value: autoBatchSize.

This batching considers past running times to create batches of tests where each batch has approximately equal running time. Quick-running tests will be batched together, while longer-running tests may belong to a separate batch. When this option is used with the multi-agent job setting, the total test time is reduced to a minimum.

customRunTimePerBatchValue - Running time (sec) per batch
string. Required when distributionBatchType = basedOnExecutionTime && batchingBasedOnExecutionTimeOption = customTimeBatchSize. Default value: 60.

Specifies the running time (in seconds) per batch.

dontDistribute - Replicate tests instead of distributing when multiple agents are used in the job
boolean. Default value: False.

Choosing this option will not distribute tests across agents when the task is running in a multi-agent job. Each of the selected test(s) will be repeated on each agent. This option is not applicable when the agent job is configured to run with no parallelism or with the multi-config option.

testRunTitle - Test run title
string.

Specifies a name for the test run.

platform - Build platform
string.

Specifies the build platform against which the tests should be reported. If you have defined a variable for the platform in your build task, use that with this input.

configuration - Build configuration
string.

Specifies the build configuration against which the tests should be reported. If you have defined a variable for configuration in your build task, use that with this input.

publishRunAttachments - Upload test attachments
boolean. Default value: true.

Opts in or out of publishing run level attachments.

failOnMinTestsNotRun - Fail the task if a minimum number of tests are not run.
boolean. Default value: False.

Fails the task if a minimum number of tests are not run. This may be useful if any changes to task inputs or underlying test adapter dependencies lead to only a subset of the desired tests to be found.

minimumExpectedTests - Minimum # of tests
string. Optional. Use when failOnMinTestsNotRun = true. Default value: 1.

Specifies the minimum number of tests to run for the task to succeed. The total tests executed is calculated as the sum of passed, failed and aborted tests.

diagnosticsEnabled - Collect advanced diagnostics in case of catastrophic failures
boolean. Default value: false.

Collects diagnostic data to troubleshoot catastrophic failures, such as a test crash. When this option is checked, a sequence XML file is generated and attached to the test run. The sequence file contains information about the sequence in which the tests had run, so a potential culprit test can be identified.

collectDumpOn - Collect process dump and attach to test run report
string. Optional. Use when diagnosticsEnabled = true. Allowed values: onAbortOnly (On abort only), always, never. Default value: onAbortOnly.

Collects a mini dump that can be used for further analysis.

• onAbortOnly - a mini dump will be collected only when the test run is aborted.
• Always - a mini dump will always be collected regardless of whether the test run completes or not.
• Never - a mini dump will not be collected regardless of whether the test run completes or not.

rerunFailedTests - Rerun failed tests
boolean. Default value: False.

Reruns any failed tests until they pass or until the maximum number of attempts is reached.

rerunType - Do not rerun if test failures exceed specified threshold
string. Optional. Use when rerunFailedTests = true. Allowed values: basedOnTestFailurePercentage (% failure), basedOnTestFailureCount (# of failed tests). Default value: basedOnTestFailurePercentage.

Avoids rerunning tests when the failure rate crosses the specified threshold. This is applicable if environment issues lead to massive failures. You can specify the percentage of failures or number of failed tests as a threshold.

rerunFailedThreshold - % failure
string. Optional. Use when rerunFailedTests = true && rerunType = basedOnTestFailurePercentage. Default value: 30.

Avoids rerunning tests when the percentage of failed test cases crosses the specified threshold. This is applicable if environment issues lead to massive failures.

rerunFailedTestCasesMaxLimit - # of failed tests
string. Optional. Use when rerunFailedTests = true && rerunType = basedOnTestFailureCount. Default value: 5.

Avoids rerunning tests when the number of failed test cases crosses the specified limit. This is applicable if environment issues lead to massive failures.

rerunMaxAttempts - Maximum # of attempts
string. Optional. Use when rerunFailedTests = true. Default value: 3.

Specifies the maximum number of times a failed test should be retried. If a test passes before the maximum number of attempts is reached, it will not be rerun again.

Task control options

All tasks have control options in addition to their task inputs. For more information, see Control options and common task properties.

Output variables

None.

Remarks

Use this task to run unit and functional tests (Selenium, Appium, Coded UI test, and more) using the Visual Studio Test runner. Along with MSTest-based tests, test frameworks that have a Visual Studio test adapter can also be executed, such as xUnit, NUnit, or Chutzpah.

Tests that the target .NET core framework can be executed by specifying the appropriate target framework value in the .runsettings file.

Tests can be distributed on multiple agents using version 2 of this task. For more information, see Run tests in parallel using the Visual Studio Test task.

Check prerequisites

If you're using a Windows self-hosted agent, this prerequisite must be installed:

• .NET Framework 4.6.2 or a later version

Demands

The agent must have the following capability:

vstest

The vstest demand can be satisfied in two ways:

1. Visual Studio is installed on the agent machine.
2. By using the Visual Studio Test Platform Installer task in the pipeline definition.

How can I run tests that use TestCase as a data source?

To run automated tests that use TestCase as a data source, the following is needed:

1. You must have Visual Studio 2017.6 or higher on the agent machine. Visual Studio Test Platform Installer task cannot be used to run tests that use TestCase as a data source.
2. Create a PAT that is authorized for the scope “Work Items (full)”.
3. Add a secure build or release variable called Test.TestCaseAccessToken with the value set to the PAT created in the previous step.

I am running into issues when running data-driven xUnit and NUnit tests with some of the task options. Are there known limitations?

Data-driven tests that use xUnit and NUnit test frameworks have some known limitations and cannot be used with the following task options:

1. Rerun failed tests.
2. Distributing tests on multiple agents and batching options.
3. Test Impact Analysis.

The above limitations are because of how the adapters for these test frameworks discover and report data-driven tests.

Does the VSTest task support running tests that target multiple target frameworks at a time?

Yes, starting from version 17.3 VSTest does support running tests that target multiple target frameworks at a time. Prior to that, this wasn't possible due to a limitation from the VSTest platform side.

If you want to run tests that belong to multiple target frameworks, you'll need to install a compatible version of VSTest via Visual Studio Test Platform Installer and set vsTestVersion to toolsInstaller to use it.

While publishing the test result, getting this error: Failed to publish test results: Invalid Priority specified?

This error occur if any of the test methods has priority set above 255, fix the test method priority in the code and execute the tests again. You can review the trx file generated to see all the tests having priority greater than 255.

Requirements

Requirement	Description
Pipeline types	YAML, Classic build, Classic release
Runs on	Agent, DeploymentGroup
Demands	Self-hosted agents must have capabilities that match the following demands to run jobs that use this task: vstest
Capabilities	This task does not satisfy any demands for subsequent tasks in the job.
Command restrictions	Any
Settable variables	Any
Agent version	2.103.0 or greater
Task category	Test