Creating an Automated Test That Runs an Executable Using Generic Tests

Article
09/29/2016

You use generic tests to call external programs and tests. After you have done this, the generic test is treated by the test engine as any other test type. For example, you can run generic tests from Test Explorer and you can obtain and publish results from generic tests just as you do from other tests.

Use a generic test to wrap an existing test, program, or third-party tool that behaves as follows:

It can be run from a command line.
It returns a value of Pass or Fail.
Optionally, it also returns detailed results for 'inner' tests, which are tests that it contains.

Visual Studio treats generic tests just like other tests. You can manage and run them using the same views, and you can obtain and publish their results.

Generic tests are a simple form of extensibility for Visual Studio. They let you run other tests, such as previous tests and custom tests, in addition to the predefined test types, which are Web, load, unit, manual, and ordered.

Requirements

Visual Studio Enterprise, Visual Studio Test Professional

In this topic:

Tasks
How to Create a Generic Test
Generic Test Command-Line Arguments
Deploying Files for Generic Tests

Tasks

Use the following topics to help you in creating generic tests:

Key Tasks	- Associated Topics
Creating a generic test: You can create a generic test by using the generic test template provided in Visual Studio and by specifying the program that you want to call. For more information, seeHow to Create a Generic Test in this topic.	- Walkthrough: Creating and Running a Generic Test
Using environment variables in generic tests: You can use environment variables to represent paths to folders that contain files including your target executable and items to deploy.	- Using Environment Variables in Generic Tests
Run generic tests: You can run the generic tests that you have created from Unit Test Explorer, or by using test categories from the command line.	- Walkthrough: Creating and Running a Generic Test - How to: Run Tests from Microsoft Visual Studio - Running automated tests from the command line - Defining Test Categories to Group Your Tests
Determining code coverage: To know how much you are improving the quality of your code, you must determine how much of the code your generic tests are covering when you run them.	- Specifying Test Settings for Visual Studio Tests - Code Coverage configuration using Test Settings is deprecated
Create and use a summary results file: Through the use of a summary results file, a generic test can generate specific, detailed, test results.	- Using a Summary Results File with a Generic Test

How to Create a Generic Test

Note

The following procedure presumes that you have an existing test, program, or third-party tool in the form of executable file to wrap as a generic test. You must be able to run this executable file from the command line, and it must return a value of Pass or Fail.

To create a generic test

In Solution Explorer, open the shortcut menu for either a unit test project or a coded UI test project, point to Add, and then choose Generic Test.

A template for a generic test is added to your test project. It appears in the main editing window. The new generic test is given a default name, such as GenericTest1.generic, and is displayed in Solution Explorer within the test project.
Under Specify an existing program (a test, test harness, or test adapter) to wrap as a generic test, indicate the path and file name of the test, program, or third-party tool to wrap into a generic test.
(Optional) Under Command line arguments to pass to the generic test, type one or more arguments to pass. Separate multiple arguments with spaces. For more information, seeGeneric Test Command-Line Arguments in this topic.
(Optional) Under Additional files to deploy with this generic test, specify any files that the test must have to run correctly.

Note

Before the test runs, the files that you specify in this step are copied to a deployment directory that is created by Visual Studio.

For more information, see Deploying Files for Generic Tests in this topic.
(Optional) Under Working Directory, specify a directory that the executable file is to use as a working directory as it runs.
(Optional) Specify the name of a results file.
Save the generic test.

Tip

You can create and use an XML based summary results file to provide detailed results for portions of you generic test. For more information, seeUsing a Summary Results File with a Generic Test.

The test is now ready to run; it will return a result of 0 or another number. The test engine interprets 0 as Passed and any other number as Failed.

Generic Test Command-Line Arguments

You can pass command-line arguments to the program that your generic test wraps.

For example, to pass a single argument, simply type the argument in the box indicated by Command-line arguments to pass to the generic test, in the generic test.

To pass multiple arguments, type them in the Command-line arguments to pass to the generic test box, separated by spaces. For example, type: 12 "%TestDeploymentDir%\mydeployedfile.txt".

You can see this same example in context in Walkthrough: Creating and Running a Generic Test.

Deploying Files for Generic Tests

If either your generic test or the executable file that it wraps need additional supporting files when they run, you can deploy these files before the test is run.

This is shown in the Deploy a File When You Run the Generic Test procedure in Walkthrough: Creating and Running a Generic Test.

To deploy files for a generic test

In Solution Explorer, open a test project that contains your generic test.
Under Additional files to deploy with this generic test, specify any files that the test must have to run correctly.

Note

Before the test runs, the files that you specify in this step are copied to a deployment directory that is created by Visual Studio.
Save the generic test.

Security

Not only can a generic test call any executable program on your computer, it uses your credentials as it does so. For this reason, make sure that you trust the author of the program and know what the program will do.