What is it ?

new command line test runner “vstest,console.exe” which would run tests written against any test framework – MSTest framework, Nunit, Xunit and so on.

Where can I find it ?

vstest.console.exe could be invoked from directly from “Visual Studio Command Prompt”. The executable would be found under “Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow” folder. Vstest.console.exe /? would give more details on the various switches provided by the test runner.

How is it different from Mstest.exe ?

With Visual Studio 2011, the tests written using visual studio unit test framework could be run using either vstest,console.exe or using mstest.exe. The behavior would be same with subtle differences (which is briefed in the below table).

vstest.console.exe has some additional options /platform /framework which would let the user to run their tests in a specific platform/framework to increase test coverage.

Mstest and Vstest:

Below is a brief summary on the various switches provided by Mstest.exe and the equivalent switches in vstest.console.exe

Visual studio 11 Beta unit test framework provides a way to plug-in third party test runners and run their tests right inside VS IDE. Here are the list of test adapters that are already available for download.

In this post, let's quickly go through the steps involved in writing a new test adapter.

Components of a test adapter:

A test adapter plug-in needs to have two basic components :

• TestDiscoverer - A component that gives you the list of test cases from your test source. The test source could be a .dll file or a .xml or anything. Given a test source a discoverer knows how to load and get the test case out of it.
• TestExecutor -A component that executes the tests. Given a list of tests, the executor executes them and gives back the results.

To design these components the VS Unit testing extensibility model provides you with two interfaces :

a. ITestDiscoverer

b. ITestExecutor

.

To implement these interfaces all you need to do is to create a class library and add a reference to "Microsoft.VisualStudio.TestPlatform.ObjectModel "assembly which would be found under "%Program Files%\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow" folder.

Xml Test Adapter:

Let us walk through the implementation of a simple XML test adapter which would run tests in a given XML file. [Sample code attached with the post].

ITestDiscoverer:

ITestDiscoverer defines just one method : "DiscoverTests". Given a list of test sources, this method would pull out the test cases from them.

If you take look at the parameters :

1. Sources --> Refers to the list of test sources that are passed to the test adapter from the client [VS or command line]. It would be a list of .xml files in our case.
2. Discoverycontext --> Refers to the discovery context/runsettings for the current test run. Discoverer shall pull out the tests in a specific way based on the current context.
3. Logger --> This is used to relay the warning and error messages to the registered loggers. Console Logger and TRXLogger are the built-in loggers that ships with Visual Studio 11. User shall register other custom loggers like FileStreamLogger or DBLogger as required.[ more details on writing custom loggers in future posts].
4. DiscoverySink --> Discovery sink is used to send back the test cases to the framework as and when they are being discovered. Also it is responsible for the discovery related events.

Discoverer Attributes:

The discoverer has two attributes associated with it:

FileExtension - Specifies one or more file extensions that the discoverer supports. It would be ".xml" in our case.

Executorurl - specifies the Url of the executor that is tied to the discoverer. The discover to executor mapping is 1:1.

Xml discoverer:

Lets quickly look at our sample XML discoverer. It scans the given XML source and identifies each "test case" based on XML Node attribute "Name".

It associates the executor (XMLExeuctor in our case) with each test case and sends the test back to the framework using discovery sink.

…..

…...

ITestExecutor :

The ITestExecutor interface provides you with two simple overloaded methods to run tests :

1. RunTests ( given the sources)

This could be something which maps to the "RunAll" functionality. All the sources are given to the executor in one go and executor defines how to run them.

2. RunTests ( given a test collection)

This maps to "RunSelected" test functionality, where a bunch of test cases are passed to the executor.

The runContext object and frameworkHandle object are passed are parameters to RunTests method, which provides all the settings related to the current run.

XmlExecutor

Our XML executor just iterates through the given tests and sets the test results based the user specified "Outcome" value in the XML.

In this case RunTest(sources) overload  just calls the discoverer and calls RunTests(tests) overload. This could vary in other test adapters – say to have some optimizations for RunTests(sources).

Install test adapter and run tests :

Now that we are done with implementing the test adapter, we shall go ahead and register them. You shall register them using the steps mentioned here in Aseem's blog. Method 2 disucssed over there via VSIX is the generally recommended way to install extensions.

To quickly check if the test adapter has been registered properly, you can try "vstest.console.exe /listdiscoverers" from a Visual Studio command prompt. [vstest.cosnsole.exe is the new framework agnostic test runner.More details here on MSDN].

If you have installed your adapter as a VSIX make sure that you specify /useVsixExtensions:True switch in your command. The below image shows the XMLDiscoverer being listed along with the other built-in test discoverers.

Similar to /listdiscoverers, we have a /listexecutors switch as well.

Let’s quickly try running our tests with a sample xml test file as below :

Now the new test adapter is registered and we are able to run tests from command line. We should be able to run these tests from "Unit test explorer" window as well, if these tests are based on ".dll" or ".exe" extension.

Since our tests are based on ".xml", we need to implement one additional interface to make “Unit test explorer” aware of it. This is because, visual studio by default scans only ".dll" and ".exe" project items and passes them on to the registered discoverers.

Lets take a look at it in our next post.

MyXmlExtension.zip

Unit testing has been redefined in a more framework agnostic way in Visual Studio 11. Refer our MSDN articles and introductory post to get started with it and expect more posts from us in the coming weeks.

Today, let’s quickly go though the steps to get a simple metro style test running in your nightly build.

Installing and configuring the build agent:

1. Install the Team foundation Build agent on the Windows 8 box.
2. Configure the build agent to run as interactive process. This is needed since metro style apps could not be launched from a background service.
1. Enable auto-logon for the build agent service account [so that the build machine always has an active logged in session].
2. Make sure that the build agent service account has permissions to write to the build drop and it is a member of “ProjectCollection Build service Account group”
3. Install Visual Studio Win8 express on the build agent machine. This is needed to bring in the Windows 8 specific dependencies.
4. Install developer license in the build agent box. This is needed to run metro style tests.

Authoring the tests:

1. Create a simple C# metro app and unit tests for it.
2. Launch VS. While choosing project template, select "Window metro style" and chose the appropriate project type.
3. Author the tests and build the project. More details can be found on MSDN.
4. On a successful build, "Unit test explorer" window would open on the left pane showing the test run progress.

Creating build definition:

            Couple of things to take care of while specifying test parameters in the build definition:

1. Specify "Visual Studio Test Runner” in the TestRunner dropdown. This would be the default value.This ensures that the tests are run via the new Visual Studio 11 based test runner instead of MSTEST.exe.
2. Specify *.appx for test assembly specification instead of .dll. For running metro style tests .appx file has to be given as input to the test runner instead of .dll.

Installing the certificate:

Just one more step to go before triggering a build.The certificate of the test assembly need to be installed on the build agent machine. This is a one time operation and can be done by one of the following ways :

From an admin cmd prompt on the build agent box, execute “Certutil -importpfx *.pfx”  where the pfx file could be found in the project folder of your test project. Note the password for the default pfx file is blank. Also, you can use the same pfx file for all your test projects (instead of having multiple files). This can be done by setting the <PackageCertificateKeyFile**.pfx</PackageCertificateKeyFile> node in the csproj file.

Else, install the certificate as part of post build event using the following steps :

  a. Update the csproj file so that post build event gets executed only in the case of Teambuild and not local build.

  b. In post build event specify a script to execute “Certutil -addstore root *.cer” where the *.cer file could be found in a sub folder in the build drop. Create a wrapper script  which would get the full path of the .cer file present in the build drop and pass it to the Certutil command.

Trigger a build and analyze the test results:

Now queue a build and analyze the results of your metro style tests.

Frequent errors and resolutions:

1. Could not start tailored test run because Unit tests for Windows Metro style app cannot be run from a service or non interactive process. Please run unit tests from an interactive process.
1. This error occurs if the build agent is running as service.
2. Access is denied. (Exception from HRESULT: 0x80070005
1. This error occurs if the logged in user in the build agent machine is different from the build agent interactive process account . Make sure that the logged in account and the build agent account are same for Metro style apps to be run properly.
3. Installation of package 'C:\Builds\1\proj1\CalcBuild\Binaries\CalculatorTest\AppPackages\CalculatorTest_1.0.0.0_AnyCPU_Debug_Test\CalculatorTest_1.0.0.0_AnyCPU_Debug.appx' failed with Error: (0x80073CF1) Package was not found. (Exception from HRESULT: 0x80073CF1).
1. This error occurs if the certificates are not installed on the build agent box. Make sure that you install the certificate either manually (one time) or using post build scripts. Also make sure that the certificates of newly added test projects are also installed. [ or have all test projects point to same pfx file which is installed one time]
4. Code coverage collection with metro style tests

                       Code coverage doesn’t work with Metro style tests ( either in VS or in Team build or via the command line runner).

One of the FAQ around MSTEST is on the folders exposed by TestContext. What each of them means and which one to use when.

Below is a quick summary of the various folders and their usage.

TestRunDirectory (TestDir)

Indicates the high level folder created for the current test run. This would have “In” and “Out” subfolders inside it.

TestDeploymentDir

Indicates the “Out” sub folder under the “TestRunDirectory”. This is where the items needed for the current run would be deployed prior to the test run.

  If you need to refer to any of your deployed files, you should be using this variable.

TestRunResultsDirectory (TestLogsDir)

Indicates the “In” subfolder under “TestrunDirectory”. This is the folder where the run level attachments and data collector files would be stored after the test run.

If you need to create temp files during the test run, you should be using this variable.

TestResultsDirectory

Indicates a sub folder inside “In” folder corresponding to the current test case.

For each test case in the test run, a sub folder would be created which would have the test case level data collector logs like “EventLogs”.

If you add a file to the test result using TestContext.AddResult(), the file would go under the “TestResultsDirectory” corresponding to that test case.

Hope you found it useful. if you have further questions, please post your queries.

1. Do I need to install full VS to use MSTEST? Is there any light weight option?

You need not have full VS to bring in MSTEST. Test agent is a light weight option to bring in MSTEST. Refer to the below link for details:

http://blogs.msdn.com/anutthara/archive/2009/12/16/running-tests-in-mstest-without-installing-the-vs-ide.aspx

2. Does MSTEST work for 64 bit binaries? Any known issues with it?

MSTest doesn’t work for 64bit binaries. However mstest can be used for 64bit testing. To do this, test assembly needs to be compiled for “AnyCPU” and test setting should specify 64bit host process for test execution.  Also the test should be running on 64bit machine. This will load the test assembly in 64bit and testing will happen in 64bit mode.

3. Does Side by side installations work?

For VS:

VS 2008 and VS 2010 on same machine, works fine.

For test agent and test controllers:

For agent and controllers, side by side install is not supported.

4. What kind of logs do I need to debug issue with a test run?

For VS 2010, the devenv logs and QTAgent.exe logs are needed.

In VS 2008, you need to enable the logs for VSTESTHOST process:

http://blogs.msdn.com/yutong/archive/2006/10/03/How-to-Enable-VSTS-Logging_3F00_.aspx

Also take out the logs from the event viewer under event source “VSTTAGENTProcess”

5. Publish issues with MSTEST

VS 2008 and TFS 2008:

Error:

Publishing results of test run geobuild@B88PSRACK017 2010-05-05 10:47:20_Any CPU_Release to http://tkbgitvstfat04:8080/............Publish completed successfully.

Done executing task "TestToolsTask".

Resolution:

In 2008, MSTest pushes results to the server, which then copies them out to the drop share.

Hence the TFS AT service account (not the TFS *Build* service account) needs permissions to the share.

VS 2010 and TFS 2008

Error:

Unit tests are executed properly, but code coverage is not reported back to the TFS 2008 data warehouse as the build summary says "No Code Coverage Results". I get an additional log file on the build server in ...\TestResults\CoverageLog5.log saying::

The Code Coverage Analysis Service could not analyze the code coverage data for build DevLineDailyBuild>20100630.3 (Any CPU/DevDeploy). This is frequently caused by publishing different versions of the same binary to a particular build.

Processing of code coverage data from run XXX@ServerName 2010-06-30 02:34:05_Any CPU_DevDeploy caused the following exception to be thrown:

Error when creating coverage info: Error loading symbol file. Symbol and binary files should be in the same folder as the coverage file or on the symbol path: C:\Documents and Settings\usdg-andromeda\Local Settings\Temp\Bode\DevLineDailyBuild\Binaries\DevDeploy\XYZ.dll.
The data.coverage file is generated at “TestResults\In\ServerName\” folder. “TestResults\Out” folder is empty.

Resolution:

This a known issue where publishing code coverage from 2010 client to a 2008 server was broken.

6. Is it possible to pass custom parameters to through MSTEST that will be passed to Setup and Cleanup scripts?

Or

My “buildrop” location keeps changing everyday. I need to deploy some binaries from my build drop for my test run. Does the VS unit testing framework supports any token for %builddrop% ?

It is not possible to pass custom parameters through MSTEST. Alternative could be to set the required parameters as environment variables and let you script take those values.

7. What is the order in which initialize and cleanup methods get called?

All cleanups (assembly clean and class cleanup) will be called only during the end of the test run after all the tests in all the classes are executed.

Let’s consider an example: Two test projects: A.dll (class 1 - test 1a, test 1 b, class 2- test 2) and B.dll (class 3- test 3)

Below will be the rough flow of execution:

· Assembly init for A

· Class init for class1

· Test init for Test 1A

· Test 1A

· Test cleanup for test 1A

· Test init for Test 1b

· Test 1b

· Test cleanup for test 1b

· Class init for class 2

· Test init for Test 2

· Test 2

· Test cleanup for test 2

· Assembly init for B

· Class init for class 3

· Test init for Test 3

· Test 3

· Test cleanup for test 3

· Class cleanup for class 1

· Class cleanup for class 2

· Assembly cleanup for A

· Class cleanup for class 3

· Assembly cleanup for B

Below posts explains them in detail:

http://blogs.msdn.com/nnaderi/archive/2007/02/17/explaining-execution-order.aspx

http://blogs.msdn.com/b/ploeh/archive/2007/01/06/classcleanupmayrunlaterthanyouthink.aspx

8. How will I make sure that all the test cases in one class get executed before the tests in another class?

There is no way to guarantee that all tests in one class get called before the tests in another class. One work around to achieve this is to put the tests in the class inside an ordered test. But even with this approach, the class cleanup of this class will not be called right after the execution of the ordered test. If you have to dispose objects before starting execution of tests in another class, you need to add the cleanup logic in test cleanup of the last test method in the ordered class.

9. How can I create my custom test type? Do we have samples for Custom host adapter sample?

VS SDK 2008 downloads: http://www.microsoft.com/downloads/details.aspx?familyid=30402623-93ca-479a-867c-04dc45164f5b&displaylang=en

VS SDK 2010 Beta 2 downloads: http://www.microsoft.com/downloads/details.aspx?FamilyID=cb82d35c-1632-4370-acfb-83c01c2ece24&displaylang=en

10. I get the below error while trying to run my custom test type. “File extension specified '.aii' is not a valid test extension.”

a. Remote execution :

1) Make sure that the host adapter is chosen properly in test settings ( to chose the host adapter in test settings it need to be registered on the machine where you create test settings. If you are running test from Microsoft Test Manager this has to be present in MTM machine as well)

2) Register the test type and host adapters in the test controller machine and the agent machines. And you restart the services after doing the registration.

b. Local execution:

Make sure that the test type is registered properly. HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\10.0\EnterpriseTools\QualityTools\TestTypes for VS .

11. "One or more of the tests (directly or indirectly) included in this run could not be found" –

This error happens when no test case can be found with the given ID. In the case ordered test, the contained test method name would have changed. For custom test types, make sure that you are not generated a random GUI for test ID during each build. Instead you can return the hash of Test method name, class name and namespace name as the test ID (similar to how out of box test types work).

12. Any way to out a hyper link or custom sections in the results file?

No.

13. Is there a way to run unit tests in parallel?

Check out http://blogs.msdn.com/vstsqualitytools/archive/2009/12/01/executing-unit-tests-in-parallel-on-a-multi-cpu-core-machine.aspx for enabling parallel execution of unit test. Note that it is up to you to make sure that each individual test in the run can be run side by side with some other test without any conflicts (like cleanup issues, one negative test running in parallel with a positive test…).

Deployment

14. How does deployment work?

Test execution deploys the below items :

· DeploymentItem()] items placed on [TestMethod] function

· Those in the /runconfig file in the Deployment files section

· Those that have reference (strong) from within the binary.  This usually requires a reference in code to make sure the meta data is in the assembly

“CopyLocal=true” has no effect on whether MSTest commandline deploys the item

Mstest goes only one level deep to deploy the reference assemblies . If A refers B and B refers C, it wont deploy C.

15. Deployment is very slow

If the _NT_SYMBOL_PATH environment variable points to a non existent path, then deployment might be slow.

Details in this blog post.

16. How does deployment of .config files work?

By default only the config file of the test binary will be deployed and not other config files . More details here. For other .config file, you need to add them as deployment items explicitly.

18. What is the best way to deal with deploying dependent assemblies while running tests?

It is a normal for the test code to take dependency on product binaries. In such cases, the whole product binaries need not be copied to the location of the test binaries.

You can simply set the "Root folder" in the test settings to the location of your product binaries. This path will then act as probing path during assembly resolution.Refer to the this blog post for details on the same :

http://blogs.msdn.com/vstsqualitytools/archive/2010/01/09/assembly-resolution-for-unit- tests.aspx

Note that, whatever binaries that are present in the probing path will be resolved properly, irrespective of whether they are deployed along with the test binaries or not.

19. How to customize the default deployment directory?

You can change the default deployment folder by editing the test settings file in the XML editor:

<Deployment userDeploymentRoot="C:\TestResults" useDefaultDeploymentRoot ="false" />  

Note that ,if the test settings is modified by XML edit (instead of using the default editor) VS need to be closed and reopened ( since the editing is done in XML , the changes will not be updated in the loaded settings.)

Test Rig:

20. Is there a way to programmatically control the rig operations?

The controller SDK is not a public one. However you can achieve various rig options programmatically as below:

a) Register agent with controller - TestAgentConfig.exe /Configure ( this command line will be present in %ProgramFiles%Common7/IDE in the agent machine )

b) Start/Stop service - sc \\machinename stop vsttagent ; sc \\machinename start vsttcontroller. If the agent and controller are in different domain, you can use IP instead of machine name.

c) Remove an agent from the controller - TestAgentConfig.exe /delete

Alternatively, you can edit the QTController.config.xml file (present in %programFiles%Visual Studio 2010/Common7/IDE folder in Controller machine) as well to add agents ( and add properties to agent ). The edit of XML can be done programmatically as well. Make sure to restart the controller service after editing the XML.

21. Will Test Agents allow you to impersonate another user? Or does the Test Agent always kick of tests as the user its configured start up as?

There is no built in support for impersonation, so by default tests run under the identity of the test agent.  If this is something you really need, you could write a unit test extension that does the impersonation for you.

22. Agent doesn’t go to ready state or any issue with controller-agent configuration

Refer to Test agent-controller trouble shooting guide

The test controller, test agent logs and the event viewer error messages in the agent/controller machine will also give an idea.

23. How can I distribute my tests across agents?

You can achieve this by setting the bucket size in the test settings file. Refer to this blog for details on it:

http://blogs.msdn.com/b/vstsloadtestblog/archive/2006/02/08/527597.aspx

If you are running tests from VS, then the test settings file can be edited using an XML editor. But for the test settings created using Microsoft Test Manager, there is no easy way to modify the bucket size. This is something we are planning to fix in next release.

24. What happens if I queue multiple test runs on a test rig at the same time?

It depends on the number of available agents in the controller. Consider there are three agents which matches the criteria. The first run that gets queued will reserve ExecutionAgent1. And the second run shall use EA2 and the third will pick up EA3. The fourth run will fail with error mentioning “No agent found …”. But if the count of test cases in the first run is > the bucket size ( which is 100 by default), then appropriate number of execution agents are reserved. For example: If the first run has around 250 test cases, then all three agents will be reserved by the first run. 100 tests will send to ExecutionAgent1 and ExecutionAgent2 and 50 tests will be sent to ExecutionAgent3. And the second run will fail with “no agent …” error.

25.  I have successfully deployed test agents and a test controller.  The agents show up healthy when I ‘manage test controllers’ from VS.  However when I try and use the agents to run a test case VS just shows the test results as Pending and it never completes.

It might be because of blocking firewall in the VS client machine.

VS client doesn’t listen on one specific port for the incoming calls from controller. The port will be chosen in random (default range 1000-5000 or so).  Hence you need to add the QTAgent.exe process (instead of port) in firewall exception list in the client machine.

The Lab management end-to-end workflow works leverages the Team Foundation Build Agent/Build Controllers to do the deployment.

Given that build agent is primarily targeted for compiling sources, a getting-started user will get a bunch of questions :

a) How does the build agent inside a lab environment work ?

b) Can I use the build agent inside the lab environment for both compiling sources and deploying my application ?

c) How are the build agents selected for compiling sources ?

Let me try to answer the above questions. And before that, take a look at our Getting started with Lab Management series and Lab management videos

How  does the build agent inside a lab environment work ?

The build agent inside a lab environment also does the regular functionality of executing the tasks given by build controller. In the case of an end-to-end (E2E) workflow ( which is nothing but a build definition), the task done by the agent will be deployment ( any script that user specifies : WIX, MSI, batch file, ps …) instead of compiling sources.

If you have lab service account set for your project collection, then the build agent will use that identity while talking to TFS.

Example: If you have your build agent running as domain\user account in your VM and set your lab service account as domain\svcaccount, then any out going call made by the build agent to the TFS will go with this identity. As the name implies, lab service account will be a service account with less privileges and hence this account will not be granted permissions on source control by default.

Can I use the build agent inside the lab environment for both compiling sources and deploying ?

The recommended way is to have a separate build agent machine for compiling sources.

The reasons being :

Example: To build a share point app and build a wsp package, the sharepoint target binaries are required on the build machine and it is not possible to have that in the lab environment ( since the VMs need to be clean before deployment).

How is the build agent selected for compiling sources ?

• For running the E2E workflow, you will be creating a build definition for compiling sources. The default "Agent selection criteria" for this build definition will be "Agent which has any name and which doesn’t have tags ". The default " Tags is empty" criteria is to ensure that lab build agents are not picked up for compiling sources ( lab build agent always carries few custom tags that are pushed by lab and will not satisfy this criteria).

  

  If you have added tags to your non-lab build agents and just go with the default "Agent settings" options while creating the build definition, the tagged build agents will not be picked up for compilation and you might get an error which says "The build controller xxx does not contain an enabled build agent with name * and no tags".

  

  To workaround the issue, chose the tags explicitly while chose the agent for compiling sources (like below) or have separate build controllers - one for managing compilation agents and another for managing lab build agents.

  

  If you chose any of the lab build agents explicitly for compiling sources, then you will get an error saying "No matching items found in S/ in your workspace, or you do not have permissions to access them". This happens because the lab build agents will not have source control permission by default.

  

  You might get the above error, when "MatchAtleast" is chosen instead of "MatchExactly" in which case the lab build agents might also get picked up.

    How can I compile the sources using build agents inside the lab environment:

       Though the general recommendation is to not use the build agents inside the lab environment for compiling sources, you might have a scenario to do so. In that case you need to do the below steps to acheive this :

•         Grant the lab service account "Read" permissions on the source control. By default when the a lab service account is set, it is denied source control permissions (to be "secure by default"). But, if you want your lab machines to compile sources then, the lab service account need to get the sources and hence it needs read permissions. This post explains how to give read permissions using "tf permisisons" command.
•         While creating the compilation build definition, make sure that you either explictly chose the specific lab build agent or chose a tag or set some name filter so that the lab machines get chosen ( the reason being, by default lab machines will be excluded from the selection as mention above).

Hope this helps!

Attached is the customized xaml that you can download and use directly, after checking-in the XAML to your source control and creating a new build definition based on that.

Let me go through the additional things I have done on top of what is explained in the previous post :

1. Drag and drop the activity "SnapshotLabEnvironment" from "TeamFouundation Lab Management activities" section in addition to the "OpenWorkItem" activity.

2. Define two new string variables in the scope of the "Sequence" and name them "LocalForwardLinkID" and "LocalsnapshotID".

3. Let's take a look at the "SnapshotLabEnrionment" activity .

This activity takes the "LabEnvironmentUri" and "SnapshotName" as input parameters and snapshots the given environment and returns the ID of the newly taken snapshot. In addition it has an output parameter called SnapshotChainID.

For "LabEnvironmentUri" input parameter, specify the variable LabEnvironmentUri ( the value for which would have been already set in the workflow).

Specify some value for "SnapshotName" input parameter. The snapshot I have given is "TestRunFailureSnapshot" + DateTime.Now.Ticks.ToString. The date time is appeneded to ensure uniqueness of snapshot name.

Assign the return value and the out parameter value of the snapshotLabEnvironment activity to LocalSnapshotID and LocalForwardLinkID respectively.

4. Next step is to pass on the snapshot detail to the newly created bug. As described in this post, it is possible to create Environment /Snapshots links if we know their IDs.

So let's compose a url for the newly taken snapshot and write that url in the bug description and we are done :)

Specify the below value for the "Comment" parameter of the "OpenWorkItem" activity.

String.Format("Snapshot of the environment when the test run happened is : <a href=""lvr://ConnectToFwdLink/?CollectionUri={0}&ForwardLink={1}"">here</a>", LabWorkflowParameters.EnvironmentDetails.TfsUrl, LocalForwardLinkID)

The above string is essentially the url for the newly taken snapshot composed using its ID . It is surrounded by a HREF tag to make it appear as a hyperlink in the bug.

5. Run the build and the new bug will be created when the test run fails. The new bug will look something like below.

6. Clicking on the "here" hyperlink from the history section in the bug will pop up the below dialog where can you chose to revert to the snapshot and debug your failure.

Cheers !

CustomizedLabDefault.xaml

One of the common requirement while running E2E build is to log a bug automatically when the test run fails. It is very simple to achieve this functionality by customizing the workflow.

Lets go through the steps for the same :

1.Make a copy of "labdefaulttemplate.xaml " and check-in the new xaml.

This post talks about getting started with E2E customization. Follow this post to make a copy of labdefaulttemplate.xaml and check-in the new XAML to source control. Let's call it "CustomizedE2E".

2.Now open the xaml in the workflow editor. Locate the "If all tests have not passed" block in the workflow.

3.Add a sequence activity inside the if block and put the "Set build status" activity inside the sequence. Drag and drop the "OpenWorkItem" activity from "Team Foundation Build activities" section in the toolbox into this sequence.

4.Specify values to the parameters of the "OpenWorkItem" activity.

Below are the values that I have specified for each parameter :

5.Save the xaml and check it in to the source control. Create a new E2E build definition based on this template.

6.Now run the E2E build and you can notice a bug being logged on test failure.

E2E build summary pointing to the newly logged bug ..

The bug that was logged automatically by the E2E workflow..

Hope that helps ! Cheers :)

And this time we have "Getting started videos" to help you out. I am sure you would love it :)

Well, that's pretty much about "what's new in RC for over all for Lab management". Let's have a closer look at the E2E workflow functionality and briefly skim through the RC  enhancements made in RC.

1. Link to test results in the build summary page :

Now while looking at the E2E build summary from visual studio, to analyze your test run you can click on the "View Test Results" link which will launch Microsoft Test Manager and connect to the appropriate test run.

2. Link for post deployment snapshots :

In Beta2, the post deployment snapshot file ( *.lvr) will be stored in a location that is specified in the E2E build definition. It is no more needed in RC. No need to specify any location and maintain the lvr files. Just specify the snapshot name. The build summary will have a hyperlink to the snapshot, clicking on which will open "Microsoft environment viewer".

3. Build configuration :

From the E2E build definition while selecting the compilation build definition(or a specific build), you can chose a specific build configuration of interest.

The selected configuration will be picked up for deployment and running tests.

4. Improved reliability and diagnosibility:

We have made bunch of fixes that will enhance the robustness of the E2E workflow.

For example : Few of the workflow activities used in E2E now takes an additional parameter called "RetryCount". This will help in making the workflow more robust by avoiding failures due to intermittent network issues.

Also bunch of error messages were revamped to enable the user to recover himself on hitting any issue.

5. No option to create environment but just restore :

In Beta2, user can either restore an existing environment to a clean state or create a new environment for doing E2E. We have removed the latter option in RC, since restoring to clean snapshot is more prevalent. But if you need to create new environment for your nightly, you can do so by customizing your template.

Note: If you have existing Beta2 build definition which creates new environment, you need to update your build definition to pick up an existing active LE. Our upgrade guide has details on the same.

Catch you shortly with more interesting posts on E2E …..

In a typical nightly build scenario involving a multi tier app, the deployment and test scripts running on one tier might need the machine name of the other tier.

Example:

• Let's say we have an environment by name "Nightly". It has two virtual machines by name "Webserver" and "DbServer".

• Have a simple program which will open a given file(arg0) and do a string.Replace of String 1( arg1) with String2( arg2). Lets call it ReplaceText.exe.

• In the checked in version of Web.config file edit the DB connection string to some constant value, say "DBSERVERNAME".

• Now in your deployment script copy the web.config file to "WebServer" VM along with other binaries.
• Invoke ReplaceText.exe on Web.config file and pass the computer name of the "DBServer" virtual machine as an argument by using $(InternalComputerName_DBServer) macro.

In the below example, first all the binaries and config files are copied from build drop location to C:\IBuySpy folder. Then ReplaceText.exe is invoked with following parameters : target file ( Web.Config ), string to be replaced (DBSERVERNAME ) and replacement string ($(InternalComputerName_DBSERVERName) macro which will be expanded to the machine name of DBServer VM). Following that other scripts are invoked.

Few things to note :

• Make sure the build agent service account has read permissions on the share in which ReplaceText.exe tool is present.
• In the case of network isolated environments with a private domain controller, the VM s will be joined to the private domain. If the corp domain has IPSEC policies set, then the VMs from the private domain cant access the corp resources. Hence for the build agent service account to access the network share, the machine need to have IPSEC disabled. In the above case "ddrelqa" machine need to have IPSEC disabled.