In an effort to reduce the number of places you need to search for ALM Content including Diagnostics, Testing and Lab Management we are combining these blogs to one location:

https://blogs.msdn.com/b/visualstudioalm/

Come on over tell us what you think!

Chuck

 In this walkthrough, we will:

• Install additional software necessary for our scenario. In particular, we will install a Test Controller in the virtual machine, and then configure the Build Controller and Build Agent.
• Create a Standard Environment from Microsoft Test Manager. We will use the same virtual machine as the test machine for the standard environment.
• Create a Build-Deploy-Test definition in Visual Studio, and run it on top of the environment.

 Before going through this walkthrough, ensure that you have:

• Downloaded and setup the virtual machine as per the instructions in Brian Keller's blog post on a Hyper-V host with an internal network. We will refer to this VM as VSALM machine in the following description.
• Login to the VSALM machine as Administrator to perform the following steps.

Install additional software and create additional assets

1. Install and Configure the Test Controller.
1. Download the Visual Studio Agents 11 Developer Preview ISO to the host machine.
2. Mount the ISO into the VSALM machine using Hyper-V console.
   
3. From the VSALM machine, browse the DVD, and launch TestController\vstf_testcontroller.exe. Reboot if you are prompted to do so.
4. After the installation, launch Visual Studio Test Controller Developer Preview Configuration Tool. Configure using default settings and select the DefaultCollection from TFS. Wait for all the configuration steps to finish
   
2. Create a test case in TFS using one of the MSTest unit tests.
1. Open Visual Studio. open the solution for FabrikamFiber.CallCenter. Compile the solution. Open Unit test explorer, and run the unit test 'CreateInsertsCustomerAndSaves'. Using unit test explorer is covered in one of Brian's hands-on-labs.
2. In Visual Studio, open Team Explorer, and create a new Test Case work item. In the associated automation tab, select the FabrikamFiber.Web.Tests.CustomersControllerTest.CreateInsertsCustomerAndSaves test.
   
3. Save the test case.
3. Configure the Build Controller and Build Agent in the virtual machine.
1. Open Team Foundation Server Administration Console, and select Configure Team Foundation Build Service.
   
2. Go through the wizard. For the account to run the build service, choose .\Administrator. This is required for the build agent to be able to access the drop location.
3. At the end of this configuration, you should have one controller and one agent configured for builds.
4. Click on 'Properties' of the VSALM - Controller, and change the number of workflows that the controller can execute in parallel to "2". This required so that you can run the Build-Deploy-Test automation workflow, which in turn spawns the build workflow.
   
4. Create a Build definition in TFS to compile the FabrikamFiber solution.
1. From Visual Studio, select Team Explorer, and then Builds.
2. Create a new build definition. On the "General" tab, type in a name for the build definition - say, Fabrikam Build. On the "Build Defaults" tab, select a share where the builds should be generated. For instance, type \\vsalm\c$\drops. Ensure that you have created a folder called C:\drops, and have given "Everyone" (or atleast .\Administrator) read-write permissions to this folder. In the "Process" tab, select the Default Template. Save the build definition.
3. To verify that the build definition is correctly setup, queue a build, and verify that the solution compiles, and that the build output is generated in C:\drops folder. If you get a permission error, ensure that the build service hold is running under .\Administration account, and that you have given the permission to this account to access the build share.
5. Create an automated test plan in Microsoft Test Manager.
1. Open Microsoft Test Manager with elevated permissions (Shift-right-click Microsoft Test Manager in Start Menu and select Run as Administrator).
2. Create a new test plan called Automated Test Plan.
3. Add the test case created in the step above to the test plan.
   

Create a standard environment

1. Open Microsof Test Manager with elevated permissions (Shift-right-click Microsoft Test Manager in Start Menu and select Run as Administrator). Go to Lab Center.
2. Select "New" under Environments.
3. On the "Type and Name" tab, type in a name for the environment - say, Fabrikam BDT.
4. On the "Machines" tab, add a machine and type "vsalm" for the name of the machine. Select "Web Server" as the role of the machine.
   
5. Select "Verify". If the verification fails, ensure that you have launched MTM with elevated privileges.
6. Once the verifications pass, select "Finish".
7. The creation of standard environment takes a few minutes. During this time, the agent is automatically downloaded and installed on the machines of the environment (in this case, the vsalm machine itself). At the end of the process, you should see that the environment goes into "Ready" state.
   

Create and run a Build-Deploy-Test definition

1. Create a Test settings.
1. From Microsoft Test Manager, go to Lab Center, and to Test Settings.
2. Create a new automated test settings. On the "General" tab, type in the name of the test settings - for e.g., Fabrikam BDT Test Settings. Select "Automated". On the rest of the tabs, choose the defaults, and save the test settings.
   
2. Create a Build-Deploy-Test definition.
1. In Visual Studio, go to Team Explorer, and open Builds. Select "New Build Definition".
2. On the "General" tab, type in a name for the definition - say, BDT.
3. On the "Build Defaults" tab, uncheck "This build copies output files to a drop folder".
4. On the "Process" tab, select LabDefaultTemplate.11.xaml as the build process template.
   
5. Click "..." next to the Lab Process Settings, and enter the following settings:
1. Environment: "Fabrikam BDT" or the name that you gave to your environment
2. Build: Set the build definition to "Fabrikam Build" or the name you gave to the compilation build
3. Deploy: Check the option to Deploy, and Add a row in the grid. For the Machine role, select "Web Server". For the Deployment script, type "cmd /c dir". We are not really doing a deployment in this walkthrough since the application is already deployed in the virtual machine.
4. Test: Select the automated test plan and the test settings that you created.
6. Save the build-deploy-test definition,
3. Queue an instance of the Build-Deploy-Test definition.
1. Select the build definition that you just saved, and queue it. This should trigger a compilation, followed by deployment, and then the exeuction of the test cases.

What next?

In this walkthrough, you have seen how easy it was to create a standard environment. The process of installing agents onto the machines of the standard environment is automated. You also saw how you can run a build-deploy-test workflow using a standard environment. These are some of the enhancements in Visual Studio 11.

In the walkthrough, you have used the same machine for creating the standard environment. In a more realistic scenario, you would typically have test machines that you would compose into an environment. Accomplishing that using the VSALM machine can be done as follows:

• Create another virtual machine with Windows 2008 R2 operating system, joined to workgroup, and present on the same Internal Network on the same Hyper-V host. We will call this virtual machine as "Test Machine" in the following description.
• When you setup this machine, ensure that the Administrator's password is the same P2ssw0rd.
• On the Test Machine, edit the C:\windows\system32\drivers\etc\hosts file. Add an entry <IP address of VSALM Machine> www.fabrikam.com to the hosts file. This is required for the test machine to be able to find www.fabrikam.com.
  
• 
  Ensure that the Network and File Sharing exception is enabled in the Firewall settings for the Test Machine.
• The project that comes with the VM is an MVC project. In order to run MVC tests on the Test Machine, you need to install additional software on it. (E.g., Visual Studio with MVC3. Or, if you are like me, and are ok with the test run failing, then skip this step :). You can always do it later.).
• From Microsoft Test Manager, create a new Standard environment. This time, provide the computer name of the Test Machine. In the wizard, select the 'Advanced' tab, then select the option to run Coded UI Tests. This will cause the test agent to be run using the provided account and authenticate succesfully with the test controller.
  
• Once the environment becomes Ready, queue the same Build-Deploy-Test definition on this environment.

Give the standard environments a try, and let us know how they work for you. Brian Harry's post on how to use standard environments for Build-Deploy-Test automation can be found here.

This requirement can be achieved by making the test code read the Server names/other configuration through environment variables:

1. The following post by Gautam explains the changes that will be required in the test code: https://blogs.msdn.com/b/gautamg/archive/2010/02/23/how-to-get-same-test-running-under-different-environments.aspx
2. Extra requirement from VS Lab Management is that, the environment variable binding should be done using the following code:
   
   Environment.GetEnvironmentVariable("TestServer", EnvironmentVariableTarget.Machine);

Now, we need to configure our test machines to have the correct values of the environment variables, for achieving this we can do:

1. Set this environment variables on the test execution machine before we trigger a test run manually. This approach can be used in cases where the values will not change frequently or from environment to environment.

or

1. The values to the environment variables can be set as part of Deploy of Lab E2E Workflow. For doing this we add a deployment script to run the following command on test execution machine:
   
   cmd /c setx /M TestServer $(InternalComputerName_testServerName)

Please refer https://msdn.microsoft.com/en-us/library/ee471614.aspx for information about Deploy and macros of Lab E2E workflow.

Note: The capability to run Lab E2E Workflow is only available for virtual environments in VS 2010. It is available for both Standard (Physical) and SCVMM environments in VS vNext.

Well, we now have an avenue for you to add and vote for your favorite suggestions. The UserVoice site for Visual Studio is here: https://visualstudio.uservoice.com/ It is simple to enter your idea, and if a lot of people agree with you, chances are high you'll see your idea incorporated in the product.

Note that User Voice is really not the best avenue for reporting bugs in the current version of the products. https://connect.microsoft.com/visualstudio is really the best avenue for that. It’s got a direct connection to our internal bug database and increases the possibility that it will get fixed. UserVoice is really better for larger feature suggestions.

So, get started, vote for your favorite ideas, and help us make the product better!!

In addition, this guidance also describes a way of creating lab-ready golden templates using VM Factory. This is an alernative to using VM prep tool, if you are familiar with the VM Factory.

Bookmark the Visual Studio Lab Management Guidance TOC blog and monitor the Visual Studio ALM Rangers blog, using tag tfslm, for the latest information on this project. Give this a try when you are planning for or using Lab Management. Give feedback on additional topics that you would like to see included in the guidance either here or at the Rangers Lab Management site.

Here is a summary of the list of issues fixed by the QFE, to help you decide whether you should apply the QFE:

Issue 1: When you run tests on the test agent that is installed on a computer that has Visual Studio 2010 SP1 installed, the tests may not run, and the following error message is logged:

"Attempted to access an unloaded AppDomain. (Exception from HRESULT: 0x80131014)"

Issue 2: When you run a playback of a Coded UI Test on certain Windows Presentation Foundation (WPF) controls, a Microsoft.VisualStudio.TestTools.UITest.Extension.UITestControlNotFoundException exception occurs, and then you receive the following error message:

"Search may have failed at '<name>' <control type> as it may have virtualized children. If the control being searched is descendant of '<name>' <control type> then including it as the parent container may solve the problem."

Issue 3: When you try to create a work item from the Test Results pane in Visual Studio 2010 after you connect to a Team Foundation Server (TFS) server, a System.OutOfMemory exception occurs, and the creation operation fails. This issue usually occurs if the TFS server has many builds.

Issue 4: When some tests run in a build operation, some builds that are queued in a TFS server stop responding. Additionally, the following error message is logged in the build log:

Waiting to publish...
Publishing results of test run <build name> to https://<TFS server address>:8080/tfs/DefaultCollection...
The process cannot access the file '<directory>\data.coverage' because it is being used by another process.
Publish failed or canceled.

To accomplish this with Lab Management, we will need

1. Visual Studio 2010 Feature Pack 2 (To be installed on developer box, build box, test controller and test agents)
2. Customized build template derived from LabDefaultTemplate. (Attached to this post)

Lab Management uses Test Settings to define the Role of the environment to use for running test and also for data collection. If there are more than one machines of the Execution role in the environment, then the tests are distributed among them. For configuration matrix testing between IE and Firefox we want to be sure that all our tests run both on IE and Firefox.

So to accomplish our goal we will be using multiple test settings and multiple automated test runs within our environment.

Let us setup the environment now, to do that we do the following:

1. Create a Lab Environment for our application with the additional requirement that we add one more Web Client machine and name the roles for the two machines WebClientIE and WebClientFF.

 

2. Create two test settings by going to the test settings page in Lab Management for this environment; name them TestSettingIE and TestSettingFF. In TestSettingIE, we set WebClientIE as the execution role and enable the data collectors on other roles as required (you should disable data collectors on WebClientFF role as this machine will not participate in the test run for this test setting). In TestSettingFF, we set WebClientFF as the execution role and enable data collectors on the other roles.

 

3. Login into the WebClientFF role machine and we need to set the environment variable CodedUITestCurrentBrowser to specify to the Test Agent to run the Coded UI tests in Firefox. Restart the Test Agent process.

 

4. Make sure all other pre-requisites for deploying the application are also done and we can then take a checkpoint for our daily Lab E2E workflow build.

 With these steps our environment is ready. Now we will do the setup of the test cases.

1. After installation of Feature Pack 2, we open our test project from source control and record a new Coded UI test. This will add the required references for running our Coded UI test against Firefox. After this we check-in our test project back to source control. (A good idea at this point will be to make sure that all tests which we plan to run on both IE and FF are run on your development box with Firefox as the execution browser to avoid any test issues later.)

Now we need to setup the build:

1. Please download the attached LabDefaultTemplate_ConfigTest.xaml; We need to check it in the source control in the BuildProcessTemplates directory. This template is the default Lab build template with the changes to execute two test runs. You can open the file in Visual Studio to understand the changes.

 

 

2. Now we define a new build and select our LabDefaultTemplate_ConfigTest as the build template (to do so you will need to select the add existing xaml option).

 

 

3. We enter all the parameters for the environment created earlier in this build as we normally do for Lab E2E workflow builds, the test setting selected on the Test page of the workflow parameters wizard will be ignored  by our build.
4. We need to enter the test settings on the Process page of build under the values TestSettingRun1 and TestSettingRun2, we need to enter the values as ID of the test settings which we can get from the Test Settings page of Lab Management. We enter the ID for our TestSettingIE and TestSettingFF in the two fields.

 

Now we can run this build and it will go through the build, deploy & test phases. It will create and execute two test runs by the name BuildNumber & BuilNumber_2, we can check them under Test > Analyze Test Runs page of Microsoft Test Manager.

With this we have accomplished our goal of executing same set of tests on both IE and Firefox using Lab E2E workflow.

For running the test in Firefox, we used a feature introduced in Feature Pack 2 (more info) to capture Coded UI test in IE and run them in Firefox. If you want to run the configuration matrix across various versions of IE, we do not need to install Feature Pack 2, we just need to have the different versions of IE installed on the two WebClient machines we have in our environment.

You can also extend this testing to beyond two runs by modifying the build template to include more test runs and creating the additional web client machines and test settings in the environment.

LabDefaultTemplate_ConfigTest.xaml

In order to use the new Windows 2008 R2 SP1 hosts and features such as dynamic memory, here is what you need to do:

(a) Upgrade your SCVMM server from 2008 R2 to 2008 R2 SP1.

(b) Upgrade the SCVMM admin console on TFS Application Tier machines to SCVMM Administration Console 2008 R2 SP1.

(c) Ensure that every host group in SCVMM has hosts of only one type - Do not mix Windows 2008 R2 hosts and Windows 2008 R2 SP1 hosts in the same host group.

(d) You have to set the dynamic memory value on the template in SCVMM library before importing the template into Lab management. If you already have imported templates, use MTM to delete them from all team projects, set the dynamic memory value on the templates using SCVMM admin console, and then re-import them into MTM.

(e) Now, when you create fresh environments using these templates on 2008 R2 SP1 host groups, dynamic memory value will be honored. You will not be able to see the dynamic memory values from MTM. MTM would only show the startup memory. However, you can use SCVMM admininstration console to see the memory usage of VMs. 

DISCLAIMER: The information in this post is provided “as-is”, we make no guarantees that the steps will work reliably, and we will not provide any support or upgrade for the definitions you create using the steps described here.

Lab Management feature in Visual Studio 2010 enables automation of the build-deploy-test cycle. This scenario is supported only on virtual environments managed by Lab Management in the current release. This post talks about how this capability can be extended to run workflow on physical machines, with some limitations.

Pre-requisites

1. Create a physical environment comprising of the machines on which you want to run the build-deploy-test workflow.

Note: In Microsoft Test Manager, you’ll see the Workflow capability status as ‘Not Configured’ for all physical environments. This is OK and your workflow can still run if you follow the steps in this post.

2. Install the build service and configure a build agent on each of the machines in this physical environment. Configure all the build agents with one build controller.

Note: Do not install lab agent on these physical machines.

3. Add a tag to the build agents you configured in step #2. You can do this either from the Team Foundation Administration Console on the build agent machine, or from ‘Builds -> Manage Build Controllers’ in Visual Studio. If you do not do this, the build definitions authored using the ‘Default Template’ (to build your code) may pick this agent to run your builds (the default matching criteria is agent with no tags) – and this might not be a desirable behavior.

Steps to setup and configure the workflow

1. Download the attached zip file, extract the workflow template ‘BuildDeployTest-physical.xaml’ and check-in to source control (example: $/<projectName>/BuildProcessTemplates).

2. Follow the steps to create a build definition described here. Instead of choosing LabDefaultTemplate.xaml, choose the template that was checked-in in the above step.

Note: If the new template does not appear in the drop down, click on the ‘New’ button besides the drop-down, choose ‘Select an existing XAML file’ option and browse for the new template.

3. This definition should be run only on the build controller to which the build agents are configured.

4. Following are the important arguments to be specified for the workflow.

a. ‘Build: New Build’ section (Fill in this section to queue a new build)

1. Build Definition Name: Name of the build definition to be queued.
2. Configuration: Configuration of the build to be used for deployment and testing (ex: Debug, Retail)
3. Platform: Platform of the build to be used for deployment and testing (ex: x86, x64)
4. Queue New Build - Indicates whether new build should be queued.

b. ‘Build: Pick Existing Build’ section (This will be used only if new build is not queued)

1. Build Location - Path from which the build should be picked up
2. Build Number - Value that identifies the build. Typical format is BuildDefinitionName_Date.Revision. This is used to associate the test run with the build. This argument is optional.

c. ‘Deployment’ section

1. Deployment Scripts – List of scripts that perform the deployment. Each script is of the format BuildAgentName # PathToExecutable # Arguments

          Example 1: Agent1 # $(BuildLocation)\install.exe # /q

          Example 2: Agent2 # cmd.exe # /c copy $(BuildLocation)\install.exe C:\installer

• BuildAgentName – name of the build agent on the machine where the script should be executed. You can check the build agent name from ‘Builds -> Manage Build Controllers’ in Visual Studio.
• PathToExecutable – file path to the executable that needs to be executed. This can contain the macro $(BuildLocation) which will be substituted by the value of BuildLocation workflow argument.
• Arguments – arguments to be passed to the executable. $(BuildLocation) macro is allowed.

2.  Run deployment scripts – Indicates whether deployment should be performed.

d. ‘Test’ section:

1. Test Environment Name – Name of the physical environment
2. Test Run Parameters – Identifies the test suites whose test cases should be run along the required test configuration and test settings.
3. Run tests – Indicates whether tests are to be run.

 

Limitations

The limitations compared to LabDefaultTemplate.xaml are:

1. Ability to pick the latest successful build. This template supports queuing a new build and picking the build from a share, but not picking the latest build, given a build definition.

2. No support to specify environment variables (the macros $(ComputerName_...) and $(InternalComputerName_...)) and working directory for the scripts.

3. No support for snapshots – Given that snapshots are not applicable for physical environments, this is a difference rather than a limitation.

Refer this article to further customize the workflow template.

We hope you find this info useful. Just to remind you again, the above steps are a work-around that enable you to run the workflow on physical machines. We will not provide any support or upgrade for the definitions you create using the steps described in this post.

BuildDeployTest_Physical.zip