1.     To begin:

We should have a web test project at hand. Right-click on the test, and select “Add Web Service Request” from the context menu.

The URL property of the newly added web service request should point to the asmx file of the web service which you wish to test.

e.g. https://www.w3shools.com/webservices/tempconvert.asmx

2.     SOAPAction Header:

The SOAPAction header must be added to the web service request. This is a hint to the servers that the HTTP POST contains SOAP request and the value of the header is the URI indicating the intent of the SOAP request.

e.g. SOAPAction = https://tempuri.org/CelsiusToFahrenheit

This could be selecting “Add Header” from the context menu of the web service request. Then set the “Name” property as “SOAPAction” and the value as URI of the web service function to be invoked.

3.     String Body:

This is the actual body of the SOAP request and is basically a SOAP envelop which looks something like:

<?xml version="1.0" encoding="utf-8"?>

<soap:Envelope xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="https://www.w3.org/2001/XMLSchema" xmlns:soap="https://schemas.xmlsoap.org/soap/envelope/">

  <soap:Body>

    <CelsiusToFahrenheit xmlns="https://tempuri.org/">

      <Celsius>value to be sent</Celsius>

    </CelsiusToFahrenheit>

  </soap:Body>

</soap:Envelope>

The Content type of the String body should be kept as: text/xml.

NOTE:

1. This SOAP envelope and the SOAPAction header are dependant on the web service in question and could be easily obtained by visiting the URL of the web service provider.
2. We may need to send some arguments to the invoked web service, which could be hard coded in to the String body of the web service request or we may intend to use some context parameter for the purpose, which is actually a better way of doing the things as it would aid in:

a.)    Ease of changing the value.

b.)    Data driving the whole web service test.

The name of the context parameter should be used inside double braces.

e.g. {{param-name}}

4.     Validation Rules:

The pre-canned validation rules provided with the web tests are not good enough for validating the web service tests, as they are designed to work with the HTML documents whereas the web services (SOAP) responses are XML documents. So we have to write our custom validation rules for the web services validation.

Following are the steps to write a custom validation rule:

1. Derive your new validation rule from the base class ValidationRule.
2. Provide the read-only RuleName and RuleDescription properties.
3. All public properties of your validation rule would be available for interaction in the UI for adding validation rules.
4. Override the Validate method of the base class providing your implementation of validations.
5. For example, to validate the content of a particular tag in SOAP response we may have validation rule of the sort:

    public class CustomValidateTag : ValidationRule

    {

        public override string RuleName

        {

            get { return "Validate TagContent"; }

        }

        public override string RuleDescription

        {

            get { return "Validates whether tag in question contains the intended value"; }

        }

        private string TagNameValue;

        public string TagName

        {

            get { return TagNameValue; }

            set { TagNameValue = value; }

        }

        private string IntendedTagValue;

        public string IntendedValue

        {

            get { return IntendedTagValue; }

            set { IntendedTagValue = value; }

        }

        public override void Validate(object sender, ValidationEventArgs e)

        {

            e.IsValid = false;

            XmlNodeList nodes  = e.Response.XmlDocument.GetElementsByTagName(TagName);

            if (nodes.Count > 0)

            {

                e.IsValid = true;

                foreach (XmlNode node in nodes)

                {

                    if (!string.Equals(node.Value, IntendedValue,

 StringComparison.InvariantCultureIgnoreCase))

                    {

                        e.IsValid = false;

                        break;

                    }

                }

            }

            // If the validation fails, set the error text that the user sees

            if (!e.IsValid)

            {

              e.Message = String.Format("{0} Tag does not contain {1}",TagName,IntendedValue);

            }

         }

      }

6. Similarly for validating the absence of “fault code” in the SOAP response the validation rule may look like:

 public class FaultCodeValidaionRule : ValidationRule

    {

        public override string RuleName

        {

            get { return "Fault Code Validation"; }

        }

        public override string RuleDescription

        {

            get { return "Validates if SOAP response contains some fault code"; }

        }

        public override void Validate(object sender, ValidationEventArgs e)

        {

            bool validated = true;

            XmlNodeList faults = e.Response.XmlDocument.GetElementsByTagName("soap:Fault");

            if (faults.Count > 0)

                validated = false;

            foreach (XmlNode node in faults)

            {

                e.Message = String.Format("SOAP Error: ");

                foreach (XmlNode child in node.ChildNodes)

                {

                      e.Message += String.Format("{0}= {1}, ", child.Name, child.InnerText);

                }

            }

            e.IsValid = validated;

        }

  }

7. NOTE: while writing validation rules for the SOAP responses we should working with the XML Document in the response object and not the HTML Document, as that would be null.

5.     Extraction Rules:

On the parallel line we shall create custom extraction rules for the web service tests. That would require extending ExtractionRule base class, providing the RuleName, RuleDescription read-only public properties, and overriding the Extract method.

Here is an example extraction rule which evaluates the given XPath query against the SOAP response.

    public class EvaluateXPathQuery : ExtractionRule

    {

        public override string RuleName

        {

            get { return "Evaluate XPath Query"; }

        }

        public override string RuleDescription

        {

            get { return "Evaluates the given XPath Query"; }

        }

        private string XPathQueryValue;

        public string XPathQuery

        {

            get { return XPathQueryValue; }

            set { XPathQueryValue = value; }

        }

        public override void Extract(object sender, ExtractionEventArgs e)

        {

            XmlNamespaceManager nsManager

 = new XmlNamespaceManager(e.Response.XmlDocument.NameTable);

            nsManager.AddNamespace("soap", "https://schemas.xmlsoap.org/soap/envelope/");

            XPathNavigator navigator = e.Response.XmlDocument.CreateNavigator();

            XmlNode body = e.Response.XmlDocument.SelectSingleNode("//soap:Body", nsManager);

            // Very first child of soap response envelope's body tag is the response tag.

            string responseNameSpace = body.FirstChild.Attributes["xmlns"].Value;

            if (responseNameSpace != null && responseNameSpace.Length > 0)

            {

                nsManager.AddNamespace("response", responseNameSpace);

                // Now prefix temp namespace for every node in the query.

                int index = 0;

                while ((index = XPathQuery.IndexOf('/', index)) != -1)

                {

                    if (XPathQuery[index + 1] == '/')

                    {

                        index++;

                    }

                    if (!XPathQuery.Substring(index + 1, 4).Equals("soap",

 StringComparison.InvariantCultureIgnoreCase))

                    {

                        XPathQuery = XPathQuery.Insert(index + 1, "response:");

                    }

                    ++index;

                }

            }

            XPathNodeIterator iterator

 = navigator.Evaluate(XPathQuery, nsManager) as XPathNodeIterator;

            if (iterator.Count > 0)

            {

                string val = string.Empty;

                while (iterator.MoveNext())

                {

                    val += String.Format("{0}: {1}", iterator.Current.Name,

iterator.Current.Value);

                }

                e.Success = true;

                e.WebTest.Context.Add(this.ContextParameterName, val);

            }

            else

            {

                e.Success = false;

                e.Message = String.Format(CultureInfo.CurrentCulture,

"No result for query: {0}", XPathQuery);

            }

            return;

        }

 }

6.     IMPORTANT:

XPath expressions do not have the concept of a default namespace.  Omitting the prefix from an XPath looks for nodes in the empty namespace, and finds nothing. So we must use an arbitrary prefix for xml namespace in the SOAP response envelope. This namespace is attribute of the Response tag, which is the first child of the soap:Body tag.

7.     XPath Query Tools:

https://www.stylusstudio.com/xpath_editor.html

https://www.freewarefiles.com/program_4_50_14082.html

https://www.topxml.com/xselerator/

Load Test Modeling: How to Simulate User Logon and Execute Tests in Test Mix

The following sample shows how to model the common scenario where a user logs on once to a Web site and performs a number of tests.  

In the sample, the logon action is definded by a set of login requests. Each Web test contains the same set of login requests plus its own non-login requests.

A “new user” as defined by “Percentage of New Users” under the Load test scenario property always executes Login once and one iteration of a Web test (the non-login section)  from the Test Mix. A “return user” always executes Logon once and a random number of Web tests (the non-login sections) from the Test Mix.

Modeling can be achieved by using plug-ins or coded web tests .

Understanding the Visual Studio Load Agent Controller

Ed.

            Visual Studio 2005 Team Edition for Software Testers

          Visual Studio 2005 Team Test Load Agent

Summary: A deeper Look at settings and how the controller in Microsoft Visual Studio Team Load Agent 2005 works.

Introduction

The Microsoft Visual Studio Team Load Agent 2005 Controller (controller) is a Windows Service that can manage one or more agent machines.  The configuration settings of a test run and attributes of the test cases in the run determine the actions and scheduling the controller will perform.  This document will describe the algorithms the controller runs through to start and monitor test runs.

What Happens At Startup

Configuration File

When the controller service starts it first opens the configuration file, QTController.exe.config.  The settings in the configuration file determine what port the controller should listen on, the working directory for temporary files, whether to create a log file or not and a few other items described here: 

• WorkingDirectory: Base directory for staging deployment and reverse deployment files. Setting this value overrides a call to ‘Environment.SpecialFolder.LocalApplicationData’.  The directory you specify here must already exist.  Subdirectories are created for each test run submitted to the controller and have the form: VSEQT\QTController\<GUID>.  The GUID is a generated value and uniquely identifies each test run.
• CreateTraceListener: Indicates to the service it needs to create a trace log (VSTTController.log ) file in installation directory. Default: no, override with "yes".  When trouble shooting issues with test runs, creating this log file is very helpful, although it can get fairly large quickly depending on the trace level.
• ControllerServicePort: TCP port used by controller.  Changing this value will require you to change the controller port the agent service uses to connect to the controller.  These values must match.
• BindTo: the name of the NIC to bind to for communication.  This allows you to dedicate a single NIC card to test tools infrastructure communication and any other cards for load tests and other network activities.
• AgentSyncTimeoutInSeconds: The number of seconds before the agent is considered unresponsive synchronizing test run start.  The test run is aborted when this timeout period is hit.  This applies only to load tests.
• LogSizeLimitInMegs: Maximum allowable size of the VSTTController.log file.  Setting this value to ‘0’ allows size limited by disk space.  The default is 20megs.  When the log file hits this size it is backed up and a new file is created.  If there is already one back up it is overwritten.  The controller monitors the log file size on 20 second intervals, so it is likely the file will not be backed up at exactly the file size limit.
• ControllerJobSpooling: Set to false to turn off automatic test run spooling.  When spooling is enabled test run events are captured and stored on the controller.  This feature allows you to disconnect from a test and reconnect at a future time through the ‘Test Runs’ window in Visual Studio, even if the test run completed.  All the events are played back to the connecting instance of Visual Studio.
• ControllerUsersGroup: When the controller is installed setup creates a group called ‘TeamTestControllerUsers’.  By default this setting references that group.  This is a group the controller will do role checking against.  Any user added to this group can submit, stop and view owned tests.  They can not view details or stop another users test runs.
• ControllerAdminsGroup: A setting used for administrative role checking.  By default the setting references a group created during setup, ‘TeamTestControllerAdmins’.  Users in the group can submit, stop and view any test and perform administrative actions.  Note that ‘Administrators’ on the controller machine can perform these same actions.

If any of these settings are changed, the controller service needs to be stopped and restarted.  Removing some settings will cause the controller to default to the value specified in the configuration file.  Removing functional settings such as the ‘ControllerServicePort’ will keep the controller service from starting.  Any event keeping the controller from starting will be logged in the event viewer. Some events may be difficult to diagnose the exact cause.  For instance, if any invalid xml formatting is used an error will be logged, but the exact line or value is not always determinable. Removing the settings ‘ControllerUsersGroup’ and ‘ControllerAdminsGroup’ effectively turns off role validation.  It is recommended that you use role validation as a security emasure.

Here are the default values set in the qtcontroller.exe.config file:

  <appSettings>

     <add key="LogSizeLimitInMegs" value="20"/>

     <add key="AgentSyncTimeoutInSeconds" value="120"/>

     <add key="ControllerServicePort" value="6901"/>

     <add key="ControllerUsersGroup" value="TeamTestControllerUsers"/>

     <add key="ControllerAdminsGroup" value="TeamTestControllerAdmins"/>

     <add key="CreateTraceListener" value="no"/>

  </appSettings>

Say you want to change the working directory to, “c:\ControllerWorkingDir”.  First create the directory on the controller machine.  And then add the ‘WorkingDirectory’ key to the ‘<appSettings>’ section in the config file:

  <appSettings>

     <add key="WorkingDirectory" value="c:\ControllerWorkingDir"/>

     <add key="LogSizeLimitInMegs" value="20"/>

     <add key="AgentSyncTimeoutInSeconds" value="120"/>

     <add key="ControllerServicePort" value="6901"/>

     <add key="ControllerUsersGroup" value="TeamTestControllerUsers"/>

     <add key="ControllerAdminsGroup" value="TeamTestControllerAdmins"/>

     <add key="CreateTraceListener" value="no"/>

  </appSettings>

Agent Settings

After reading the settings above the controller reads the agents it will manage, and the assciated agent attributes (persisted in qtcontrollerconfig.xml).  There is a user interface for modifying this file so you should not need to modify it by hand.  Any agent not in this list is ignored if it tries to connect to the controller.  It is denied the connection if the user the agent is running as is not in the ‘TeamTestAgentService’ group.

Submitting a Test Run – What Happens?

When a test run is submitted to the controller it is placed in a queue. If any files are deployed with the test run a deployment job is created.  Deployment must complete before a test run is placed into a ‘ready to execute’ state.  If a problem occurs during deployment from the host application, such as Visual Studio, the test run will be aborted.

Deployment

If files are deployed as part of the test run, the controller does not use a regular file copy buts copies the data over the connection from VS. The path of each file in the file list is relative to the client machine (the machine from which the test run was submitted).  The controller calls back into the host application and asks it for a block of the file currently being copied.  The files are copied into a directory under the WorkingDirectory. Using the example above:

c:\ControllerWorkingDir\VSEQT\QTController\620bd97a-94aa-401e-b58e-0ec1c09558ca\deployment

Once all the blocks of a file has been copied, the same steps are taken for the next file.  Once all the files are copied the test run is placed in a ‘ready to execute’ state.  When the test run begins to execute, the same deployment process occurs between the controller and each of the agents the test is submitted - before actual test execution can begin.

When an agent completes the test run it may return files back to the controller, these are stored here:

c:\ControllerWorkingDir\VSEQT\QTController\620bd97a-94aa-401e-b58e-0ec1c09558ca\results\<agent_machine_name>

Test Run Scheduling – Part 1

After completion of the deployment stage the test run will remain idle until it is popped off the queue for execution.  Scheduling the test run is determined by a number of variables contained in the test run configuration and the list of test cases in the test run.

Initially the controller splits the test cases up into a set of tasks.  This allows for the possibility of multiple test cases to execute simultaneously.  Based on what was just stated, a test run can be thought of as a job, and the job manages a set of one or more tasks.  Before you go further some terminology needs to be defined.

Terminology

• Bucketing is the term used to describe a way to split up a large number of non-load tests.  Simply put, it is a subset of the tests specified in a test run.

• BucketThreshold - is the number of non-load tests in a test run before buckets are created.  The default value is 1000. This is really optimized for unit tests – small, fast running tests. Consider a much smaller bucket size for slower tests as it will increase the parallelism achieved in the run.
• BucketSize – the size of each bucket once the threshold has been hit. The default value is 200.

An agent can only execute one test at a time, and should a catastrophic error occur during execution the run may be aborted leaving many tests unexecuted.  Bucketing helps in a couple of ways: a) it allows the test cases to execute simultaneously on different agent machines; b) if a catastrophic error occurs during execution, only the remaining tests in a bucket will go unexecuted.

One bucket at a time is sent to each agent that has been selected to execute the test run.  If more than one agent is available to execute a test run, then more than one bucket can be executing at a time.  As an agent completes the execution of a bucket, it is sent another if more exist.

If 1000 or more non-load tests exist in a submitted test run, buckets of size 200 will be created.  You can change these values, in a test run configuration file, but note there is no user interface to make the changes.  In the test run configuration file (*.testrunconfig), the following two tags exist:

<bucketSize type="System.Int32">200</bucketSize>

<bucketThreshold type="System.Int32">1000</bucketThreshold>

You change at your own risk – do not save the file without making a back up.  If the file is saved with invalid xml formatting the file may become unreadable and you will not be able to run tests with this test run configuration.

• A Controller Job (job) is the executing component of a test run. A job manages a set of tasks that need to happen for a test run to execute and complete.

• A Controller Task (task) is managed by a Controller Job.  It manages a specific set of tests and agent selection to execute those tests.  The set of tests can be a bucket as defined above, a load test, or a simple test list. Tasks can execute independently of one another as long as the agent resources exist.

Test Run Scheduling - Part 2

When a job is scheduled to execute it first creates a set of tasks to execute the tests within a test run. Tasks are created using the following algorithm:

• If the test is a load test type, a task is created to manage the load test.   Tasks manage a single load test.  If there is more than one load test in a test run, a task is created for each.
• Once the load tests have had tasks created the remaining number of tests is checked.  If the remaining total is greater than the bucketThreashold, then the tests are broken up into buckets of size bucketSize.  Each bucket gets its own task.
• If agent constraints are specified so agents meeting specific criteria are used, this will override bucketing.  For instance, say 4 available agents meet the selection criteria and there are 100 tests in the test run.  Each agent will get 25 tests and they will execute simultaneously.

For example, say you have a test run with the following tests in it:

• LoadTest1
• UnitTest1
• LoadTest2
• OrderedTest1
• UnitTest2-UnitTest999

Task1: LoadTest1.

Task2: LoadTest2. 

All the non-load tests total 1000 (UnitTest1-UnitTest999 + OrderedTest1).  Five tasks would be created:

Task3: OrderesTest1 + 199 Unit tests

Task4: 200 unit tests

Task5-7: 200 unit tests each

If there are enough agent machines, it is possible Tasks3-Task7 will execute simultaneously.  Load tests use all available agents so Task1 and Task2 are idle until all agents are available to execute.

Multiple test runs can execute at the same time.  The initial test run popped from the queue will always have first rights to any agents meeting the test run configuration criteria.  Subsequent test runs can use any agents still available that meet test run configuration criteria associated with the test run.  Note that an agent must be marked online to be available for execution.

See section ‘Miscellaneous’, ‘Examples of Agent Selection and Scheduling’ for some example test run submissions.

Executing Test Types

Load tests require a synchronized start.  As each agent selected to execute a load test completes any deployment it signals the controller it is ready to execute the test.  Once all the agents have entered this state, the controller will send a start event to all agents.  The agents will then begin execution.  Load tests will use a thread-pool when executing Web Tests and multiple threads to execute non-Web tests.  This allows multiple tests to execute simultaneously.  This is all managed by the test adapter plug-in.

Non-load tests do not require a synchronized start.  If more than one agent is used to execute non-load tests they will begin executing immediately after deployment, no matter the state of the other agents.  These tests are executed synchronously one after another.

Monitoring Test Runs

The controller task is charged with monitoring the test list the job assigned it – making sure the tests/agents remain responsive.  It does this by monitoring events returned by the agents.  If an agent does not return a test result within the timeout period specified in the test run configuration, the task will attempt to abort the current test case and execute the next one on the agent.  This applies to all test types.  Note that load tests send aggregated result messages back typically every few seconds.

The controller job is charged with monitoring the overall execution time of the entire test run.  If the test run does complete within the specified run timeout, the job will stop all the tasks and eventually the entire run.

Timeouts can be accessed from the Visual Studio ‘Test’ menu.  Click ‘Test’, selected ‘Edit Run Configurations’ and select the run configuration you want to edit.  A dialog will appear, in the left pane select ‘Test Timeouts’.

Miscellaneous

• Connecting to the Controller

Firewall and some proxy devices can block the communication between the controller and agents, or the controller and Visual Studio.  Be sure to open the listening ports used for the controller and agent to allow communication.  See the InstallGuide.htm on the installation CD for more information on this. As noted above the listening port setting for the controller is in the qtcontroller.exe.config file. The setting for the agent is in it’s respective configuration file, qtagentservice.exe.config or qtagentserviceui.exe.config. 

By default Visual Studio uses any available port to communicate with the controller.  Since the port can change from run to run, you would need to open all ports on your firewall device for proper communication between the controller and Visual Studio. You can set a specific port or port range for Visual Studio to use when communicating with the controller by doing the following:

You can tell Visual Studio to use a specific port by adding this registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\MICROSOFT\VisualStudio\8.0\EnterpriseTools\QualityTools\ListenPortRange

After creating the key, add two new DWORD values:

PortRangeStart

PortRangeEnd

To use a single port you can just specify a value in ‘PortRangeStart’.  If you specify a value in ‘PortRangeStart’ and ‘PortRangeEnd’ Visual Studio will attempt to listen on the ports starting from ‘PortRangeStart’ to ‘PortRangeEnd’. Once it successfully opens a port it will stop looking for a port in the port range.

If there is a proxy or firewall device between Visual Studio and the controller machine you will need to open the port or port range you add.

You can verify Visual Studio is listening on the specified port or a port in the specified range by:

netstat -a

You should see one of the ports you specified in a listening state.  For example:

TCP    <machine>:6905          mymachine.microsoft.com:0  LISTENING

• Examples of Agent Selection and Scheduling

For more information of ‘Agent Constraints’ see Distributed Functional Testing with Visual Studio 2005 Team System

Here are some examples the controller scheduling and agent selection given various test run configurations:

Controller is managing 3 agents

• Example 1

bucketSize  = 5

bucketThreshold  = 6

Using a mix of 12 Unit and Web tests.   

No agent constraints specified.

Submit the test run.

The test list is split into 3 buckets of sizes 5, 5, and 2 and the buckets are executed simultaneously.

• Example 2

bucketSize  = 5

bucketThreshold  = 6

Using a mix of 12 Unit and Web tests.   

The same name/value attributes are assigned to each agent.  All 3 agents meet the selection criteria. 

Submit the test run.

The test list is split into 3 buckets of sizes 4, 4, and 4 and the buckets are executed simultaneously.

• Example 3

Using a mix of 12 Unit and Web tests.  

The same name/value attributes are assigned to two agents.  Two agents meet the selection criteria. 

      Submit the test run.

      Only the two agents are used for the test run.

• Example 4

Using a mix of 12 Unit and Web tests. 

The same name/value attributes are assigned to two agents.  Two agents meet the selection criteria. 

Submit the test run. 

While the test is executing submit another test run with 10 more Unit and Web tests.  This run will execute on the unused third agent.

• Example 5

Using a load test.

The same name/value attributes are assigned to two agents.  Two agents meet the selection criteria. 

Submit the test run.

• Example 6

Using a load test.

The same name/value attributes are assigned to two agents.  Two agents meet the selection criteria. 

The load test is executed on the two agents meting the criteria.

Submit the test run. 

The load test is executed on the two agents meting the criteria.  While the test is executing submit another test run with 10 more Unit and Web tests.  This run will execute on the unused third agent.

• Example 7

Using a load test.

The same name/value attributes are assigned to two agents.  Two agents meet the selection criteria. 

The load test is executed on the two agents meting the criteria.

Submit the test run. 

The load test is executed on the two agents meting the criteria.  While the test is executing submit another load test with no agent constraints specified.  This run will idle until all agents become available for execution.

• Example 8

Using a load test.

The same name/value attributes are assigned to two agents.  Two agents meet the selection criteria. 

Assign a different name/value attribute to the third agent.

The load test is executed on the two agents meting the criteria.

Submit the test run. 

The load test is executed on the two agents meting the criteria.  While the test is executing submit another load test with agent constraints specified that match the third agent.  This run will execute without waiting for the other two agents to become available.

Conclusion

The Microsoft Visual Studio Team Load Agent 2005 Controller offers a lot of flexibility for executing test runs.  A combination of the test run configuration and the settings in the qtcontroller.exe.config file allow a number of flags to adjust.  The controller will use the settings as hints for agent selection, the scheduling of test runs and the scheduling of test cases to utilize the agents efficiently.  It will monitor the test runs and take appropriate action if a test or test run gets into a bad state. 

Distributed Functional Testing with Visual Studio 2005 Team System

We'll also be posting different articles about web and load testing, as well as using a test rig.

Ed.

Applies to:

          Visual Studio 2005 Team Edition for Software Testers

          Visual Studio 2005 Team Test Load Agent

Summary: This article describes the features of Microsoft Visual Studio Team Test Load Agents that can be used for distributed functional testing.

Introduction

Belying its name, Team Test Load Agents were designed to run both functional tests and distributed load tests. It is easy to set up a “rig” consisting of a controller and set of test agents. The agents can then be used both for distributed load testing and for distributed functional testing.

The key features that agents provide for load testing are:

•        Scale out load generation

•        Efficient load generation (1000 users from a typical machine running a typical web test)

•        IP Switching for testing web farms

•        Per CPU licensing (no vUser licensing)

•        Performance counters collected on Load Agents and machines under test

•        Data (stats and logs) rolled up in central SQL Server for analysis

•        Load Test Monitor enables monitoring a test in progress

The key features agents provide for running unit and functional tests are:

•        Parallel distributed functional test execution

–       Run a large set of unit and functional tests at the same time

–       Agent selection enables matrix testing

–       Code Coverage

–       Better quality of service: test restart if test process dies

•        Run tests from VS or Command Line

•        Deployment of test code, reverse deployment of logs/test results

•        Share test lab hardware with a team

–       Multiple clients can submit test runs

–       Controller queues runs and manages agents

•        Platform for test tools

–       Extensible architecture supports third party test types

•        Team Foundation Server integration

–       Reporting

–       Work Item tracking

–       Build Server

–       Source Code Control

Visual Studio Team Test Edition can execute tests locally, which can cover many needs of verifying product code. 

This article focuses on the feature that can be used to perform distributed functional and unit testing on a rig.

Test Run Configuration

Test Run Configurations are files that keep different settings for how you wish to execute tests.  There are many settings that can affect the data collected and how the test will be executed.  The focus here will be on settings that can affect the execution of a test run: timeout values, executing locally, executing remotely, setup and cleanup scripts, executing with specific agent machines, and deployment.

You can have many configuration files within a single solution. This allows you to quickly select a group of settings for a set of given tests.  Any Test Run Configuration can be used for any selected set of tests. 

The Test Run Configuration dialog is accessed from the ‘Test’ menu.  Selecting the ‘Test’ menu and clicking on the ‘Edit Test Runs Configurations’ one or more available Test Run configuration files will be available. 

Local and Remote

Selecting ‘Local’ the test will run in a separate test process called VSTestHost.exe on the same machine as VS. VSTestHost.exe will run under the same user account running Visual Studio.  Running locally allows you to do much the same testing you would do in a remote environment.

Some things to note:

·        With this configuration the execution of tests is affected by any locally running processes, which may skew results. 

·        You can not disconnect from a test and reconnect at a later time.  That is, if you stop VS the test will stop.

·        You can do User Interface (UI) testing.

·        If the test crashes VSTestHost.exe there is no restart functionality.

Selecting ‘Remote’ you need to also type in the IP or computer name of the machine the Microsoft Visual Studio Team Load Agent controller has been installed.  This controller may have one or more agents it manages.  You will see later how to determine what agent it manages.  The controller may reside on the local machine or may be in a remote lab (ensure your firewall has exceptions for Visual Studio and the controller so you can connect to remote machines).  You can also specify agent properties to select which agents to run your tests on. We will cover this feature in more depth later. 

Once you select a controller and save the Test Run Configuration any tests executed using this Test Run Configuration will be sent to the specified controller using the specified settings.

Deployment/Reverse Deployment

Deployment allows you to select files you want copied to the working directory on the agent(s) and available for your tests to use during execution.  Bring up the Test Run Configuration dialog.   Click on ‘Deployment’ in the left pane.  You can add a single file by clicking ‘Add File…’ or add all the files in a given directory by clicking ‘Add Directory…’ Target assemblies are deployed using this mechanism.  This is useful if you have a special file containing data you need for the product code (also referred to as code under test) or data files for your tests (say a file with a list of users).

You can find the file location using the test context passed into your test.

string deployDir = this.TestContext.TestDeploymentDir;

This will return a path to the deployment files.  Add your file name to the path returned and you can open your file for reading.

Deploying files from the agent(s) back to the local machine is called Reverse Deployment.  To have a generated file returned from the agent(s) create or copy your file at the end of the test execution to the path returned by this variable:

string logDir = this.TestContext.TestLogsDir;

Setup and Cleanup Scripts

Setup and Cleanup scripts are deployed to agent(s) and executed before execution of a test run and/or at the end of a test run respectively.  Bring up the Test Run Configuration dialog.  Click on ‘Setup and Cleanup Scripts’ in the left pane.  To specify a setup or cleanup script type in the path to a *.bat or *.cmd file or click on the appropriate ‘…’ and select the file in the file list. 

The scripts are sent as strings to the agent(s).  The agent(s) take the string and generate a setup (qtsetup.bat) and cleanup (qtcleanup.bat) script file as appropriate, adding test run specific environment variables before executing your commands.  For example, say you specify myCleanup.cmd as the cleanup script.

myCleanup.cmd:

echo CleanupScript

echo %AgentName%

The final script on the agent will look like this:

REM ****************************************************************************

REM *  This is a generated file by Quality Tools Agent Service

REM *  Copyright(c) Microsoft Corporation 2004

REM * 

REM ****************************************************************************

set TotalAgents=1

set ControllerName=ERICNWA2

set AgentName=ERICNWA2

set AgentWeighting=100

set TestLogsDir=D:\Documents and Settings\LocalService.NT AUTHORITY.000\Local Settings\Application Data\VSEQT\QTAgent\d93be121-5407-47ce-84b1-61b52dbec744\Results

set TestDeploymentDir=D:\Documents and Settings\LocalService.NT AUTHORITY.000\Local Settings\Application Data\VSEQT\QTAgent\d93be121-5407-47ce-84b1-61b52dbec744\Deployment

set AgentId=1

set TestDir=D:\Documents and Settings\LocalService.NT AUTHORITY.000\Local Settings\Application Data\VSEQT\QTAgent\d93be121-5407-47ce-84b1-61b52dbec744

REM ****************************************************************************

REM *  User Commands

REM * 

REM ****************************************************************************

echo CleanupScript

echo %AgentName%

Notice the executed cleanup script in this example contained the two ‘echo’ lines which are the ‘User Commands’ sections above.  These are the script commands from myCleanup.cmd. The environment variables in the first section of the script file (shown in blue) are test run specific.  Meaning they are only valid for that test run, other test runs will have a different generated ID.  A setup script would have the same environment variables set at the top of the script as the example cleanup script. 

Note that these variables are not available when the tests is actually executing, they are only available during the execution of the user script.

Agent Selection

Agent Selection is one of the more powerful features of the controller.  By entering constraints on what properties an agent must have for the given Test Run Configuration you get a lot of flexibility utilizing resources.  Some example selection criteria: 

·        Only use agents that have Windows XP sp2

·        Only select agents that have software package Y installed

·        Only use agents that belong to my group

·        Use all Windows 2000 machines no matter what SP is installed

It was mentioned above when you selected a controller in the Test Run Configuration, you could also specify agent properties.  Click on the ‘Test’ menu and click ‘Edit Test Run Configurations’ and select a run configuration.  The test run configuration dialog will come up. Select ‘Controller and Agent’ in the left pane.

The agent properties grid is unavailable if a remote controller is not selected.  Select remote if it is not.

The grid represents name value pairs each agent must have defined to be selected for executing a test run using this test run configuration.

Say we have the grid defined as follows:

OS                WindowsXp

SP                SP1

Group A

When a test run is submitted using this test run configuration, any agent with the same above properties assigned will be selected to execute the test run.  Not necessarily all of them are used.  All the properties specified above must exactly match those defined for each agent; otherwise, the agent is not selected.  If the agent only has two of the above name value pairs defined, it will not be a selectable agent.

You can have policy that test runs submitted by a certain group must have A, B, or C defined for the ‘Group’.  This would allow multiple teams to share a agent lab through a single controller.   The above example is stating that the agent must have a property name of ‘OS’ and it’s value must be ‘WindowsXP’, AND it must have a property defined ‘SP’ with a value of ‘SP1’, finally the agent must have a property, ‘Group’ with a value of ‘A’.

You could set up a test run configuration to select all agents having the property named ‘OS’ with a value ‘WindowsXp’.  In the ‘Name’ column type ‘OS’ and in the value column, type ‘WindowsXP’. 

This is the only row of data needed to get all agent machines with the same property and value.  Agents may have many name/value pairs defined, name/values pairs that apply to other test run configurations.  In order for them to be selected for a test run they need to match the name/value pairs entered in this dialog grid.  You enter the agent name/value pairs in the ‘Administer Test Controller’ dialog.  See below for details.

Test Timeouts

Bring up the Test Run Configuration dialog.  Click on ‘Timeouts’ in the left pane.

There are two types of time outs available for any given test run; a) total execution time of all tests in a given test run; b) execution time of individual tests.  Total execution time of the entire run allows a test run to be aborted if the run exceeds the amount of time specified.  Your organization may have a policy that states verification tests must complete in 20 minutes.  Any tests that were not executed before this timeout will be marked as ‘not Executed’.

Individual test timeouts will cause an individual test to be aborted if it exceeds the execution time specified.  The next test will be started and the run will continue to execute.   The default is 5 minutes.

Note: Load tests do not use these timeout values, except in the case a unit test is added to a load test.  Unit tests will have the individual timeouts applied if a value is provided in the dialog box.

Administering the Test Controller

Administering the controller and setting agent properties is handled through the ‘Administer Test Controller’ dialog.  To access, click on the ‘Test’ menu and select, ‘‘Administer Test Controller’.

Initially this dialog will come up with a controller: <local>.  ‘<local>’ is the local machine, not a controller installed on the local machine.  You can only delete temporary files from the local machine using this dialog.

Enter the name of a computer that has the Microsoft Visual Studio Team Load Agent 2005 controller installed.  If there are not agents assigned to the controller you will get a pop indicating no agents are configured.

Agent Status

The initial dialog shows the agents the controller manages and their status.  Possible status values:

Ready: The agent is ready to execute a test

Running test: The agent is currently running a test.

Disconnected: the agent is marked online, but is disconnected.  Here might be why:

·        The agent service or agent service UI version may not be running

·        The agent can not connect to the controller (firewall, security settings)

·        The agent may not have restarted yet.  Click the ‘Refresh’ button

Offline: the agent is marked offline

Editing Agent Properties

Once you have added an agent to controller, you can set the agent’s properties in the Administer Controller dialog. Agents are automatically added to a controller when the agent is setup. Agent properties are:

Weighting: This affects the load generated on the agent.  Each agent is assigned a weighting, faster machines should be given a bigger weighting.  A bigger weighting indicates the agent is capable of simulating a larger portion of users for a load test.  For instance say you had the following two agents with the specified weightings and the number of users you want to use for my test:

Agent1 Weighting: 50

Agent2 Weighting: 100

Total weighting: 150

Agent1 will get 1/3 of the load, Agent2 will get 2/3.

So if the load = 1000:

Number of users Agent1 would simulate: ((50 * 1000) / 150) = 333

Number of users Agent2 would simulate: ((100 * 1000)/ 150) +1= 667

If all the agents have the same weighting, then the user distribution is the same on each agent.

IP Switching:  Will allow agents to cycle through an IPv4 address range during a load test.  This allows you to effectively test a web farm that may have a load balancer distributing requests to the backend web servers.  If all the requests were coming from a single all IP they would be routed to the same web server each time.

NIC: This is the network interface card to use to add IPv4 addresses.  Note: if the machine status is not ‘Ready’ the available NIC cards will not appear.  Close this dialog and mark the agent as online and make sure the state is ‘Ready’ and reopen this dialog.

Base Address: Enter the first three segments of you Ipv4 address.  For example: 192.168.0

Start Range: This is the starting number to use in the IPv4 address range.  Say the value it 10.  When the agent begins generating IPv4 addresses with will start with base address and this number: 192.168.0.<start range> = 192.168.0.10

End Range: This is the last number used in the IPv4 address range.  Say it is 15.  The agent will generate addresses from the Start Range to the End Range:

     192.168.0.10

     192.168.0.11

     …

     192.168.0.15

Subnet Mask: The subnet mask to use with the generated IPv4 addresses.  For example: 255.255.255.0

Notes: if the static or DHCP address of the machine is in the range specified here, it will cause an error and not all of the addresses will be added to the NIC card.  Also note if the IPv4 address specified is not routable in your network, you will not see any load generated by the client.

Attributes:  These are name/value pairs assigned to the agent.  These name/value pairs are used for agent selection when the test run configuration has agent constraints specified.

Attribute Name: The name of the attribute you want to use for agent selection, for example, ‘OS’.  Refer to ‘Agent Selection’ for more context.

Attribute Value: The value for the given name, for example, “WindowsXP’.

Say you have Agent1 and you assign a number of name/value pairs in the grid:

Attribute Name        Attribute Value

OS                                              WindowsXP

SP                                              SP1

Group                               A

Software                           MSOffice

Processor                           MULTICORE

Agent2 has the following attributes:

Attribute Name        Attribute Value

OS                                              WindowsXP

SP                                              SP2

Group                               A

Processor                           SINGLECORE

Using the sample data from the ‘Agent Selection’ in the previous section, you can now select agents for a given test run.

Say you want agents that have ‘Processor’ attribute equal to ‘MULTICORE’:

Click on the ‘Test’ menu and click ‘Edit Test Run Configurations’ and select a run configuration.  The test run configuration dialog will come up. Select ‘Controller and Agent’ in the left pane.  In the agent properties grid, enter the following:

Name                      Value

Processor                 MULTICORE

When a test run is submitted Agent1 will match the agent constraints.

You want agents that have ‘OS’ attribute equal to ‘WindowsXP’

In the agent properties grid in the test run configuration dialog, enter the following:

Name                      Value

OS                                    WindowsXP

Both Agent1 and Agent2 can be used for the test run since both match the agent constraints.

You want agents that have ‘OS’ attribute equal to ‘WindowsXP’ and ‘GROUP’ attribute equal to ‘B’

In the agent properties grid in the test run configuration dialog, enter the following:

Name                      Value

OS                                    WindowsXP

GROUP                              B

Neither agent has ‘GROUP’ attribute equal to ‘B’, both agents have ‘GROUP’ attribute equal to ‘A’.  No agent is returned that can execute the test run and the test run is aborted.

Remove

Clicking ‘Remove’ removes an agent from the agent list.  Note that if the agent is running, it will continue to try to connect to the controller (and fail). So in addition you will need to stop the agent service running on the agent.

Offline/Online

If an agent is marked offline the controller will not attempt to use the agent to execute a test run. You can use offline to perform maintenance on the agent or investigate test results.

Restart

The controller will attempt to restart the agent through the service control manager interface (unless the agent name has an ‘*’ next to it).  Typically the service control manager will stop and restart the agent service. Note that this is the same interface that is used in the Computer Management administrative tool.

If the controller cannot contact the agent (firewall) or the controller service user does not have the rights to stop/start a service on the agent machine, or the agent name has an ‘*’ next to it, the controller will bypass the service control manager and call into the agent service to tell it to exit the process.  If the agent was running as a service, the service controller manager will restart it.  If the agent was running as a normal process (has a ‘*’ next to the name) and was started through the supplied script file, QTAgentServiceUI.bat, the process will be restarted.

It takes a few seconds for the agent to restart and reregister with the controller.  Clicking the ‘Refresh’ button will refresh the agent states in the dialog box.

Refresh

The ‘Refresh’ button causes the controller to query the agents for their state.  Ant state changes are reflected in the main dialog’s agent list.

Delete Temp Files

Test runs will leave temporary files on the controller and agent machines (these may be deployed files, temporary test files, code coverage data files, …).  They are kept in case there is a need to reference them.  It is recommended that the temporary files be deleted when it is known they are no longer needed.

Restart Rig

Restarts the controller service and the agents the controller manages.

UI Testing on an Agent

If you want to run a test that has a UI or interacts with the desktop or user settings, you will need to use QTAgentServiceUI on the agent instead of QTAgentService.exe. QTAgentServiceUI works exactly like QTAgentService, except that it runs as a user process rather than a service. We recommend you configure your agent to automatically logon as a test user, and then in the startup for that user run QTAgentServiceUI.bat. This bat file simply runs QTAgentServiceUI over and over, so that if the agent service dies it will be automatically restarted.

Conclusion

The combination of Visual Studio Team Test Edition 2005 and Microsoft Visual Studio Team Load Agent 2005 allow for a flexible and powerful approach to testing.  The combination allows you to execute tests on the local machine or with a pool of agents managed by a controller.  You can create custom environments on agent machines and execute test runs in those environments using agent selection.  You can control the load of each agent through weighting and simulate many browser requests from different locations using IP Switching. You can deploy custom data to agent machines and return custom data utilizing Deployment and Reverse Deployment.  Through scripting you can further modify an agent environment and revert it to its original state when a test completes.

Visual Studio Team Test Edition 2005 and Microsoft Visual Studio Team Load Agent 2005 give you the tools and flexibility you need to effectively test in today’s complex environments.