Load Test run won’t be able to start successfully and will get aborted with this error:

“The specified module could not be found”

This is known issue with Visual Studio 11 Beta. There is a simple workaround for this issue: Please install Test Agent on your Test Controller box as well and you are done. You need not to do anything extra like configuring installed Test Agent on Test Controller box etc. Just installing Test Agent on Test Controller will help in resolving above issue. There is no side impact of this workaround and it won’t take long also to install Test Agent on Test Controller also.  This issue is fixed for later releases.

Send Your Feedback

If you've any feedback on Visual Studio 11 Beta or if you face any other issue then please do send us on Connect or in the Forums.

Good news is that Visual Studio 11 Beta supports 64 bit Test Controller. You need not to do nothing special to run your Test Controller in 64 bit mode. It will automatically run in 32 bit / 64 bit depending upon architecture of Test Controller box as Test Controller process has been targeted to “Any CPU” platform . So this means that even though you are doing a 32 bit run using 32 bit Test Agent boxes then also your Test Controller might be running in 64 bit mode if Test Controller box is 64 bit box and vice-versa.

Just to re-iterate, Below shown option in Test Settings has nothing to do with Test Controller mode.  It decides about Test Agent mode as explained here. Test Controller mode (32 bit / 64 bit) will be decided by architecture of Test Controller machine.

Please note that Virtual User License Packs for Visual Studio 2010 won’t work with Visual Studio 11 Beta so we are sharing with you Virtual User License Pack for Visual Studio 11 Beta. You can use this Virtual User License Pack: G9WK7-3369B-H6RJD-RQHJ2-T8488.

Please note that above Virtual User License Pack is applicable for just Visual Studio 11 Beta. It won’t work with earlier / later releases of Visual Studio.

Creating Web Performance Test / Load Test

There has been some changes around creating Web Performance Test / Load Test. Now we've separate test project type for Web Performance Test / Load Test which is called Web Performance and Load Test Project as shown below:

Please note that your existing test projects containing Web Performance Test / Load Test will work seamlessly between Visual Studio 2010 SP1 and Visual Studio 11 Beta. But if you are creating new test projects then please use Web Performance and Load Test Project.

This project template will automatically add an empty Web Performance Test to your project. This is considering that most of our customers use Web Performance Test as part of Load Test. You can always delete it if you don't need it.

Once Test Project is created then you can add more web performance / unit / load tests to it by right clicking on test project and selecting appropriate test type:

Above experience of adding new tests to test project is same as we had in Visual Studio 2010 with the difference that now this is the only way to add tests to existing test project as we've got rid of Add New Test dialog as part of overall Visual Studio Test Framework clean up.

Rest of authoring experience of Web Performance / Load Test remains same as it was in Visual Studio 2010 and there has been no changes here.

Running Web Performance Test / Load Test

There has been some changes around the way you run Web Performance / Load Test. This is because of fact that Test View window that we had in earlier versions is no longer there. We've removed it as part of overall Visual Studio Test Framework clean up. But you can continue to run your Web Performance / Load Test from its editor like it was in Visual Studio 2010.

As there was no similar gesture for Coded Web Performance Test so we've enabled running / debugging of Coded Web Performance Test in its context menu. Please see screenshot below:

Validating Tests from Load Test

Sometimes after adding bunch of tests to Load Test we want to run them to see if all of them are able to run together / not without any issues. For this we've enabled a new UI gesture in Visual Studio 11 which enables you to run tests that are part of Load Test. You can either run a single test / multiple tests / all tests that are part of a scenario etc. You can do this via newly added "Run Selected Tests" gesture in Load Test editor context menu as shown below:

Please note that it will simply run tests that are part of load test and won't kick start Load Test run e.g. this is what happened when I used this operation for my simple Load Test which contained a simple Web Performance Test and Coded Web Performance Test:

Running Multiple Tests together

As I mentioned above Test View window is removed from Visual Studio 11 Beta . Additionally Test lists (.vsmdi files) are deprecated in Visual Studio 11 Beta. You will no longer be able to create new test lists (.vsmdi files) or run test lists from Visual Studio so the question is how can we run multiple tests?

You can run multiple Web Performance Tests from Load Test (check above section) as that's the point when we usually want to run multiple tests together. Additionally, You can run multiple tests using command line as explained here.

In case, if you are an existing user and have existing test lists then to maintain compatibility you still will be able to edit the .vsmdi files using Visual Studio. Although you can’t run migrated test lists from with Visual Studio, you can still run them using mstest.exe from the command line. For more information, see How to: Run Automated Tests from the Command Line Using MSTest

Other Miscellaneous Changes

We need to deal with Test Controller / Test Agent frequently in context of Load Test so Manage Test Controller option is moved to Load Test editor as shown below:

It's also available in test setting editor now:

Option to select active test setting for test project is now present in context menu of test setting:

Options to open "Test results" or "Test Runs" window is moved to View->Other Windows as shown below:

Known Issue

If you try to record Web Performance Test using Internet Explorer 10 then first recording may not happen successfully as Web Performance Test Recorder plugin won’t load successfully.  Please note that this issue is just restricted to Internet Explorer 10 and it will come maximum once post Visual Studio 11 Beta installation. We are working over fixing this issue.

Send Your Feedback

If you've any feedback on changes done then please do send us on Connect or in the Forums.

You can specify network mix while creating load test. The network mix gives you a way to simulate load more realistically in a load test scenario. Load is generated by using a heterogeneous mix of network types instead of one single network type. You create a closer approximation of how end-users interact with your applications. You can read here regarding how to specify n/w mix while creating load test.

Right now these profiles are available out of box for a Visual Studio Load Test user:

• LAN (default)
• 3G
• Cable-DSL-1.5Mbps
• Cable-DSL-768k
• Cable/DSL-384k
• CDMA
• Dial-up 56k
• Intercontinental slow WAN 300 Kbps
• Intercontinental WAN 1.5 Mbps
• Intercontinental WAN 1.5 Mbps

You can find *.network file for each of above profile at %ProgramFiles%\Microsoft Visual Studio 10.0\Common7\IDE\Templates\LoadTest\Networks e.g.Cable-DSL 1.5Mbps.network. This file contains all imp settings for a network file like Latency, Packet Loss, Queue Management etc.

You can either modify an existing file / create a new file according to your requirement to create a new custom network profile. After you restart VS, This new profile will be available to you while creating n/w mix in load test so creating a new custom network profile is very simple as such. But imp thing is understanding the various imp parameters available inside network profile that you can tweak according to your requirements. So let’s have a look at them:

Upstream / Downstream LinkRule

upstream network traffic flows away from the local computer toward the remote destination. Conversely, downstream traffic flows to the user's computer. The following are the imp network attributes that can be defined for them:

• Bandwidth
• Latency
• Packet loss
• Queuing behaviour

Let’s cover these network attributes in detail:

Bandwidth

Bandwidth describes “the amount of data that can be transferred to or from the website or server, measured in bytes transferred over a prescribed period of time.” –Wikipedia.  You can specify the bandwidth for the upstream and downstream of a link.  If no bandwidth is specified then network emulation will do nothing to modify the rate at which the packet passes through.

e.g.

<Bandwidth>
    <Speed unit="kbps">1500</Speed>
</Bandwidth>

Latency

Latency is “the time taken for a packet of data to be sent from one application, travel to, and be received by another application. This includes transit time over the network, and processing time at the source and the destination computers.” -Wikipedia.

There are five type of latencies that you can specify:

1. Fixed delay: packets are delayed for a fixed amount of time.
2. Uniform delay: packets are delayed according to a uniform probability distribution.
3. Normal delay: packets are delayed according to a normal probability distribution.
4. Linear delay: packets are delayed for an amount of time that linearly increases from a minimum to a maximum in a given period of time, and the delay time starts all over from the minimum again when it reaches the maximum.
5. Burst delay: packets are delayed according to a given probability, and once one packet is delayed, multiple consecutive packets.

e.g.

<Latency>
  <Fixed>
    <Time unit="msec">150</Time>
  </Fixed>
</Latency>

Packet Loss

In the real world, packets traveling across a network may fail to reach their target and packet loss occurs.  You can use any one of four packet loss patterns specified below:

1. Periodic loss: packet loss occurs periodically, that is, one packet is dropped per given number of packets.
2. Random loss: packet loss occurs randomly in a given loss rate.
3. Burst loss: packet loss occurs according to a given probability, and when a packet loss occurs, multiple packets losses occur consecutively.
4. G-E loss: packet loss occurs obeying the Gilbert-Elliot Model that consists of two states, good state and bad state. Packet loss rates are specified respectively for these two states. And the network transit between the two states is at given transition probabilities.

e.g.

<Loss>
   <Random>
     <Rate>0.035</Rate>
   </Random>
</Loss>

Queue Behaviour

Queuing behaviour describes how received packets are queued and processed. You can specify normal / RED queue in network profile:

• Normal queue: all received packets are put into a First In, First Out queue of a given queue size.
• Randomly Early Detection (RED) queue: all received packets are put into a RED queue. If the queue size is smaller than a given lower threshold, the queue is assessed as not congested and nothing happens. If the queue size is larger than a given higher threshold, the queue is assessed as being congested, and some packet will be dropped based on dropping rules. In between, some packets are dropped according to a probability which is computed based on the lower threshold, higher threshold, and current queue size.

Dropping rules are also defined for queuing behaviour, and the following dropping rules are available:

• Drop front: drop packet at the front of the queue when needed.
• Drop tail: drop the packet at the tail of the queue when needed.
• Drop random: when needed, randomly drop a packet in the queue according to a uniform probability distribution.

e.g.

<QueueManagement>
  <NormalQueue>
    <Size>100</Size>
    <QueueMode>packet</QueueMode>
    <DropType>DropTail</DropType>
  </NormalQueue>
</QueueManagement>

I hope now you’ll be able to create a custom network profile easily. Let me know in case of any Qs / issues. You can post your query here as well.

But still people have some questions / confusions around it because of these reasons:

1)Above link talks about h/w requirements for VS2008 and there is no similar link available for VS 2010.

2) The number of virtual users that an test agent can generate will vary widely from test to test because of variance in think times, response size, response time etc. so people want to get more details here so that they can correlate it with their scenario and accordingly evaluate how much load they can generate using an test agent of a particular config.

Reason for #1 is that There was NO change in h/w requirement for Test Agent / Controller in VS2010 and it remains as of VS2008.

To address #2, We did some baselining runs using IBuySpy sample app. Our pages had average response size of ~32KB, avg response time of 3ms and avg think time used was 3secs. We kicked off multiple runs of different different durations on different config boxes and came up with these h/w recommendations for test agent box:

Number of vUsers	Avg RPS (Req / sec)	CPU	Memory	Comments
<1000 virtual users	<3000	Single Proc Dual Core 2.6GHz	4GB	We can scale up to 3000 or more  vUsers on this config but it will cause dip in RPS. We've seen Avg RPS of 2000 at user load of 3000.
<2000 virtual users	<6000	Single Proc Quad Core 2.6GHz	8GB	We can scale up to 6000 or more vUsers on this config but it will cause dip in RPS. We've seen Avg RPS of 4000 at user load of 6000.
N x 1000 virtual users	Nx3000	Scale out to N agents each with Single Proc Dual Core 2.6GHz	4GB

Please note that the guidelines presented here are provided as general guidance for hardware planning. As mentioned above, Number of vUsers and RPS will vary from test to test depending upon many factors like think time, response size, response time etc.

Let me know in case of any Qs / issues. You can post your query here as well.

It’s bit confusing as 2 different values are there so which one to rely upon for analysis. In this post, I’ll explain why the values are different and which one you can use for deeper analysis.

Actually graph shows you samples at various instances of time which depends upon sampling interval. For each interval, an average is computed for all the requests that were processed during that interval. So, say in the first interval, there are 2 requests processed – one with 2 sec and other with 4 sec, the average for the interval is 3 sec. This is what we show in graph. If the request with 2 sec happens to be the absolute min, the samples graph has not captured it, but the table view will show it. If that interval’s average, 3 sec, is min across all sample intervals, we show 3 sec as min in samples graph. Same explanation goes with max.

In short, graph view shows averages of intervals and hence cannot show absolute min/max. So in case you need absolute min / max data for a counter then table view is more appropriate place to get that data.

Let me know in case of any Qs / issues around it. You can post your query here as well.

1. RPS decreases though user load increases
2. RPS decreases gradually over the duration of run though user load is constant

RPS Pattern like shown below in graph indeed looks confusing:

As part of investigations we found that these issues are caused by:

1) Caching of dependent requests

Please note that dependent requests are always cached even though you’ve turned off caching by setting “Percentage of new users” property to 100%. This blogs talk about it in detail: https://blogs.msdn.com/b/billbar/archive/2007/01/22/simulation-of-browser-caching-in-vsts-load-tests-and-web-tests.aspx

Please note that we are NOT recommending to disable caching of dependent requests. We are just clarifying that RPS dips because of it. Additionally, if user wants to disable caching of dependent requests because of his scenarios requirements then he can do it. Above blog link talks about how to do it.

2)Unhealthy CPU (85%+) on agent box

We’ve observed that if CPU is above 85%+ on test agent box then RPS dips so recommendation here is to restrict CPU on test agent box within 85%.

3) Anti-virus / anti-malware

We’ve observed that anti-virus / anti-malware spikes CPU on test agent box. This is because QTAgent does lots of activity on agent box so anti-virus thinks there is some something suspicious about QTAgent process. So we are recommending here to make sure that no other processes are eating CPU on Agent machines and add QTAgent (/QTAgent32) process in excluded process of antivirus to avoid CPU spike by anti-virus.

I hope this will help you in understanding RPS pattern in your load test runs and taking appropriate action accordingly.

Let me know in case of any Qs / issues. You can post your query here as well.

1) Not able to access performance counter category on a computer

You’ll get this kind of error message in this scenario:

The performance counter category 'Network Interface' cannot be accessed on computer 'ComputerName' (Access is denied) ; check that the category and computer names are correct.

2) Timed Out while reading a performance counter category

You’ll get this kind of error message in this scenario:

The performance counter category ‘Memory’ cannot be accessed on computer ‘'ComputerName' (Timed out trying to read performance counter category ‘Memory’ on computer ‘'ComputerName' ); check that the category and computer names are correct.

There are  are multiple reasons which can cause issues mentioned above / other issues during counter collection. In this blog post, we’ll discuss what causes these issues and how to resolve them.

These are the main reasons which creates problems in collecting counters:

1. Firewall Issue: Performance Logs & Alerts should be in allowed program list in firewall on a box from which you want to collect counter.
2. Permissions issue: User account with which controller is running should be member of  Performance Log Users, Performance Monitor Users, or Administrators group on box from where you want to collect counters.
3. Service Issue: 
   • Performance Logs & Alerts service should be running on box from where you want to collect counter.
   • Remote Registry service should be enabled and running on box from where you want to collect counter.
4. Timeout Issue: Timeout may happen if you are collecting counters from remote / slow box e.g. remote web server. We can increase the timeout by adding this key in controller config file Controller.exe.config:

<appSettings>
<add key="LoadTestCounterCategoryReadTimeout" value="60000"/>
<add key="LoadTestCounterCategoryExistsTimeout" value="60000"/>
< /appSettings>

You’ll find Controller.exe.config inside %ProgramFiles%\Microsoft Visual Studio 10.0\Common7\IDE at your test controller box. In case you are not using test controller and doing a local run then please add above mentioned key in devenv.exe.config which you’ll find inside %ProgramFiles%\Microsoft Visual Studio 10.0\Common7\IDE at your client (VS Ultimate) box.

Above steps should solve your issue but in case it still persists then you can post it here

Exception        LoadTestGoalCounterNotFoundException        1        A goal based load pattern can not be applied because it specifies a performance counter (RUBELSVM1\MyCategory1\MyCounter1\) that could not be read: Category does not exist.

In error message above, MyCounter1 is custom counter, MyCategory1 is counter category and RubelsVM1 is box where this counter is defined. This is how my goal based load pattern looks like:

This error occurs because of a known issue with Dev10 because of which test controller always look for counter on test controller machine for a remote run or on the visual studio client machine for a local run.

To workaround this issue, you need to create the custom category and counter on your test controller box when running remote runs or on the visual studio client box in case of local run.

It is easy to create custom counter on a box. This is the sample code which you can use to create custom category and counter:

using System.Diagnostics;

namespace CreateCustomCounter
{
    class Program
    {
        static void Main(string[] args)
        {
            // Create a collection of type CounterCreationDataCollection.
            System.Diagnostics.CounterCreationDataCollection CounterDatas =
               new System.Diagnostics.CounterCreationDataCollection();
            // Create the counters and set their properties.
            System.Diagnostics.CounterCreationData cdCounter1 =
               new System.Diagnostics.CounterCreationData();
            cdCounter1.CounterName = "MyCounter1";
            cdCounter1.CounterHelp = "help string1";
            cdCounter1.CounterType = System.Diagnostics.PerformanceCounterType.NumberOfItems64;
            // Add both counters to the collection.
            CounterDatas.Add(cdCounter1);
            // Create the category and pass the collection to it.
            System.Diagnostics.PerformanceCounterCategory.Create(
                "MyCategory1", "Category help",
                PerformanceCounterCategoryType.SingleInstance, CounterDatas);
        }
    }
}

Your load test should run fine without any errors after you've created the custom counters on test controller / client box. Please note that you need to run the above program with admin privileges. You can refer here for more details on creating custom counters.

If you have any questions or comments please email the author at: rubels@microsoft.com