In this post we discuss how we load test Visual Studio Online. This post illustrates how vital load testing is to our development lifecycle and to having a healthy service.

Cloud Cadence

First some background on our engineering processes. This is an important context for the rest of the paper.

Our development teams work in 3 week sprints, and at the end of each sprint, in parallel with the first week of the next sprint, we validate the bits and then deploy to the service:

An important principle we have is that our product is ready to ship at the end of the sprint. Verification week is not for finishing up testing new features, it is for doing final verification that we are ready to deploy. We do not want to find new issues during verification week, as our schedules are tight and blocking bugs will knock us off schedule and cause us to miss our deployment day. Once we get off schedule, it ends up snowballing into the next sprint. During verification week, we spend Monday through Wednesday verifying the product in our test environments. If we are green on Thursday we deploy to a pre-production environment, which we verify on Thursday and Friday. Once we have the green light in pre-prod, we deploy to production, which is targeted for Monday morning.

How do we make sure that at the end of the sprint our bits are ready to ship? We keep a constant high level of quality in our working branch. The tests we use during verification week are the same tests we run continuously in our working branch. If a test goes from green to red, we react quickly to get it back to green.

We have a similar approach with our load tests. We never want to stray far from being ready to deploy. However, our load test results require analysis to determine if we are green or red, so we do not run them continuously but we do run them regularly throughout a sprint.

Why Load Testing?

Load testing is a critical part of our software development process. We find many serious issues in load testing, everything from performance regressions, deadlocks and other timing-related bugs, to memory leaks that can’t be effectively found with functional tests. Load testing is critical to being able to confidently deploy.

Visual Studio Online and Team Foundation Server

Throughout this paper I refer to both Visual Studio Online and Team Foundation Server. Several of the services available in Visual Studio Online were previously packaged as an offering named Team Foundation Service (which has been retired as a product name). Most of the capabilities of Visual Studio Online are derived from a common code base that is shared with Team Foundation Server, Microsoft’s on-premises offering for software development teams. Our product team builds both Visual Studio Online and Team Foundation Server, and these offerings will continue to co-evolve. Since they are derived from the same code base, we are able to leverage test assets from TFS to test Visual Studio Online.

Load Testing Visual Studio Online and Team Foundation Server

As you will see, we use load tests to test a variety of different things in VSO and TFS. The TFS team was the very first adopter of Visual Studio load tests back in the 2003 and 2004 time frame, and TFS continues to use Visual Studio load tests for all load testing today. In addition to using load agents to test our on-premises product, we recently began using Visual Studio Online cloud-based load testing service to load test Visual Studio Online.

At the heart of all our load tests are a set of “scripts” that we use to drive load to the various subsystems in TFS. The test scripts we use in our load tests use different test types and target different layers of our architecture, depending on the subsystem they are targeting. TFS has web service end points, rest end points, and web front ends. Visual Studio and other thick clients call TFS via the client object model, while our web clients call TFS via a set of private REST endpoints on the server.

In order to generate the massive loads we need, our test scripts must be very efficient. We use Web tests to drive load to our Web site and Web pages, and “unit” tests to drive load to our Web services. I put the quote around unit tests as these are really API integration tests, not unit tests in the purist sense. Most of our tests drive load at the client OM layer. We have a few tests that target the underlying (non-public) Web services proxy layer because the client OM doesn’t scale well on a single client as we scale up the load.

All test scripts use common configuration to determine which TFS instance to target in the test, and scripts can be configured to target an on-premises server or an instance of Visual Studio Online running on Azure. This is a fundamental point, and key to us being able to re-use the same scripts to drive load to different TFS instances configured in a variety of ways.

We vary the script test mix and load patterns to suit the purpose of each load test. Below in this paper we will go through the different types of load tests we run and their purpose, and as we do we will outline the type of load patterns and test mix we use.

Testing On Azure vs. On-premises

Two critical aspects to successfully load testing TFS are 1) carefully scripting the server to simulate realistic load, and 2) carefully managing our data shape. TFS runs both on-premises and in the cloud on Azure. The code that runs on-premises and in the cloud is largely the same, but the topology and data scale are different. In our on-premises installations, we see scale up of a single instance, while in our cloud service we see scale out of many, many customers and many collection databases.

One difference between TFS on-premises and TFS on Azure is TFS on Azure we have many thousands of customer collections. In our on-premises product, a large installation will have on the order of tens of collections, but each of these collections can have many terabytes of data.

When load testing TFS on Azure, we start by load testing on Azure dev fabric. Azure dev fabric is a high-fidelity emulator that runs in a local development environment. While running on dev fabric is not sufficient, we are able to test a significant part of the product, and as a result catch a significant number of bugs before going to real azure. Azure dev fabric has the advantage that it is fast to set up our large data scale environments, and easier to debug. But of course there are major architectural differences, the most prominent being in dev fabric we use on-premises SQL, whereas on Azure we use SQL Azure. To get to an “interesting” number of collections on dev fabric, we really crank up the tenant density per database.

Once we have stabilized on Azure dev fabric, we move to load testing on Azure. When testing on Azure, we use the Visual Studio Online load testing service.

Here is the test setting we use to our load tests using the Visual Studio Online load testing agents:

And here is the Agent Count property in the load test run settings in the load test, which controls the number of agents our test will run on:

See the help topic here for a complete walkthrough of how to use the load testing service.

Load Generation on Azure

A big difference between testing on Azure and testing on-premises is the load generators we use. When testing our on-premises product, we simply use on-premises load agents. Until recently when we started dogfooding the Visual Studio Online load testing service, when testing on Azure we did not have a good answer for load generators. We tried a lot of different things:

• Load agents on corpnet. Our Corpnet network proxy was the bottleneck and shut us down at relatively light loads.
• Load agents on azure with load controller on corpnet. The network connection between the agents and controller was too flaky, leading to unreliable test runs (run would fail part way in to a many hour run).
• Load agents and controller on Azure VMs. Again the agent and controller network connection wasn’t reliable enough, causing runs to fail. Fundamentally we see more network noise in the cloud, and the connections between the agent and controller weren’t built for the noisy network we see in the cloud.
• Load agents running independently on VMs, test results manually merged. Merging results was painful, but at least it was reliable!

When we are on our tight deployment schedules, and our load runs go for 8 hours, so we must be able to reliably get results from our load tests or we risk deploying with blind spots for failed runs, or delaying the deployment until we get a complete set of results.

Over the past six months we have been dogfooding the Visual Studio Online load testing service which is hosted in Azure. This is a great service that has enabled us to drive load reliably at scale from Azure, solving one of our critical needs for load testing the service.

Data Collection on Azure

Another big challenge testing in Azure has been collecting performance data from the system under test. For TFS, we have the following types of interesting data we collect from the TFS application tier:

• Perf counters. We collect CPU, memory, disk, network, plus TFS-specific counters and ASP.NET counters
• Event log data
• TFS Activity logs. The activity log has a log entry for each call into the TFS web service or REST APIs. It records the command, time of the request and response (so we can compute server response time), the outcome (success/failure), as well as a few other fields.
• TFS ETW traces. These are highly configurable internal traces in our product code that we use to debug issues.

The on-premises Visual Studio load testing product enables you to collect performance counters from the system under test, but in order to do so the load test controller must be able to connect to the machines under test via non-internet friendly ports (read: not port 80 or 443). This simply does not work well against machines running in the cloud, and in fact is not supported by the load testing service. On-premises you can also configure a data collection agent on the machine under test, but installing and configuring this agent is not supported by the load testing service.

On our production service, we have a pretty rich data collection and monitoring system in place for our service that collects all this data and stores it in a data warehouse for mining and analysis.

We leverage this system for our cloud load tests as well. Really I believe this is the right pattern to follow for load testing – use the same system you use to collect data and monitor your production service to monitor your test runs. This has a bunch of advantages: your monitoring system gets tested too, and any improvements you make for your test environments carries over to prod, and visa-versa.

The one exception here is perfmon counters, as we collect those via a different means on the service. We are manually collecting perf counters on the TFS web and worker roles (the system under test) by kicking off perfmon to log the counters to a file before each run.

The downside with this approach is you do not get the nice view in the load test analyzer that ties together performance data collected from the load agents with the perf counter data collected from the system under test. We are able to construct these views using our monitoring system, however.

The load testing service does not solve our problem of collecting perf counters from the system under test, as this is not yet supported. I know they have plans to cover this, but it is too early to disclose them. The load testing service does supply the same rich set of data from the load test agents, and we collect our activity log data using a data collector.

Here is a screen shot of our GoBig load test running on the load testing service.

This is a simplified experience when compared to data we get when running a load test using local load agents. However, we get enough data to determine if the test is running successfully, which is the main thing we are looking for while the test is running. We want to make sure we don’t let the test run for 8 hours only to find it wasn’t configured properly.

Below are a two screen shots of using the Load Test Analyzer to analyze a GoBig load test result. Post-run analysis is the same experience as analyzing a run using local load agents.

Here is the key indicators graph:

And the Tests table with a roll up of test results per test in the load test. Again, this is the same experience whether generating load from Azure or from local load agents.

Direction We Are Headed

I have long been a proponent that you should use your production environment monitoring system to also monitor your system under test during a load test, and am happy that we are doing that in our load testing. That way you are testing your operational environment and operational readiness along with the app. After all, the tools you need to debug a problem during or after a load test are the same tools you need to debug problems in production. This is a key tenet of DevOps.

We are now actively working on adopting our DevOps monitoring solutions for Visual Studio Online. This includes performance monitoring on the app tier, as well as out-side in monitors from Global Service Monitoring. This configuration will enable us to relatively easily configure monitoring in our test environments in the same way as production.

Types of Load Testing We Do

At this point we have described how we do load testing on premises and in the cloud. Now we will move to describing the different types of load tests we run, and what the purpose of each of these tests is.

Performance Regression Testing

One of our most important Load Testing activities each Sprint is to ensure that no performance or scale regressions are introduced when an update is deployed to the TFS Service. The data scale on the service has grown very large, as have our peak loads are we regularly see over 1000 RPS on the service. Many of these requests are for a large amount of data (Version Control file downloads for example) or require a fair amount of SQL processing (Work Item queries for example).

In order to ensure that the service can handle this load, we need to test on a similar configuration with a similar amount of load.  To generate the amount of load needed we need to use the Load Testing Service with multiple load test agents because the Version Control tests included in our load test make heavy use of the local disk to create, edit, check in, and later get files.

By using a large test environment and using the Load Testing Service to generate high load levels, we have been able to identify several performance regressions during verification rather than in production. 

We use the Load Testing service to run a six hour load test with a load of over 800 requests per second. At the end of the run we download the load test result into our on-premises load test results database where we can view and analyze the results in either Visual Studio or in Excel using the Load Test reports plugin.  In Excel we ran the Load Test Comparison report to compare the result of the current Sprint with the previous one. On the sheet that compares the test results we noticed that many of the Work Item Tracking tests in our load test had a higher execution time than the baseline result from the previous sprint:

This led us to further investigation and a fix for this regression.

Scale Testing

One of the big challenges with running the TFS service has been staying ahead of the scale curve. The pattern we have seen is that with each order of magnitude increase in the number of accounts and users, we have encountered new bottlenecks in scale. Logic would tell you that scale is linear and you can predict the next server tipping point by extrapolating out resource utilization on the server to see when you will run out resources, and thus fall over. For example, if each new user takes 10KB of memory, then we can multiply out to see how many concurrent users when you will run out of memory. At that point, you can simply add another AT, or increase your role size on Azure so the machine will have more memory.

However in practice, it is never linear, and the server almost always tips over in ways you didn’t foresee. As a result, it is very hard to predict where your system will fall over. And often there is a lot of engineering work required to overcome the scale bottleneck, so if you wait until it happens in production you will not have enough time to react and keep your site healthy.

To stay ahead of outages due to scale on TFS on Azure, we do targeted scale testing. On Azure, our most interesting scale factor is the number of accounts and users. In order to run interesting scale tests, we need to run against TFS instances with hundreds of thousands of accounts and users. Our initial approach to preparing this test environment was to start fresh and then run the scripts that do account and user creation to build up interesting data sets. As you can imagine, we found a number of stress bugs in our account and user creation code this way, but it still took at two to three days to get an environment populated which was unacceptable.

We changed our strategy for data population to instead create what we called a “golden instance” of the data, which could then be replicated into our test environments. We had already developed the data replication code for our functional tests, now we needed to do it at scale, which was a matter of doing the data population one time and then saving off the data. This has the advantage of being faster and more reliable, and the data has been through the same migration path as data on the service.

In a recent run using the load testing service, we found a large regression in both memory and CPU that was caused by one bug. This table shows the results gathered using perfmon during the load test before and after the bug was fixed:

Deployment Testing

As described above, we deploy to Azure at the end of each sprint. Visual Studio Online is architected into separate services that can be deployed to and serviced independently. The current set of services are:

• Shared Services, which handles user authentication, access control, and account management.
• Visual Studio Online Services, which manages the core TFS functions of work item tracking, version control, and build.
• Machine Pool Management Service, which manages the set of machines we use to do builds for customers on Visual Studio Online.
• Load testing service, which manages running load tests on the machine pool service.

Each service has a unique architecture, but follows a similar pattern of web front ends hosted on web roles, job agents hosted on worker roles, and SQL databases hosted in SQL Azure.

Our deployments are designed to achieve zero downtime to our customers. In order to achieve that, we deploy in three phases:

Phase 1: deploy new binaries to our web front ends and job agents.

Phase 2: Run jobs to upgrade the the databases.

Phase 3: Run jobs to do any necessary modifications of the data stored in the databases.

In order for this to result in zero down time, each service must keep compatibility at the web front end, and more importantly, the service must be able to stay up even through the different phases of the upgrade, which means that new binaries must be able to work with the old db schema and data shapes.

What’s more, customer data is stored in 1000s of databases in SQL azure. The code we use to service the databases actually runs as a set of jobs in the job service. We have found that running these jobs will put a large load and thus a large stress on the service. We need to ensure the service remains operational and responsive to our customers while sustaining the load from servicing.

To test our upgrades, we run a set of functional tests at each phase of the deployment to ensure the parts are compatible. But this isn’t sufficient for measuring downtime. To test the system remains available throughout the deployment, we run a load test while the upgrade is happening. The load test has the advantage of keeping constant activity going against the service so we can detect any outages. In this test it is important that we have a data shape on the test system that is similar to the data we have in production.

In a recent run, we found that we were processing upgrade jobs “too fast”, resulting in loading account specific data into memory for all 120,000 accounts in our test deployment.  This eventually led to running out of memory. Here’s a graph showing how the number of accounts loaded went up to 120,000 accounts, driving the memory used up to 16 GB, running the process out of memory. The short term fix was to throttle the rate of upgrade jobs, while the long term fix was to manage the number of loaded account by memory consumption as well as time, and start unloading accounts on the job agent as we see memory pressure.

Directed Load Testing

Another set of load tests we run are directed tests, aimed at isolating a particular component or service for performance, stress, and scale testing. We have found that targeting load testing at a particular component is a much more effective way to quickly find stress-related bugs in that component, vs. just adding that component into the larger load test that hits everything on the service.

By focusing a load test on a particular component or service we are able to drive more load to it and thus reduce the time to find timing or concurrency related bugs. It also isolates the bugs to that particular service, which helps with debugging. Whenever we are bringing on a new service or introducing a new architectural pattern to an existing service, the team bringing the change needs to do this type of targeted testing before bringing it to production. Recent examples of these changes are introducing Shared Platform Services and the load testing service itself. An example of an architectural change was the introduction of Team Rooms to TFS, or the switch from PaaS build machines to IaaS build machines.

Testing in Production

The larger our service becomes, the harder and costlier it is to simulate scale in our load tests. A fundamental tenet is that we introduce change incrementally to avoid large destabilizing changes. In addition to preventing performance and scale issues, we are spending a lot more time analyzing the rich data we are getting back from production. One of the challenges we face is we collect so much data, how do we make sense of it all?

We systematically are analyzing the following data to look for regressions:

1. Activity log analysis. The activity log logs all calls to a service.
1. Increases in failed command counts
2. Increases in response times, which indicates a perf regression
3. Increases in call counts, which indicates a client chattiness regression
2. CPU and memory usage. We analyze CPU perf counters to look for increased CPU usage after a deployment, and memory to look for increased memory usage as well as memory leaks
3. PerfView analysis. We are now taking PerfView dumps in production after each deployment to find memory regressions and CPU regressions in production. PerfView has become our favorite tool for doing live site debugging.

Generally we want to do enough prior to deployment to ensure the deployment is successful, but we also recognize that we can improve the service a lot by systematically analyzing the rich set of data we are collecting and using that data to drive improvements back into the service.

Conclusion

You can see from this paper how intrinsic load testing is to our software development lifecycle. Regularly running load tests is critical to maintaining a high level of quality during our sprints and deploying every three weeks. The Visual Studio Online load testing service has been a welcome addition to our toolset, as it enables us to reliably generate large loads to our test environments in Azure.

I took on the role as QA manager for the Team Foundation Server (TFS) product team. I now run the TFS QA team, with half my team in being in NC and half in Redmond. While I was sad to let the load testing and TCM products go, I am excited by the new challenges and opportunity afforded in my new role.

Ed.

Second, I want to give people a flavor of what to be prepared for in an interview.

In a typical interview here we ask you to code up some function on the whiteboard and then discuss test cases for the function.

String functions in particular are popular. So something like code a function to reverse the words in a string: “The cat jumped over the fence” to “ehT tac depmuj revo eht ecnef”.

The best approach for coding is to first think about the algorithm. For example, write down on the white board the most common simple example, then think through the algorithm. Once you have that, code it. Typically the coding should be done in about 20-30 minutes, so if it seems your algorithm is not simple you are probably doing something wrong.

Then come up with interesting test cases for it:

A common case: “I went to the park”

Edge cases:

really short: “A”, “”

really long (say a book)

Extra spaces:

“  I went    to    the park  ”

“I went    to    the park”

“  I went to the park”

“I went to the park ”

“ I went to the park ”

Tabs (is this whitespace?)

Test cases with non-alpha characters would also be interesting.

You get the idea.

Another one might be insert an element into a sorted linked list. You can look on the internet for other types of coding questions.

It is also good to think about:

- running on different language OS, or different language VS (say a far east language, or bi-directional language)

- performance (how fast is it) and stress (e.g. memory leak detection)

- usability

and other types of non-functional testing.

You might also be asked to test some piece of software, like the file open dialog in excel, or some object, like a tire, coffee cup, or battery.

I hope that helps, and happy interviewing!

Ed.

Mahipal Kante, the dev lead for load test, leveraged these extensibility points to build a great set of plugins for handling the WCF msbin1 format and JSON format. It enables both editing the parameter values in the editor as well as viewing them in the results viewer.

The beta is available here: https://teamtestplugins.codeplex.com/releases/view/43166.

Ed.

Here is a list of the new and updated topics:

Ed.

https://vsptqrg.codeplex.com/.

I strongly recommend you download this documentation and keep it in “your back pocket”. One thing to note is that in its current form it does not get indexed by search engines, so you need to actually download the pdf.

Here is a list of the new topics:

--NEW-- How cache settings can affect your testing and app performance 19

--NEW-- How a Load Test Plugin is executed on a test rig 29

--NEW-- Sharing State across agents in a load test rig is not supported out of the box 30

--NEW-- Customizing and extending the available network emulation settings 66

--NEW-- New Performance Counters in .NET 4.0

--NEW-- Content-Length=0 Header not sent resulting in HTTP 411 Length Required Error 99

--NEW-- Web and load tests stay in “Pending” state for a long time before starting 99

--NEW-- Error that test could not run because the network emulation is required 100

--NEW-- Error/Crash in “Open and Manage Load Test Results” dialog 100

--NEW-- Calls to CaptchaGenerator.aspx fail during playback 101

--NEW-- Request failure with improperly encoded query strings calling SharePoint 2010 101

--NEW-- Network Emulation does not work in any mode other than LAN 101

--NEW-- Error: Request failed: No connection could be made because the target machine actively refused it 103

--NEW-- MaxConnection value in App.Config is not honored when running a load test 103

--NEW-- Web Test Authoring and Debugging Techniques 104

--NEW-- Troubleshooting Network Emulation 123

--NEW-- Best Practice: Understanding the ROI for test automation 140

--NEW-- How to create guaranteed unique user IDs for UNIT tests 171

--NEW-- How to create a sync point for starting load tests 173

--NEW-- How to set default extensions that the WebTest recorder will ignore 173

--NEW-- Gotcha: VS 2008 and out of memory 174

--NEW-- Gotcha: Timeout attribute in coded web test does not work during a load test 174

--NEW-- How to extend recorder functionality with plugins 185

--NEW-- Stand-Alone Network Emulation and CodePlex 193

Thanks for putting this new content together Geoff!

Ed.

One part of this toolkit is a great new load testing kit for 2010.

The toolkit was initially developed by the SharePoint product team’s perf testing team, and then made available publicly.

SharePoint is a real challenge to do performance testing against (or any test automation for that matter), since both field names and values contain GUIDs. Large SharePoint deployments can contain 1000s of SharePoint sites, so how can you effectively test this? If you only hit the same site over and over it could have a very different performance characteristic than if you spread load across all the sites. And typically some sites are much busier and more popular than other sites.

The load test toolkit handles both of these problems really well. It is particularly targeted initially at migrating a site from 2007 to 2010, and does a great job by reading the IIS log to get site distribution, and by then dynamically determining during a load test which sites to hit. For example, if the load test is testing actions against a document library, a web test plugin will go to the SharePoint site to dynamically select an appropriate document library to target, and then cache the url to the library on the load client. This is a brilliant way to solve both the distribution and parameterization problems. Hats off to Demetrious and Kfir on the SharePoint team for developing the solution, and James and Doron for making it available to customers.

Ed.

Special thanks to Geoff Gray for once again spearheading this effort. Geoff is one of a handful of elite VS load test practitioners. As he encounter unexpected behavior or bugs in the product, or gets undocumented information on the internals or edge cases, he captures the info for the quick reference guide.

Here is a screen shot of some of the really nice artwork in the guide:

Here is a list of topics in the guide (the numbers after each article is the page number):

NOTE FROM THE AUTHOR
HOW IT WORKS
How Web Tests Handle HTTP Headers 9
General Info (including order of execution) of load and web test plugins and rules 9
Client Code does not execute because Web Tests Work at the HTTP Layer 12
File Downloads, Download Size and Storage of files during Web Tests 12
When is the “Run unit tests in application domain” needed? 12
How the “Test Iterations” Setting impacts the total number of tests executed 12
Test timeout setting for load test configuration does not affect web tests 13
How user pacing and “Think Time Between Test Iterations” work 13
Load test warmup and cool down behaviors 13
What is the difference between Unique, Sequential and Random Data Sources 14
Comparing new users to return users 14
Goal based user behavior after the test finishes the warmup period 17
Threading models in Unit tests under load 18
Simulation of Browser Caching during load tests 19
The difference between Load Test Errors and Error Details 20
How parameterization of HIDDEN Fields works in a webtest 21
Testing execution order in Unit Tests 23
How machines in the test rig communicate 25
Changing the Default Port for Agent-Controller Communication 26
How to Add Agents To A Test Rig 26
ITEMS NEW TO VS 2010
“Find” feature now available in Webtest playback UI 27
“Go To Web Test” feature now available in Webtest playback UI 28
Recorder Log Available 29
Add extraction rule directly from the playback UI 30
New “Reporting Name” property for web requests 31
LoadTestResultsTables now differentiate between GET and POST requests 32
Virtual user visualization now available 33
New Excel reporting features built into load test results 39
New Load Test and Load Test Rig Licensing and configurations 40
New test mix: “Sequential Test Mix” 44
Query String and FORM POST URLs get parameterized 46
New options on Load Test Scenarios 47
Loops and Conditionals 48
CONFIGURATIONS AND SETTINGS
How to Change the Location Where Agents Store Run Files 50
How to set a proxy server for web tests 50
How to configure Web Tests so Fiddler can capture playback info 50
Controlling the amount of memory that the SQL Server Results machine consumes 51
How to configure the timeouts for deployment of load tests to agents 51
How to set the number of Load Test Errors and Error Details saved 52
Multi-proc boxes used as agents should have .NET garbage collection set to server mode 53
Location of list of all agents available to a controller 53
NETWORKS, IP SWITCHING, TEST STARTUPS
IP Address Switching anatomy (how it works) 54
Gotcha: IP Address Switching is ONLY for WEB TESTS 54
Gotcha: IP Addresses used for switching are not permanent 54
How to Setup IP Switching 55
Troubleshooting invalid view state and failed event validation 58
Startup: Slowness Restarting a Test Rig with Agents Marked as “Offline” 58
Startup: Multiple Network Cards can cause tests in a rig to not start 59
Startup: Slow startup can be caused by _NT_SYMBOL_PATH environment variable 59
Startup: tests on a Rig with Agents on a Slow Link 60
“Not Bound” Exception when using IP Switching is not really an error 60
How to configure the timeout for deployment of load tests to agents 61
PERFORMANCE COUNTERS AND DATA
Customizing the Available Microsoft System Monitor counter sets 62
Performance Counter Considerations on Rigs with slow links 64
Increase the performance counter sampling interval for longer tests 65
Changing the default counters shown in the graphs during testing 65
Possible method for fixing “missing perfmon counters” issues 65
How and where Performance data gets collected 66
DATA AND RESULTS
Custom Data Binding in UNIT Tests 67
Verifying saved results when a test hangs in the “In Progress” state after the test has finished 67
The metrics during and after a test differ from the results seen. 68
How new users and return users affect caching numbers 69
data sources for data driven tests get read only once 70
Consider including Timing Details to collect percentile data 71
Consider enabling SQL Tracing through the Load Test instead of separately 72
How to collect SQL counters from a non-default SQL instance 72
How 90% and 95% response times are calculated 72
Transaction Avg. Response Time vs. Request Avg. Response Time 73
Considerations for the location of the Load Test Results Store 73
Set the recovery model for the database to simple 73
How to clean up results data from runs that did not complete 74
InstanceName field in results database are appended with (002), (003), etc. 74
Layout for VSTS Load Test Results Store 74
How to view Test Results from the GUI 75
SQL Server Reporting Services Reports available for download 75
How to move results data to another system 75
Load Test Results without SQL NOT stored 76
Web Test TRX file and the NAN (Not a Number) Page Time entry 77
Proper understanding of TRX files and Test Results directory 78
Understanding the Response Size reported in web test runs 79
ERRORS AND KNOWN ISSUES 80
CSV files created in VSTS or saved as Unicode will not work as data sources 80
Incorrect SQL field type can cause errors in web tests 80
Leading zeroes dropped from datasource values bound to a CSV file 80
Recorded Think Times and paused web test recordings 80
After opening a webtest with the VS XML Editor, it will not open in declarative mode. 81
Calls to HTTPS://Urs.Microsoft.Com show up in your script 81
Possible DESKTOP HEAP errors when driving command line unit tests 81
Goal based load tests in VSTS 2008 do not work after applying SP1 82
Using Named Transactions in a Goal-Based Load Profile can cause errors 82
Debugging Errors in Load Tests 83
Debugging OutOfMemory Exceptions in Load Tests 83
Memory leak on load test when using HTTPS 83
“Not Trusted” error when starting a load test 84
Detail Logging may cause “Out of disk space” error 85
Error details and stack traces no longer available in VSTS 2010 85
VSTS does not appear to be using more than one processor 85
Changes made to Web Test Plugins may not show up properly 85
Socket errors or “Service Unavailable” errors when running a load test 86
Error “Failed to load results from the load test results store” 87
Hidden Field extraction rules do not handle some fields 87
Test results iteration count may be higher than the max test iterations set 87
In flight test iterations may not get reported 88
Completion of Unit Test causes spawned CMD processes to terminate 88
Bug with LoadProfile.Copy() method when used in custom goal based load tests 89
Errors in dependent requests in a Load Test do not show up in the details test log 90
WCF service load test gets time-outs after 10 requests 92
Loadtestitemresults.dat size runs into GBs 92
TROUBLESHOOTING
How to enable logging for test recording 93
Diagnosing and fixing Web Test recorder bar issues 93
User Account requirements and how to troubleshoot authentication 94
How to enable Verbose Logging on an agent for troubleshooting 95
Error that Browser Extensions are disabled when recording a web test 95
Troubleshooting invalid view state and failed event validation 96
Troubleshooting the VSTS Load Testing IP Switching Feature 97
Troubleshooting Guide for Visual Studio Test Controller and Agent 99
HOW TO, GOTCHAS AND BEST PRACTICES
How to call one coded web test from another 111
How to use methods other than GET and POST in a web test 111
How to filter out certain dependent requests 111
How to handle ASP.NET Cookie-less Sessions 112
How to use Client-side certificates in web tests 112
How to remove the “If-Modified-Since” header from dependent requests 113
How to handle custom data binding in web tests 113
How to add a datasource value to a context parameter 113
How to test Web Services with Unit Tests 114
How to add random users to web tests 114
How to add think time to a Unit Test 114
How to add details of a validation rule to your web test 115
How to mask a 404 error on a dependent request 116
How to parameterize Web Service calls within Web Tests 117
How to pass Load Test Context Parameters to Unit Tests 117
How to create Global Variables in a Unit Test 117
How to use Unit Tests to Drive Load with Command Line Apps 118
How to add Console Output to the results store when running Unit tests under load 118
How to add parameters to Load Tests 119
How to Change the Standard Deviation for a NormalDistribution ThinkTime 119
How to programmatically access the number of users in Load Tests 120
How to create a webtest plugin that will only execute on a predefined interval 120
How to support Context Parameters in a plug-in property 121
How to stop a web test in the middle of execution 122
How To: Modify the ServicePointManager to force SSLv3 instead of TLS (Default) 122
How To: Stop a Test in the PreRequest event 123
How to make a validation rule force a redirection to a new page 123
How to add a Web Service reference in a test project 127
How to remotely count connections to a process 129
How to hook into LoadTest database upon completion of a load test 129
How to deploy DLLs with MSTEST.EXE 130
How to authenticate with proxy before the test iteration begins 131
How to enumerate WebTextContext and Unit TestContext objects 132
How to manually move the data cursor 132
How to programmatically create a declarative web test 133
How to modify the string body programmatically in a declarative web test 134
Gotcha: Check Your Validation Level in the Load Test Run Settings 134
Gotcha: Do not adjust goals too quickly in your code 134
Gotcha: Response body capture limit is set to 1.5 MB by default 134
Gotcha: Caching of dependent requests is disabled when playing back Web Tests 135
Best Practice: Blog on various considerations for web tests running under load 135
Best Practice: Coded web tests and web test plug-ins should not block threads 135
Best Practice: considerations when creating a dynamic goal based load test plugin: 136
Best Practice: Add an Analysis Comment 136
Best Practice – Using comments in declarative webtests 136
EXTENSIBILITY
New Inner-text and Select-tag rules published on Codeplex 138
ITEMS NOT SPECIFIC TO THE VSTS TESTING PLATFORM
Using the VSTS Application Profiler 146
VSTS 2008 Application Profiler New Features 146
Using System.NET Tracing to debug Network issues 146
Logparser tips and tricks 147
Logparser WEB Queries 147
LogParser Non-Web Queries 148
OLDER ARTICLES
Content-Length header not available in Web Request Object 149
SharePoint file upload test may post the file twice 149
Some Hidden Fields are not parameterized within AJAX calls 149
(FIX) Unit Test threading models and changing them 149
Bug in VSTS 2008 SP1 causes think time for redirected requests to be ignored in a load test 150
New Load Test Plugin Enhancements in VSTS 2008 SP1 150
Four New Methods added to the WebTestPlugin Class for 2008 SP1 150
INDEX

Ed.

Dynamics AX Performance and Load Testing: Application Benchmark Toolkit

Parameterizing Tests to Run in Different Environments

Load Test Plug-in – Simulate Repeated Peak Load Modeling

Visual Studio Performance Testing Quick Reference Guide (Version 2.0)

I also moved some posts that are specific to VS 2008 into their own section.

Ed.

Howie Hilliker has done a fantastic job enhancing our documentation. We actually have screen shots and artwork in the docs now! Also thanks to the load test product team for reviewing the content.

The documentation has been completely reorganized, with performance testing topics under the Testing Application Performance and Stress topic.

For example, check out these topics:

Walkthrough: Correlating Dynamic Parameters

Load Test Analyzer Overview

Walkthrough: Using a Test Controller and Test Agents in a Load Test

We also have walkthroughs on creating plugins, which is very helpful.

Creating and Using Custom Plug-ins for Load and Web Performance Tests

Here’s a blowout of the table of contents. It is definitely worthwhile to scan through this list to familiarize yourself with what is there.

Testing Application Performance and Stress

• What's New for Web Performance and Load Testing
• Getting Started with Load and Web Performance Test Walkthroughs
• Web Performance Test Walkthroughs
• Walkthrough: Creating a Simple Web Application
• Walkthrough: Recording and Running a Web Performance Test
• Walkthrough: Adding Validation and Extraction Rules to a Web Performance Test
• Walkthrough: Adding Data Binding to a Web Performance Test
• Walkthrough: Adding Data Binding to a Web Performance Test (CSV File)
• Walkthrough: Adding Data Binding to a Web Performance Test (XML File)
• Walkthrough: Creating a Coded Web Performance Test
• Walkthrough: Creating a Web Performance Test that Requires a Login and Logout
• Load Test Walkthroughs
• Walkthrough: Creating and Running a Load Test Containing Web Performance Tests
• Walkthrough: Creating and Running a Load Test Containing Unit Tests
• Creating and Editing Load and Web Performance Tests
• Creating and Editing Load Tests
• Creating Load Tests Using the New Load Test Wizard
• How to: Create a New Load Test Using the New Load Test Wizard
• How to: Create a Scenario Used to Model User Activities, Tests, Browsers and Networks Using the New Load Test Wizard
• How to: Create a Load Pattern in the Load New Test Wizard
• How to: Create a Text Mix Model in the New Load Test Wizard
• How to: Create a Test Mix in the New Load Test Wizard
• How to: Create a Network Mix in the New Load Test Wizard
• How to: Create a Browser Mix in the New Load Test Wizard
• How to: Create Counter Sets in the New Load Test Wizard
• How to: Create Run Settings in the New Load Test Wizard
• Editing Load Test Using the Load Test Editor
• Editing Load Test Scenarios Using the Load Test Editor
• Load Test Scenario Properties
• Creating Additional Scenarios for an Existing Load Test
• How to: Insert Additional Scenarios to an Existing Load Test
• Editing Think Times to Simulate Web Site Human Interaction Delays in Load Tests Scenarios
• How to: Change the Think Profile
• Editing Load Patterns to Model Virtual User Activities
• How to: Change the Load Pattern
• How to: Specify the Percentage of Virtual Users that Use Web Cache Data
• How to: Specify the Step Ramp Time Property for a Step Load Pattern
• Editing Text Mix Models to Specify the Probability of a Virtual User Running a Test
• Emulating Expected Real-World Usage of a Web Site or Application in a Load Test Using a Test Mix Models
• How to: Edit the Text Mix Model Using the Load Test Editor
• How to: Apply Distribution to Pacing Delay When Using a User Pace Test Mix Model
• Editing the Test Mix to Specify Which Web Performance, Unit and Coded UI Tests to Include in a Load Test Scenario
• How to: Add Web Performance, Unit and Coded UI Tests to a Load Test Scenario Using the Load Test Editor
• How to: Remove Web Performance Tests, Unit and Coded UI Tests from a Load Test Scenario Using the Load Test Editor
• Specifying Virtual Network Types in a Load Test Scenario
• How to: Add Networks Using the Load Test Editor
• How to: Remove Networks Using the Load Test Editor
• Specifying Web Browsers Types in a Load Test Scenario
• How to: Add Web Browsers Using the Load Test Editor
• How to: Remove Browsers Using the Load Test Editor
• Configuring Test Iterations in a Load Test Scenario
• How to: Specify the Maximum Test Iterations for a Scenario Using the Load Test Editor
• How to: Specify the Think Times Between Test Iterations for a Scenario Using the Load Test Editor
• Configuring Scenario Start Delays
• How to: Delay the Start Time of a Scenario Using the Load Test Editor
• How to: Enable or Disable Whether a Scenario Runs During the Warm-Up Period
• How to: Specify Test Agents to Use in Load Test Scenarios
• About the Mix Control for Load Tests
• Specifying the Counter Sets for Computers in a Load Test
• How to: Manage Counter Sets Using the Load Test Editor
• How to: Add Counters to Counter Sets Using the Load Test Editor
• How to: Add Custom Counter Sets Using the Load Test Editor
• About Threshold Rules in Load Tests
• How to: Add a Threshold Rule Using the Load Test Editor
• How to: Add Computer Tags to Counter Set Mappings Using the Load Test Editor
• Configuring Load Test Run Settings
• How to: Add Additional Run Settings to a Load Test
• How to: Select the Active Run Setting for a Load Test
• How to: Select a Load Test Run Setting to Use from the Command-Line
• Load Test Run Setting Properties
• How to: Modify the Run Settings Properties in a Load Test Using the Load Test Editor
• How to: Specify the Number of Test Iterations in a Load Test Run Setting
• How to: Specify the Sample Rate for a Load Test Run Setting
• How to: Specify the Timing Details Storage Property for a Load Test Run Setting
• How to: Add Context Parameters to a Load Test Run Setting
• Modifying Load Test Logging Settings
• How to: Specify How Frequently Test Logs are Saved Using the Load Test Editor
• How to: Specify if Test Failures are Saved to Test Logs Using the Load Test Editor
• Collecting SQL Trace Data to Monitor and Improve Performance in Load Tests
• How to: Integrate SQL Trace Data Using the Load Test Editor
• How to: Start a Performance Session on a Web Application from the Load Test Editor
• How to: Specify 64-Bit Process Using Test Settings
• Considerations for Load Tests
• Using Coded UI Tests in Load Tests
• Creating and Editing Web Performance Tests
• Web Performance Test Engine Overview
• Creating Web Performance Tests Using the Web Performance Test Recorder
• How to: Create a New Web Performance Test Using the Web Performance Test Recorder
• How to: Add a Comment to a Web Performance Test in the Web Performance Test Recorder
• Customizing Web Performance Test Recordings Using Web Performance Test Editor
• Web Performance Test Editor Overview
• Setting Web Performance Test Properties Using the Web Performance Test Editor
• How to: Edit an Existing Web Performance Test Using the Web Performance Test Editor
• How to: Add a Web Recording to an Existing Web Performance Test
• How to: Create a Coded Web Performance Test
• How to: Add Comments in the Web Performance Test Editor
• How to: Use Reporting Names to Easily Identify Requests in a Web Performance Test
• How to: Set Think Times to Simulate Human Pausing in a Web Performance Test
• How to: Set Page Response Time Goals in a Web Performance Test
• How to: Set Request Details in a Web Performance Test
• Binding a Data Source to a Web Performance Test
• How to: Add a Database Data Source to a Web Performance Test
• How to: Add an XML File Data Source to a Web Performance Test
• How to: Add a CSV File Data Source to a Web Performance Test
• How to: Add a Schema Definition to a Text File Data Source for a Web performance Test
• How to: Add Data Binding to a Web Request
• How to: Add Data Binding to a Web Service Request
• How to: Configure Data Binding Access Iterations in a Web Performance Test
• How to: Bind a Web Performance Test to a Text File
• Using Validation and Extraction Rules in Web Performance Tests
• How to: Add a Validation Rule to a Web Performance Test
• How to: Add an Extraction Rule to a Web Performance Test
• How to: Create a Custom Validation Rule for a Web performance Test
• How to: Create a Custom Extraction Rule for a Web performance Test
• How to: Create a Test-level Validation Rule for a Web performance Test
• Adding Loop and Branch Logic to Web Requests in Web Performance Tests
• How to: Add Loops to Web Performance Tests Requests
• How to: Add "If Then" Branch Conditions to Web Performance Tests Requests
• How to: Add a Transaction to a Web Performance Test
• How to: Insert a Call in a Web Performance Test to Another Web Performance Test
• How to: Set Credentials on a Web Performance Test
• How to: Parameterize the URL for a Web Performance Tests Web Server
• How to: Promote Dynamic Parameters to Web Performance Test Parameters
• How to: Use Find and Replace in a Web Performance Test
• How to: Use Context Parameters in a Web Performance Test
• How to: Start a Performance Session on a Web Application from the Web Performance Test Editor
• How to: Extract Requests in a Web Performance Test to Create a New Web Performance Test
• How to: Specify a Proxy Server for a Web Performance Test
• How to: Configure Visual Studio Ultimate Options for Web Performance Tests
• Running Load and Web Performance Tests
• Running Load Tests
• How to: Run a Load Test
• How to: Run a Load Test Using Test Controllers and Test Agents Specified in a Test Setting
• How to: Run a Load Test from the Command Line
• How to: Add a Comment to a Running Load Test Using the Load Test Analyzer
• Distributing Load Tests Across Multiple Test Machines Using Test Controllers and Test Agents
• Configuring Test Controllers and Test Agents for Load Testing
• How to: Add Virtual User License Keys for Load Testing
• How to: Add Virtual User License Keys for Load Testing Using the Command-Line
• How to: Create a Test Setting for a Distributed Load Test
• Walkthrough: Using a Test Controller and Test Agents in a Load Test
• Strategies for Troubleshooting Test Controllers and Test Agents in Load Tests
• Running Web Performance Tests
• How to: Run a Web Performance Test
• How to: Change the Web Performance Test Run Settings in the Web Performance Result Viewer
• How to: Run a Web Performance Test from the Command Line
• Analyzing Load Tests Results Using the Load Test Analyzer
• Load Test Analyzer Overview
• How to: Access Load Test Results for Analysis
• Load Test Results Summary Overview
• How to: Add Comments While Analyzing a Completed Load Test Using the Load Test Analyzer
• How to: View Data and Diagnostic Attachments Using the Load Test Analyzer
• Analyzing Threshold Rule Violations in Load Tests Using the Load Test Analyzer
• How to: View Web Page Response Time in a Load Test Using the Load Test Analyzer
• Analyzing Load Test Results in the Graphs View of the Load Test Analyzer
• Using the Graphs View Legend to Analyze Load Tests
• How to: Add and Delete Counters on Graphs in Load Test Results
• How to: Zoom in on a Region of the Graph in Load Test Results
• How to: Tile Graphs in Load Test Results
• How to: Specify Plot Options for Graphing Counters
• How to: Create Custom Graphs in Load Test Results
• How to: Export Graph View Data to Microsoft Excel
• Analyzing Load Test Results in the Tables View of the Load Test Analyzer
• How to: Tile Load Test Tables
• Analyzing Errors in Load Tests Using the Errors Table
• How to: View SQL Trace Data in Load Tests Using the SQL Trace Table
• Using the Counters Panel in Graphs View and Tables View
• How to: Analyze Threshold Violations Using the Counters Panel in Load Test Analyzer
• How to: Analyze Errors Using the Counters Panel
• Analyzing Load Test Virtual User Activity in the Details View of the Load Test Analyzer
• How to: Configure Load Tests to Collect Full Details to Enable Virtual User Activity in Test Results
• How to: Analyze What Virtual Users Are Doing During a Load Test Using the Virtual User Activity Chart
• Walkthrough: Using the Virtual User Activity Chart to Isolate Issues
• Examining Results Using the Web Performance Test Results Viewer
• Web Performance Test Results Viewer Overview
• Resolving Web Performance Test Result Viewer Playback Issues in Web Performance Tests
• How to: Use Quick Find to Isolate Dynamic Parameters in Web Performance Test Playback
• How to: View Recording Log in Web Performance Test Playback
• How to: Jump to Web Request from the Web Performance Test Playback
• How to: Add an Extraction Rule from the Web Performance Test Playback
• How to: Resolve Web Performance Test Playback Issues Caused by Dynamic Parameters
• Walkthrough: Correlating Dynamic Parameters
• Managing Load Test Results in the Load Test Results Repository
• How to: Create a Load Test Results Repository Using SQL
• How to: Select a Load Test Results Repository
• How to: Delete Load Test Results from a Repository
• How to: Import Load Test Results into a Repository
• How to: Export load Test Results from a Repository
• Reporting Load Tests Results for Test Comparisons or Trend Analysis
• How to: Create Load Test Performance Reports Using Microsoft Excel
• Creating and Using Custom Plug-ins for Load and Web Performance Tests
• How to: Use the Load Test API
• How to: Create a Load Test Plug-In
• How to: Use the Web Performance Test API
• How to: Create a Web Performance Test Plug-In
• How to: Create a Request-Level Plug-In
• How to: Create a Web Service Test
• How to: Create a Recorder Plug-In
• How to: Create a Custom HTTP Body Editor for the Web Performance Test Editor
• How to: Create a Visual Studio Add-In for the Web Performance Test Results Viewer
• Troubleshooting Load and Web Performance Tests
• Troubleshooting Load Tests
• Troubleshooting Web Performance Tests
• About JavaScript and ActiveX Controls in Web Performance Tests

Ed.