Advanced Load Testing Features of Visual Studio Team System
Visual Studio Team Test Edition provides two capabilities that can allow users to create load tests of web sites quickly and easily: the Web Test recorder and the Load Test wizard. In fact, these tools make this process so easy that it is tempting to use the web tests and the load tests that result from using these tools without much modification. However, to use the web testing and load testing capability most effectively it is beneficial to understand how to use other web test and load test properties that are not set by these tools. This note describes some of the most important considerations for load testing with Visual Studio Team System that are not addressed by the web test recorder and the load test wizard.
Verify web tests and unit tests
Before running a load test, it is a good idea to make sure that all of the tests contained in the load test will pass when run by themselves by running them from either the Test Explorer or Test View windows, or from the Web Test editor in the case of a web test. For data bound web tests, run through all of the data values.
See Josh Christie’s excellent paper Web Test Authoring and Debugging Techniques at https://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnvs05/html/WTAuthDebug.asp for more details on creating and verify web tests. Some additional web test properties that are useful when running load test are described later in this document.
Choose an appropriate load profile
Choose a load profile for each Scenario in your load test that is appropriate for your test goals. Here is a summary of the three choices for load profile.
Using a Constant Profile
A Constant Load Profile can be used to run the same user load throughout a load test. Be careful about using a Constant Load Profile with a high user count; doing so may place an unreasonable and unrealistic demand on your server(s) at the beginning of the load test. For example, if your load test contains a web test that starts with a request to a home page, and you set up the load test with a constant load of 1,000 users, the load test will submit the first 1,000 requests to the home page as fast as possible. This may not be a realistic simulation of real-world access to your web site. To mitigate this, consider using a Step Load Profile that ramps up gradually to 1,000 users, or specify a warm-up period in the Load Test Run Settings. If a warm-up period is specified, the load test will automatically increase the load gradually during the warm-up period.
Using a Step Load Profile
A Step Load Profile can be used to increase the load on the server(s) as the load test runs so that you can see how performance varies as the user load increases. For example, to see how your server(s) perform as the user load increasing to 2,000 users, you might run a 10-hour load test using a Step Load Profile with the following properties:
Initial User Count: 100
Maximum User Count: 2000
Step Duration (seconds): 1800
Step Ramp Time (seconds): 20
Step User Count: 100
These settings have the load test running for 30 minutes (1800 seconds) at user loads of 100, 200, 300, up to 2,000 users. The Step Ramp Time property is worth special mention here because it is the only one of these properties that is not available to choose in the Load Test Wizard. This property allows the increase from one step to the next (for example from 100 to 200 users) to be gradual rather than immediate. In this example, the user load would be increased from 100 to 200 users over a 20 second period (an increase of 5 users every second).
Using a Goal-Based Load Profile
A Goal Based Load Profile is useful when you want to determine the number of users that your system can support before reaching some level of resource utilization. This option works best when you have already identified the limiting resource (i.e. the bottleneck) in your system. For example, if you know that the limiting resource in your system is the CPU on your database server, and you want to see how many users can be supported when the CPU on the database server is approximately 75% busy, you could use a Goal Based Load Profile with the goal of keeping the value of the performance counter “%Processor Time” between 70% and 80%. One thing to watch out for is that if some other resource is limiting the throughput of the system, the goal specified by the Goal Based Load Profile may never be reached, and the user load will continue to rise until the value specified for the Maximum User Count is reached. This is usually not the desired load, so be careful about the choice of the performance counter in the Goal Based Load Profile, and also make a conscious decision about the value for the Maximum User Count to place an upper bound on the user load.
Choosing the location of the Load Test Results Store
When the Visual Studio Team Test Controller is installed, the Load Test Results Store is set up to use an instance of SQL Express that is installed on the controller computer. SQL Express is limited to using a maximum of 4 GB of disk space. If you are going to run many load tests and want to keep the results for a while, you should consider configuring the Load Test Results Store to use an instance of the full SQL Server product if available. See the Visual Studio Team Test documentation for instructions on setting up the database to be used as the Load Test Results Store.
Increase the performance counter sampling interval for longer tests
Choose an appropriate value for the “Sample Rate” property in the Load Test Run Settings based on the length of your load test. A smaller sample rate, such as the default value of five seconds, requires more space in the load test results database. For longer load tests, increasing the sample rate reduces the amount of data collected.
Here are some guidelines for sample rates:
Load Test Duration Recommended Sample Rate
< 1 Hour 5 seconds
1 - 8 Hours 15 seconds
8 - 24 Hours 30 seconds
> 24 Hours 60 seconds
Consider including Timing Details to collect percentile data
There is a property on the Run Settings in the Load Test Editor named "Timing Details Storage". If Timing Details Storage is enabled, then the time to execute each individual test, transaction, and page during the load test will be stored in the load test results repository. This allows 90th and 95th percentile data to be shown in the load test analyzer in the Tests, Transactions, and Pages tables. The amount of space required in the load test results repository to store the Timing Details data may be very large, especially for longer running load tests. Also, the time to store this data in the load test results repository at the end of the load test is longer because this data is stored on the load test agents until the load test has finished executing at which time the data is stored into the repository. For these reasons, Timing Details is disabled by default. However if sufficient disk space is available in the load test results repository, you may wish to enable Timing Details to get the percentile data. Note that there are two choices for enabling Timing Details in the Run Settings properties named "StatisticsOnly" and "AllIndividualDetails". With either option, all of the individual tests, pages, and transactions are timed, and percentile data is calculated from the individual timing data. The difference is that with the StatisticsOnly option, once the percentile data has been calculated, the individual timing data is deleted from the repository. This reduces the amount of space required in the repository when using Timing Details. However, advanced users may want to process the timing detail data in other way using SQL tools, in which case the AllIndividualDetails option should be used so that the timing detail data is available for that processing.
Consider enabling SQL Tracing
There is a set of properties on the Run Settings in the Load Test Editor that allow the SQL tracing feature of Microsoft SQL Server to be enabled for the duration of the load test. If enabled, this allows SQL trace data to be displayed in the load test analyzer on the "SQL Trace" table available in the Tables dropdown. This is a fairly easy-to-use alternative to starting a separate SQL Profiler session while the load test is running to diagnose SQL performance problems. To enable this feature, the user running the load test (or the controller user in the case of a load test run on a rig) must have the SQL privileges needed to perform SQL tracing, and a directory (usually a share) where the trace file will be written must be specified. At the completion of the load test, the trace file data is imported into the load test repository and associated with the load test that was run so that it can be viewed at any later time using the load test analyzer.
Don’t Overload the Agent(s)
If an agent machine has more than 75% CPU utilization or has less than 10% of physical memory available, add more agents to your test rig to ensure that the agent machine does not become the bottleneck in your load test.
Add an Analysis Comment
After the load test is complete and you have spent some time analyzing the results, you can a short one line description and an arbitrarily long analysis comment to be stored permanently with the load test result. To do this, in the load test result viewer, right click and choose the “Analysis” option. This brings up a dialog that allows you to enter your analysis text which is stored in the load test results database you click OK to close the dialog.
Consideration for Load Tests that contain Web Tests
The following considerations apply to load tests that contain web tests, but are not applicable to load tests that contain only unit tests.
Choose the Appropriate Connection Pool Model
The property WebTest Connection Model found on the RunSettings property sheet in the Load Test editor allows you to choose between two models for managing the connections that the load test runtime engine uses for connecting to the target web site(s):
The ConnectionPerUser model most closely simulates the behavior of a real browser: each virtual user running a web test creates one or two connections to the web server that are dedicated to that virtual user. The first connection is established when the first request in the web test is issued. A second connection may be used when a page contains more than one dependent request; these requests may be issued in parallel using the two connections. These connections are re-used for subsequent requests within the web test, and are closed when the web test completes. The only drawback to the ConnectionPerUser model is that the number of connections held open on the agent machine may be high (as high as 2 times the user load), and the resources required to support this high connection count may limit the user load that can be driven from a single load test agent. The extra resources used are the memory associated with the connection and extra processing time to close and reopen the connections as web tests complete and new web tests are started.
The ConnectionPool model conserves the resources on the load test agent by sharing connections to the web server among multiple virtual web test users. When the Connection Model is ConnectionPool, the Connection Pool Size specifies the maximum number of connections to make between the load test agent and the web server. If the user load is larger than the connection pool size, then web tests executing on behalf of different virtual users will share a connection. This means that one web test will need to wait before issuing a request when another web test is using the connection. The average time that a web test waits before submitting a request is tracked by the load test performance counter “Avg. Connection Wait Time”. The value of this performance counter should be very low; if it is not, you need to increase the connection pool size to prevent it from limiting the throughput of your load test.
Consider setting response time goals for web test requests
One of the properties of a web test request is a response time goal. If you define response time goals for your web test requests, when the web test is run in a load test, the load test analyzer will report the percentage of the web tests executed within the load test for which the page response time was less than the goal. Note that the response time goal is for the entire page, meaning the time to receive the response for the request specified in the web test including the time to receive the response for any dependent requests. By default there are no response time goals defined.
Consider setting timeouts for web test requests
Another property of a web test request is a timeout value. By default there is no timeout value specified on web test requests, so if the web server never responds to the request, the web test request (and the web test) will never complete and no error will be reported. If a timeout value (even a high one such as 300 seconds) is set, then the web test request will eventually timeout, an error will be reported, and the web test will continue.
Choose a value for the “Percentage of New Users” property
There is a property on each scenario in a Load Test named "Percentage of New Users”. This property affects the way in which the load test runtime engine simulates the caching that would be performed by a web browser. The default value for the “Percentage of New Users” property is 100%. This means that each web test run in a load test is treated like a first time user to the web site who doesn't have any content from the web site in their browser cache from previous visits. Thus all requests in the web test (including all dependent requests such as images) are downloaded (except in the case where the same cacheable resource is requested more than once in a web test). If you are load testing a web site that has a significant number of return users who are likely to have images and other cacheable content cached locally, then using the default setting of 100% for “Percentage of New Users” will generate more download requests than would occur in real-world usage. In this case, you should estimate the percentage of visits to your web site that are from first time users of the web site, and set “Percentage of New Users” accordingly.
Consider setting the “ParseDependentRequests” property of your web test requests to false
By default, when a web test is executed, HTML responses received from the web server are parsed and any dependent requests such as images or style sheets are automatically submitted. This is usually desirable as it places the most realistic load on the server. However, if the goal of your load test is to place the maximum load on your application server as opposed to your web server, you may wish to disable this behavior by setting the ParseDependentRequests on all of your web test requests to false.
Comments
- Anonymous
May 10, 2006
The additional detail on these features is a big help! - Anonymous
September 12, 2006
Thank you so much for this article. Your advice saved my company thousands of dollars and saved many development hours. - Anonymous
December 07, 2006
PingBack from http://psiman.wordpress.com/2006/12/07/team-test/ - Anonymous
December 20, 2006
Thank you Bill! Got great information here. I am presently evaluating Visual Team Studio for Testers.Also, I have a couple of question regarding the completion of a load test. Could you help me on that important point?It seems that when a load test ends all connections stop at the same time. Is there any way to stop a load test properly, when all test iterations are complete? I guess it will be very useful to setup the duration of a load test by the number of test iterations or a combination of number of test iteration and time?Also, regarding the reporting, it does not seem that I get all counters data during the warm up and due the Hard end of a load test (my previous question) I can't get data of the servers after the test. In other term, I would like to report the status (CPU, RAM etc...) of my servers (WEB, Services and SQL) BEFORE, during and AFTER the load test. it will be certainly easier to identify LEAKS.Thank you a lot in advance for your precious expertiseJeanvo - Anonymous
January 26, 2007
http://msdn.microsoft.com/msdnmag/issues/03/01/LoadTesting/ http://blogs.msdn.com/billbar/pages/517081.asp - Anonymous
October 25, 2007
HI: I am new to VSTS. am trying to understand differences between Users Load in Load test and no.of concurrent users to servers.Recently, I have to do a load test on web application which has 2 blade servers as backend. My configuration:Load Test:=======Initial User Count = 50Maximum User Count = 3000Step Duration = 30 SecsStep Ramp = 0 secsStep User Count = 50SQL Tracing:==========Minimum Duration of Traced SQL Operations = 500SQL Tracing Connect String = <empty>SQL Tracing Directory = <empty>SQL Tracing Enabled = FalseTiming:======Run Duration = 1 HourSample Rate = 5 SecsWarmup Duration = 30 SecsWebTest Connections:WebTest Connection Model = ConnectionPoolWebTest Connection Pool Size = 250When I ran load test with above configuration, VSTS shows that around 29 minute, it is able to reach the 3000 users BUT Network engineer was saying that they have only 300 user licence. How can 3000 users able to connect to server????My question is :Is Maximum User Count equal to No.of concurrent users to server? If not. PLEASE PLEASE explain to me.Thanks in advanceSuneel - Anonymous
March 17, 2008
Sorry for the very delayed response. I'm afraid I don't really look for comments on my blog very often. For questions on Web and load testing, it is best to submit them to the VSTS Web and Load Testing forum at http://forums.microsoft.com/MSDN/ShowForum.aspx?ForumID=1408&SiteID=1.But to answer your question: First of all, if the Web tests that your load test contains include think times, and the load test does not turn off think time simulation, then even though there are 3000 load test users, only some portion of the users will be submitting requests to the server at a particular time, because most of the others are thinking (think times are often higher than response times so users spend more time thinking, or viewing the page, filling in a form, which are considered "think time", than they do waiting for a response.Secondly, because your Connection Model in Connection Pool with a size of 250, the load test will never have more than 250 connections (or send more than 250 concurrent requests) to your server. If you change the connection model to Connection per User, I think you will see that you do run out of licenses if you are really limited to 300 connections. - Anonymous
June 04, 2008
Hi Bill:I am runing the Visual Studio 2008 web load test. In the load test editor, I enabled the SQL tracing, set the SQL tracing connection string and the SQL tracing directory. I run the load test on my local machine. The SQL database is running on another machine within the same domain. When I run the load test, it popups a window showing: "Could not start SQL tracing: could not create a trace file". Do you know what might be wrong.FYI, I log in as the domain administrator, who can access the remote database, and the sql trace diretory. The SQL trace directory is mapped as a network drive on my local machine where the load test runs.Thank you for any help in advance. - Anonymous
June 04, 2008
I found the problem. After I set the proper directory access permission, the web load test can create a SQL trace file.Thanks anyway. - Anonymous
June 14, 2008
Hi Bill,Pls. let me know, how to do SQL Tracing for Load Testing using Microsoft Team System - Anonymous
July 16, 2009
Right on the money - my problem has been solved from reading this, my thanks. - Anonymous
December 13, 2009
Thanks your article gave me tips to help me, thank you - Anonymous
January 06, 2010
Same for me. Your tips were step by step. Thank you - Anonymous
January 22, 2010
Love this source! great information! - Anonymous
February 11, 2010
Will this work on a piano sheet music website? I will test this code out so hopefully it will work with sheet musics for piano.-Bob - Anonymous
February 21, 2010
Nice topic . Thank you for posting it ! - Anonymous
March 14, 2010
Very nice topic to be covered. - Anonymous
April 07, 2010
Thanks for this tutorial!!Where can i find more info about reporting the satus of a server?Thanks. - Anonymous
May 03, 2010
Maximum user count setting not working: I am trying to run load test using visual studio 2008 with constant laod pattern.constant user count: 5test iterations: 1I expected that 5 users will be loaded but when I run laod test, I see only 1 user. If I change test iterations, for every test iteration, I am seeing only 1 user getting loaded. am I missing any setting? - Anonymous
June 02, 2010
You've provided a great information with detailed discussion about advanced load testing features. Thanks for sharing this post. - Anonymous
June 05, 2010
Thanks for sharing this great information. This is what I need now, at least went into details in discussing the advanced load testing feature of Visual Studio Team System. - Anonymous
July 20, 2010
whats the maximum number of concurrent user that the ms vsts supports? Its urgent pls reply guys..Thnaks in advance. - Anonymous
August 21, 2010
Hey i was searching for this code. You did an excellent job, Thanks - Anonymous
March 23, 2011
Great information thanks for sharing this wonderful information. - Anonymous
May 15, 2011
I really impressed by your post. Thank you for this great information, you write very well which i like very much.<a href="www.certificationkey.net/.../642-374.php">642-374 Testking</a> - Anonymous
September 14, 2011
Valuable information and excellent design you got here! I would like to thank you for sharing your thoughts and time into the stuff you post!! Thumbs up - Anonymous
November 15, 2012
This link doesnt work. Please fix it."See Josh Christie’s excellent paper Web Test Authoring and Debugging Techniques at msdn.microsoft.com/.../default.asp for more details on creating and verify web tests. "