Running Load Tests
After you create, edit, and set the configuration options that are appropriate for the testing needs of you load test, Visual Studio Ultimate provides several options for running load tests.
Tip
Before you run a load test, it is a good practice to make sure that all the Web performance and unit tests contained in the load test will pass when they are run by themselves. You can verify the tests by running them from the Web Performance Test Editor. For Web performance tests with data-binding, run through all the data values.
Considerations for running load tests
Before you run a load test, you should verify that the load test is appropriately configured to meet the requirements or purpose of the test.
Choose an appropriate load pattern. |
Choose a load pattern for each scenario in your load test that is appropriate for your test goals:
|
Choose the location of the load test results store. |
Load test results store considerations |
Set the performance counter sampling interval appropriately. |
Performance Counter Sampling Interval Considerations |
Consider including timing details to collect percentile data |
Considerations for including Timing Details to Collect Percentile Data |
Consider enabling SQL tracing |
Consider Enabling SQL Tracing |
Determine if additional test agents are needed. |
Do Not Overload the Test Agents |
For more information, see Considerations for Load Tests.
Graphing modes
While a load test is running, the Load Test Analyzer is in the Graphs view by default. The graphs can be displayed in one of two different modes:
Collapsing mode Collapsing is the default graph mode in the Load Test Analyzer during a running load test. A collapsing graph is used for load test while it is running to reduce the amount of data that must be maintained in memory while still showing the trend for a performance counter over the full run duration.
Scrolling mode Scrolling graph mode is available when you are viewing the result of a load test while it is running. A scrolling graph is an optional view that shows the most recent data points. Use a scrolling graph to view only the most recent 100 data intervals in the test.
Note
The zooming graph mode is available only when you view a completed load test result from a database.
Changing the graphing mode
To switch between collapsing and scrolling modes while a load test is running, use the Graph Options drop-down on the Load Test Analyzer toolbar. Choose Graph Data for Entire Run for collapsing mode, or Graph Only Recent Data for scrolling mode.
Tasks
Tasks |
Associated topics |
|---|---|
Configure load test run settings: Run settings are a set of properties that influence the way a load test runs. Run settings are organized by categories in the Properties window. |
|
Run a load test: You can use different user interface options to run a load test or run the load test from the command line. |
|
Running a load test remotely: You can use test agents and test controllers to run your load test on one or more remote computers. |
|
Viewing the test results graphically while the test is running: The results of a load test are displayed as data in several different panes while you run the test and upon the test's completion. |
|
Add a comment to the load test while it is running: If you are analyzing the load test either when it is running, or when it completes, you can add comment with a description and an arbitrarily long analysis comment to be stored permanently with the load test result. |
|
Distribute load and Web performance tests across machines: You can use a group of computers to generate simulated load for testing, and to run tests remotely and concurrently on several computers. |
|
Troubleshoot network emulation in load tests: You can verify that the network emulation is working correctly in your load tests. |
|
Running load tests with Team Build: You can add your load tests to a test category which can be configured run after a build has completed. |
Load pattern considerations
Choose one of the following load patterns for each scenario in your load test that is appropriate for your test goals.
For more information, see Editing Load Patterns to Model Virtual User Activities.
Using a constant load pattern
A constant load pattern is used to run the same user load during the run of a load test. Be careful about using a constant load pattern with a high user count; doing this could place an unreasonable and unrealistic demand on your server or servers at the beginning of the load test. For example, if your load test contains a Web test that starts with a request to a home page, and you set up the load test with a constant load of 1,000 users, the load test will submit the first 1,000 requests to the home page as fast as possible. This might not be a realistic simulation of real-world access to your Web site. To alleviate this, consider using a step load pattern that ramps up gradually to 1,000 users, or specify a warm-up period in the Load Test Run Settings. If a warm-up period is specified, the load test will increase the load gradually during the warm-up period. For more information, see Configuring Scenario Start Delays.
Using a step load pattern
A step load pattern can be used to increase the load on the server or servers as the load test runs so that you can see how performance varies as the user load increases. For example, to see how your server or servers perform as the user load increases to 2,000 users, you might run a 10-hour load test using a step load pattern with the following properties:
Initial User Count: 100
Maximum User Count: 2000
Step Duration (seconds): 1800
Step Ramp Time (seconds): 20
Step User Count: 100
These settings have the load test running for 30 minutes (1800 seconds) at user loads of 100, 200, 300, up to 2,000 users. The Step Ramp Time property is worth special mention here because it is the only one of these properties that is not available in the New Load Test Wizard. This property allows the increase from one step to the next (for example from 100 to 200 users) to be gradual instead of immediate. In the example, the user load would be increased from 100 to 200 users over a 20 second period; this is an increase of 5 users every second. For more information, see How to: Specify the Step Ramp Time Property for a Step Load Pattern.
Using a goal-based load pattern
A goal-based load pattern is useful when you want to determine the number of users that your system can support before reaching some level of resource utilization. This option works best when you have already identified the limiting resource, that is, the bottleneck, in your system. For example, if you know that the limiting resource in your system is the CPU on your database server, and you want to see how many users can be supported when the CPU on the database server is approximately 75% busy, you could use a goal-based load pattern with the goal of keeping the value of the performance counter "%Processor Time" between 70% and 80%.
Warning
if some other resource is limiting the throughput of the system, the goal specified by the goal-based load pattern might never be reached, and the user load will continue to increase until the value specified for the Maximum User Count is reached.
This is usually not the desired load. Therefore, be careful about the choice of the performance counter in the goal-based load pattern, and also make a conscious decision about the value for the Maximum User Count to place an upper bound on the user load.
Load test results store considerations
When Visual Studio Ultimate is installed, the load test results store is set up to use an instance of SQL Express that is installed on the computer. SQL Express is limited to using a maximum of 4 GB of disk space. If you are going to run many load tests over a long period of time, you should consider configuring the load test results store to use an instance of the full SQL Server product, if it is available. For more information, see Managing Load Test Results in the Load Test Results Repository.
Performance counter sampling interval considerations
Choose a value for the Sample Rate property in the load test run settings based on the length of your load test. A smaller sample rate, such as the default value of five seconds, requires more space in the load test results database. For longer load tests, increasing the sample rate reduces the amount of data that you collect. For more information, see How to: Specify the Sample Rate for a Load Test Run Setting.
Here are some guidelines for sample rates:
Load Test Duration |
Recommended Sample Rate |
|---|---|
< 1 Hour |
5 seconds |
1 - 8 Hours |
15 seconds |
8 - 24 Hours |
30 seconds |
> 24 Hours |
60 seconds |
Considerations for including timing details to collect percentile data
There is a property in the run settings in the Load Test Editor named Timing Details Storage. If Timing Details Storage property is enabled, then the time to execute each individual test, transaction, and page during the load test will be stored in the load test results repository. This allows 90th and 95th percentile data to be shown in the Load Test Analyzer in the Tests, Transactions, and Pages tables.
There are two choices for enabling the Timing Details Storage property in the run settings properties named StatisticsOnly and AllIndividualDetails. With either option, all the individual tests, pages, and transactions are timed, and percentile data is calculated from the individual timing data. The difference is that with the StatisticsOnly option, as soon as the percentile data has been calculated, the individual timing data is deleted from the repository. This reduces the amount of space that is required in the repository when you use timing details. However, advanced users might want to process the timing detail data in other ways, by using SQL tools. If this is the case, the AllIndividualDetails option should be used so that the timing detail data is available for that processing. Additionally, if you set the property to AllIndividualDetails, then you can analyze the virtual user activity using the Virtual User Activity chart in the Load Test Analyzer after the load test completes running. For more information, see Analyzing Load Test Virtual User Activity in the Details View of the Load Test Analyzer.
Note
In versions of Visual Studio prior to Visual Studio 2010, including Microsoft Visual Studio 2005 and Visual Studio 2008, the All Individual Details setting for the Timing Details Storage property was available. In Visual Studio 2010, and Visual Studio 2012, there are two important differences: First, the All Individual Details setting was not the default setting. Second, the All Individual Details setting was configured after the data was collected, the only way to access this information was by using SQL queries.
The amount of space that is required in the load test results repository to store the timing details data could be very large, especially for longer running load tests. Also, the time to store this data in the load test results repository at the end of the load test is longer because this data is stored on the load test agents until the load test has finished executing. When the load test finishes, the data is stored into the repository. The Timing Details Storage property is enabled by default. If this is an issue for your testing environment, you might want to set the Timing Details Storage to None.
For more information, see How to: Specify the Timing Details Storage Property for a Load Test Run Setting.
Consider enabling SQL tracing
To diagnose SQL performance problems, there is a set of properties on the run settings in the Load Test Editor that allows the SQL tracing feature of Microsoft SQL Server to be enabled for the duration of the load test. If the SQL tracing feature is enabled, SQL trace data can be displayed in the Load Test Analyzer on the SQL Trace table, which is available in the Tables View.
SQL tracing is a fairly easy-to-use alternative to starting a separate SQL Profiler session while the load test is running. To enable this feature, the user who is running the load test must have the SQL privileges needed to perform SQL tracing, and a directory where the trace file will be written must be specified. The directory is usually a share. When the load test is finished, the trace file data is imported into the load test repository and associated with the load test that was run so that it can be viewed later, at any later time, by using the Load Test Analyzer.
For more information, see Collecting SQL Trace Data to Monitor and Improve Performance in Load Tests.
Do not overload the test agents
If a test agent machine has more than 75% CPU utilization or has less than 10% of physical memory available, add more agents to your load test to ensure that the agent machine does not become the bottleneck in your load test.
For more information, see How to: Specify Test Agents to Use in Load Test Scenarios and Distributing Load Test Runs Across Multiple Test Machines Using Test Controllers and Test Agents.
Related Tasks
Creating and Editing Load and Web Performance Tests
Provides the directions you need to create and edit load and Web performance tests.
Provides information about how to run Web performance tests in your load tests.
External resources
Guidance
Testing for Continuous Delivery with Visual Studio 2012 – Chapter 6: A Testing Toolbox