Stress Testing Data Access Components in Windows DNA Applications

Author: Mike Schelstrate

Summary: This article discusses the importance of stress testing your data access components in Microsoft Windows DNA applications, as well as how to make the process easier. (14 printed pages)

Contents:

  • Introduction

  • Preparing to Run a Stress Test

  • Running the Stress Test

  • Evaluating Your Stress Test Results

  • Tips for Running a Successful Stress Test

  • Summary

Introduction

An often overlooked step in the development and deployment of a Microsoft® Windows® DNA application is stress testing the application to ensure that it will perform as expected when accessed by the maximum number of authorized users in the production environment. This article emphasizes the importance of stress testing as it relates to developing applications utilizing Microsoft Data Access Components (MDAC) and offers a few tips that will make the process easier to complete.

The purpose of this article is to assist the experienced developer and IT professional in designing and implementing a complete stress test scenario, in diagnosing the test results, and to recommend changes to correct deficiencies. It is assumed the reader is familiar with technologies such as Microsoft Windows NT® Server, Microsoft SQL Server™, Microsoft Internet Information Server (IIS), Active Server Pages (ASP), Microsoft ActiveX® Data Objects (ADO), and the Microsoft Component Services environment (or Microsoft Transaction Server [MTS] if you are using Windows NT).

Proper implementation of business logic and data access procedures in COM or DCOM components, including ADO components, should be emphasized. For performance and reliability reasons, these procedures should not reside in Active Server Pages. If you are concerned about how your application will operate under stress, it's likely that you have already designed your application to make use of these components and to take advantage of the benefits of the Component Services environment.

This article does not attempt to discuss stress issues caused by the client browser or by bandwidth limitations but instead focuses primarily on server-side data access components and their interaction with Internet Information Server. Neither does this document address the challenges presented by the use of Remote Data Services (RDS).

Why Stress Test My Application?

Stress testing should always be performed before deploying the application in a production environment. The basic purpose of stress testing a Web-enabled application is to accomplish the following:

  • To accurately measure the individual user experience as overall user load is increased on the system.

  • To determine the maximum capacity of the hardware utilized by the application and thus determine whether a hardware upgrade will be necessary before the deployment of the application in a production environment.

  • To define the acceptable performance threshold in terms of average page response time for the user of the application.

  • To ensure that the performance threshold remains at an acceptable level when the estimated maximum concurrent user load is placed on the system.

For most Web applications, the user experience is what matters most if the application is to be successful. However, there are many good reasons to put your application through the stress testing process, including the following:

  • Applications that work well during development can perform poorly when operating in a high stress environment. For example, when Internet Information Server or SQL Server machines are being utilized by multiple applications simultaneously, adding your new application to the mix could interfere with, and maybe even break, existing applications if you haven't designed the application to operate correctly in this scenario.

  • Your customers will form their most important impression of your application the first few times they use it. If this impression is unfavorable because of stress problems, it will be very difficult to change their minds, even after you have resolved the problems. On the other hand, by adequately stress testing your applications before deployment, you establish a reputation with your customers as a developer who creates good, fast applications that work the way they are intended to work.

  • The IT group responsible for deploying and maintaining your application will appreciate your stress testing as much or more than your customers. They are on the front line and will be the first to hear opinions and observations. If you can reliably anticipate the scalability issues that will affect the IT group, the members of this group will be much more willing to assist you when you are in need of their skills.

  • Potential business partners should also be included in the customer satisfaction matrix. They can significantly enhance the success of your application if they decide to include it as part of a larger application package.

Preparing to Run a Stress Test

To determine best-case server analysis values, also known as a baseline, a pilot system configuration should first be tested in a controlled environment. Next you should test in a simulated production environment to determine how the production environment configuration affects the results of the pilot system.

In preparation for the stress testing phase of your development cycle, you will need to focus on the following areas:

  • Hardware and software configuration

  • Server configuration

  • Security configuration

  • User load configuration

  • Selecting the right stress tools

Hardware and Software Configuration

The pilot system should mirror the production system as much as possible. Hardware configurations for CPU, RAM, and network bandwidth are the most important areas that will be exercised during stress testing. Every effort should be made to duplicate the software configurations, such as the appropriate Microsoft Windows version and service pack, the MDAC version, the Internet Information Server configuration, and any other applications that will be running on the same machine in the actual production system. Install and register your middle-tier business rules and data access COM components, and configure them as the application design dictates.

Be sure to configure your test Web site to run either in-process or out-of-process, whichever the final configuration will require. Selecting this option determines whether your Web application will run in the same address space as IIS or in its own isolated address space. This configuration will have a major impact on the stress tests that will be performed. The following illustrations show the Stress Properties property page configurations for an out-of-process application in Microsoft Windows NT 4.0 and in Microsoft Windows 2000. In Windows NT 4.0, select the Run in separate memory space (isolated process) check box to run the Web application out-of-process. In Windows 2000, select an Application Protection setting of Medium or High to run the Web application out-of-process.

Windows NT 4.0 Stress Properties page

Figure 1. Windows NT 4.0 Stress Properties page

Windows 2000 Stress Properties page

Figure 2. Windows 2000 Stress Properties page

Server Configuration

Configure Internet Information Server to mirror the production server. The Internet Services Manager properties page provides various tuning options available for IIS. Of particular importance are determining whether to enable logging (which can slow down the system considerably) and, under the Performance tab, selecting the number of expected hits per day.

The data server is where the majority of stress-related issues are likely to be resolved. For efficient query execution, proper normalization of the database design is imperative. Therefore, it is imperative that you stress test with the actual database design your application is intended to work with, and you should ensure that the tables are populated with the maximum amount of data your application will generate. In addition, ensure that your test data server configuration options (most importantly, the locking and isolation levels and the optimization techniques employed—for example, table indexing) match the configuration options of the production data server.

Security Configuration

Your application's security scheme can have a severe performance impact on your application under stress, especially if the system includes an encryption technology such as the Microsoft Cryptography API. Therefore, you should configure your test system to use the same security scheme but not necessarily the same credentials. The Microsoft Windows NT LAN Manager (NTLM) pass-through authentication system is probably the most commonly used security protocol for intranet applications that access back-end data stores. If possible, you should consider the use of roles from within Component Services (or MTS, in Windows NT) to simplify and streamline the security authentication process and to enhance performance and stability.

User Load Configuration

First determine the maximum number of users you expect to access your application, and then double this number; a successful application will most likely be utilized by many more users than expected. In addition, calculate the time of day the majority of the users will need access and then determine the load that will be on the network during that time, which should be the time when you test your application. This strategy enables you to test the user load impact as well as the system-wide hardware configuration, to ensure that your application responds as expected during peak network periods.

Selecting the Right Stress Tools

In live data center conditions, Web servers experience high connection levels because so many customers connect to the Web application via either the corporate intranet or the Internet. A Web stress tool should have the ability to simulate a high number of concurrent connections with sufficient threads to maximize the concurrent connections while throttling the packet sizes sent to the Web server. Fortunately, there are numerous tools available that are designed to simulate these exact conditions. One of these tools, Microsoft's Web Application Stress Tool, is currently available free of charge from the Microsoft at https://homer.rte.microsoft.com. It provides all necessary capabilities as well as some nice additional functionality, such as extensive reporting features.

Web Application Stress Tool

The Web Application Stress Tool activates the testing environment by realistically simulating multiple browsers requesting pages from a Web application and allows you to record a script by accessing, from your browser, the pages you want to include in the test. This script can then be saved and run from any Windows NT or Windows 2000 client that has the application installed. It is not necessary to have as many client machines available as your production application will have because the Web Application Stress Tool is able to simulate multiple clients from each individual workstation.

Note

When you run your stress test, be careful not to increase the stress level on the clients such that your test machines spend more time context switching between threads than doing actual work. To ensure that threads are distributed across clients, use several clients when running a test of a Web-based application, thereby alleviating the resource limits of one client machine.

Performance Monitor

To effectively stress test your data access components and correctly diagnose the results, a method of monitoring and recording operational statistics is of paramount importance. Performance Monitor, a tool that ships with Microsoft Windows NT and Microsoft Windows 2000, is the best tool available for monitoring and recording these statistics, both in Internet Information Server and on your data server.

Other Tools

In addition to utilizing Performance Monitor on Internet Information Server, it is necessary to monitor certain custom counters on the data server. Most high-performance data server applications, such as Microsoft SQL Server, Oracle, and Microsoft Exchange Server, include custom performance monitor counters that can be used to gauge the health of the application and the hardware it is running on.

Running the Stress Test

After you've carefully planned out your testing strategy, actually running the tests is easy. The first task in performance testing is to use a tool, such as the Microsoft Web Application Stress Tool, to apply stress to the Web site and measure the maximum requests per second that the Web server can handle. This is a quantitative measurement. The second task is to determine which resource prevents the requests per second from going higher, such as CPU, memory, or back-end dependencies; this process is more an art than it is a precise measurement technique.

First, select the ASP pages that you plan to run. (Look for the Web site's slowest pages and use those.) The specific selection should be determined by which pages access the database most frequently and with the largest or most complex queries. This selection is very important: It is better to include more pages than necessary than to leave out some critical code paths. Also, if appropriate, consider having your test application access a series of pages in a specified order, passing cookies or query strings along to each page just as the application will do in a real-world situation.

Note

Depending on the actual application, it may be necessary to prepare the ASP pages for testing. Some of your pages may require that you hard-code parameter values that are usually generated by the application and that the Web-stressing tool is unable to generate dynamically.

When running your application inside Internet Information Server, you should monitor (using Performance Monitor) the following counters:

  • Active Server Pages: requests per second, requests rejected, total queue length, and number of current sessions.

  • Inetinfo process: private bytes, virtual bytes, and number of open handles.

  • Processor: percent user time versus percent privileged time.

Note

If your application is running in its own memory space in Windows NT 4.0 (or with a setting of Application Protection: High [Isolated] on the Stress Properties page in Windows 2000), you should monitor the mtx.exe (or the dllhost process in Windows 2000) rather than monitor the Inetinfo process.

As shown in the following illustration, the Active Server Pages requests-per-second counter in Performance Monitor will display the actual throughput of your application (1.000 Requests/Sec in this example). This statistic will enable you to diagnose the performance of Internet Information Server under stress and will pinpoint the potential bottlenecks. This in turn will allow you to judge your application's ability to serve the maximum amount of users with an acceptable response time.

Performance Monitor

Figure 3. Performance Monitor

A Web server running ASP technology assigns every page request a thread from a pool established at startup; if all threads are used, subsequent page requests are placed in a queue. By monitoring the total queue length with Performance Monitor, you can determine how many clients are waiting for a response from the server.

The two most common non-hardware stress-related database problems are deadlocks and locking concurrency. On the data server, using the custom Performance Monitor counters that the data store will provide, you should monitor at a minimum the following:

  • Lock requests

  • Deadlocks per second

  • Table lock escalations per second

  • User connections

  • Active transactions

Your Web application should also be configured to take advantage of OLE DB resource pooling, which is managed automatically by the middle-tier OLE DB provider for Microsoft SQL Server 7.0. By creating the connection objects on a per-page basis and immediately releasing them, the database can handle thousands of concurrent users with a much smaller number of open database connections. This conserves database resources and provides much greater scalability. This performance enhancement should be monitored by tracking the number of user connections (using Performance Monitor) on the data server. As throughput increases, the number of user connections should remain stable as pooling controls the number of connections actually created on the data server.

The process of tuning an application against the database is critical to achieving performance goals and must be factored into the development cycle. This should include optimizing the size of the allocated memory, distribution of the application on the disk drives and controllers, and machine location of the ActiveX components. Give special consideration to eliminating the marshaling of data between processes whenever possible, because this is a very expensive operation.

Run the stress test for a period of time that is 50 percent longer than you expect the application to run in the actual user environment without interruption. Many problems, memory leaks in particular, do not manifest themselves until after the application has been running for an extended period of time.

What to Look for During the Stress Test

The average value for each Performance Monitor counter is determined by your specific application and hardware configuration. Therefore, when the stress tests are running, you should check each counter for any abnormal deviations from these average values.

Monitoring Internet Information Server

The most important areas to monitor on the Internet Information Server machine when looking for bottlenecks are the following:

  • CPU utilization

  • Memory usage

  • Throughput

Optionally, depending on your projected application environment, there are other performance areas that you might want to track during stress testing. Any of the following possible scenarios may indicate a problem with your application that should be repaired before the final release.

CPU utilization

A decrease in CPU utilization can point to a decrease in the performance of your application, possibly a thread contention issue.

As you monitor the ratio of CPU time between user and kernel time, remember that the user time should be 80 to 90 percent of total CPU time as a rule. Therefore, kernel mode time in excess of 20 percent is an indication of contention for kernel level API calls.

To get your money's worth out of your machines, you should register better than 50 percent processor utilization during peak load. A lower utilization value might indicate there are other bottlenecks in your system that need to be resolved.

Memory usage

A sudden jump or gradual increase in memory usage is another area that is frequently identified as a common problem for long-running server applications. Normally, this is where memory and resource leaks show up during the testing phase.

Throughput

Monitoring the Active Server Pages requests-per-second counter allows you to identify when or if your application begins to have performance problems. This counter will normally vary in a real-world application, but by carefully setting the number of threads and concurrent connections (for example, from the Web Application Stress Tool configuration screen), you will be able to simulate a steady number of requests. A sudden decrease in this counter spells trouble.

Optional testing areas

The following are examples of other areas you might find well worth monitoring during stress testing:

  • Total queue length. Typically, the Total Queue Length counter in Performance Monitor goes up and down. Therefore, if total queue length never increases and you are running at low processor utilization, this could indicate you have a smoothly running site with more capacity than you need for that stress load. Or, if your queue length is going up and down but your processors are running below 50 percent, this could indicate some of your requests are blocking and may require further optimization.

  • Browser response time. You can periodically access your Active Server Pages from a browser to monitor the response time and to ensure that the stress test is running correctly and the Web site is still able to serve ASP pages correctly. It is recommended that you perform this test at least twice daily during stress tests.

  • Timeout errors. During browser testing, be on the lookout for timeout errors, which are returned by Internet Information Server; these errors might indicate too many users are accessing the application simultaneously.

Monitoring the data server

The internal processing of the various MDAC services with the data server and the formatting of data for display normally consume the majority of the available server resources dedicated to the Web application. Therefore, when stress testing the application, it is imperative that special consideration be given to the performance of these components as it relates to the data access and data manipulation areas of the application.

Database user connections, lock contention, and deadlocks are the primary candidates for monitoring on the data server. Periodically view the process information in the management console for your database (for example, on a machine running SQL Server, in the Current Activity area of SQL Enterprise Manager). Check for blocking server process IDs, a common cause of data queries that do not return a response. This is a contention problem and usually requires significant changes in either the database design or the application logic.

Deadlocks can be identified in various ways. The most common method for identifying deadlocks is by using the Number of Deadlocks/sec custom counter in Performance Monitor. Your application should already be checking for deadlock problems and responding appropriately, because allowing the data server to designate the deadlock victim (that is, the user or session that will be canceled to resolve the deadlock) could cause problems with your application. The application should detect the deadlock condition when encountered and react accordingly. A common response to a deadlock situation is to wait for a few milliseconds and then retry the operation; normally, deadlocks are simply time-sensitive errors that will disappear when the operation is retried.

Evaluating Your Stress Test Results

When the stress tests are completed and the test results are available for inspection, a comparison of the target performance benchmarks with the statistics collected during testing will dictate the changes needed to ensure the throughput that your users will demand. Following are the specific areas you should review and evaluate for needed performance corrections:

  • Hardware

  • Database design

  • ActiveX components

  • Client cursors

  • ASP execution

  • IIS load

Hardware

Probably the easiest and least expensive solution to increasing the throughput of your application is a hardware upgrade. Normally, it is much more cost-effective to upgrade your hardware rather than pay a team of developers to rewrite portions of the application. For example, simply by adding more RAM to the server you can double the throughput of your application. However, if your test results indicate CPU utilization is the bottleneck, an upgrade may be more expensive because it's likely the entire machine must be upgraded to increase the number and speed of its CPUs.

Other hardware-related enhancements include increasing the speed of your hard drives and controllers and adding faster or additional network interface cards.

Database Design

When evaluating issues related to your database design, look for hot spots. Analyze the deadlock statistics, and confirm that your application is optimized to avoid deadlocks as much as possible. Consider changing the data access algorithms if necessary to avoid contention. Experiment with different index solutions. Examine your data server's query execution plan to confirm your queries are utilizing the proper indexes, and so forth.

ActiveX Components

The ActiveX components that include references to the ActiveX Data Objects type library should be carefully analyzed for possible optimizations. Do not use the default properties allowed in ADO. To avoid using a default property accidentally, always specify properties—for example, properties such as Cursor Type and Cursor Location.

Client Cursors

If an unusually large amount of memory is being consumed by your Web application, improper usage of cursor locations could be the cause of the problem. When using client cursors (recordset.cursorlocation = adUseClient), be aware that the client is actually Internet Information Server, not the browser. (The exception to this rule is when using Remote Data Services, which is not discussed in this article.) A common developer error is the assumption that use of a client cursor moves the entire recordset to the browser rather than to a machine running IIS. Therefore, remembering that you are actually storing the recordset on a machine running IIS will make you more conscious of the resources used.

For example, if your application requires access to a table that lists valid state or county codes and this information is stored at a data server, it is much more efficient to use a client cursor to create a recordset that resides on the IIS machine and then access the codes locally, avoiding the need for additional round-trips to the data server whenever this information is accessed by the application. Just be sure you are aware of the trade-offs, and do not overload IIS with large memory requirements if the memory is not available.

ASP Execution

If your ASP pages that contain data access procedures are taking too long to execute, you might need to move the data access code from ASP pages to ActiveX components, preferably placed inside packages within Microsoft Transaction Server (in Windows NT) or Component Services (in Windows 2000), depending on which operating system you are running. This compiled code runs much more efficiently than the interpreted scripted code contained within Active Server Pages.

Internet Information Server Load

Monitor the number and types of applications that are using Internet Information Server. You might need to add additional servers, move the application to another server, or consider implementing Windows Load Balancing Service.

Tips for Running a Successful Stress Test

  • Do:

    • Place all of your business logic in ActiveX components, and use ASP pages as the glue that ties everything together. Encapsulate reusable code into libraries.

    • Use MTS (Windows NT users). This great tool provides additional thread and resource pooling for server-side COM objects and easier management of the objects. In addition, MTS provides transactional capabilities.

    • Use resource/connection pooling. This MDAC feature is enabled by default but should be monitored to ensure that it is working properly.

    • Tune your data store. Use stored procedures where applicable.

    • To minimize data access operations, use caching for input, output, and transformed data.

    • Use in-process applications and ActiveX components at every opportunity.

    • Make sure debugging is disabled in a production environment. Debugging an application forces thread affinity.

    • In an intranet environment, offload work to the client browser where feasible.

    • Upgrade to Windows 2000. The performance and scalability enhancements included in this new release are immediately obvious when stress testing an application.

    • If you're unable to upgrade to Windows 2000 at this time, it is a good idea to at least upgrade to Microsoft Visual Basic® Scripting Edition (VBScript) 5.0; the performance and functionality has improved considerably in this new version.

    • Consider using Microsoft Message Queue Server (MSMQ). Proper utilization of asynchronous messaging will increase perceived user response times enormously.

  • Don't:

    • Use application or session-level objects in Active Server Pages.

    • Place your data access code in Active Server Pages. Use the scripting code to communicate with the data access functions and business rules that are inside ActiveX components.

    • Use HTTPS/Secure Sockets Layer except where absolutely necessary. This authentication protocol is very expensive to implement.

    • Use application state or session state when not necessary.

Summary

The Internet opens your applications to many more potential users than traditional client-server applications. As more and more organizations look to the Web as a strategic part of their business strategy, they need to be sure the technology they choose can handle their demanding needs. Besides easy-to-use tools, organizations need an infrastructure to meet their user load requirements. Therefore, it is more important than ever that stress testing be a fundamental part of your testing regime, especially when incorporating MDAC into your application.

Note

A preliminary requirement for operating successfully under stressful conditions is taking a best-practices approach during the development cycle. This means scheduling time both for performance testing under load and for tuning the application to achieve performance goals must be factored into the development process.

The benefits of stress testing and iteratively tuning your application under load are straightforward:

  • You get the information you need to ensure superior throughput results for your application.

  • You can accurately assess the scalability characteristics of your application so that you can adjust the application to achieve specified performance goals.

  • You can discover early the design issues that degrade performance and throughput, and adjustments can be made before the application is deployed into production.

  • Your applications will gain a reputation for high-performance dependability among your customers and business partners.