Enterprise Testing Guidance and Methodology

Article
06/02/2010

Applies To: Windows 7, Windows Vista

This topic provides recommendations for your general testing methodology and information about how to create your test environment, including the test lab requirements, suggestions about how best to model your production environment, and recommendations about how to configure your environment for automated testing.

General Testing Methodology

When testing an application in a new operating system, we recommend that you retain the default security-feature selections. We also recommend that you thoroughly test the applications, replicating as many of the usage scenarios from within your organization as possible. Finally, we recommend that you enter your issues and solutions into the Application Compatibility Manager, so that you can track the data from a central location.

When testing a Web site or a Web application, we recommend that you include both intranet and extranet sites, prioritizing the list based on how critical the site or the application is to your organization. We also recommend that you thoroughly test the Web sites and Web applications, replicating as many of the usage scenarios from within your organization as possible. Finally, we recommend that you enter your issues into the Application Compatibility Manager, so that you can share that data with Microsoft and receive potential solutions for your issues.

Creating the Test Environment

Your test environment should be a long-term investment in the overall deployment process. Retain the test environment after the deployment to assist in future deployment projects. To create the test environment, you must:

Determine how to model the production environment in the test environment.
Configure the test environment to support automated testing of the mitigation strategies.

Test Lab Requirements

We recommend that you establish a dedicated and isolated lab environment for use in developing and testing the application-compatibility mitigation. The lab should mirror your production environment as closely as possible. In some cases, you might find that it is better to open the test network to existing production services, instead of replicating your production environment in detail. For example, you might want to permit your Dynamic Host Configuration Protocol (DHCP) packets to pass through routers into the test network.

Note

Some operations can be safely conducted in the production environment, such as the application inventory-collection process.

At a minimum, your lab environment should include:

DHCP services
Domain Name System (DNS) services
Microsoft® SQL Server® 2008, Microsoft SQL Server 2005, or either version of the Microsoft SQL Server Express Edition
Lab test user accounts, with both normal user and administrative privileges
Network hardware to provide connectivity
Internet access (for downloading updates, files, and so on)
Test computers that accurately reflect production computers in both software and hardware configuration
A software library representing all the applications to be tested
Microsoft System Center Configuration Manager 2007 or Systems Management Server (SMS) 2003 with Service Pack 1 and the SMS Operating System Deployment (OSD) Feature Pack (optional)
Microsoft Virtual Server 2005 and Virtual PC 2004 (optional)
Windows® Internet Name Service (WINS) services (optional)

Determining How to Model the Production Environment

The goal of the test environment is to model your production environment. The more accurate the production environment, the greater the validity of the testing performed in that test environment.

We recommend the following best practices in creating your test environment.

Use virtual or physical images of production computers to create their test environment counterparts. Virtual or physical images can help ensure that the test environment configuration accurately reflects the production environment. In addition, the images contain live information (such as users, user profiles, and file permissions) to use in testing.
Physically separate your test environment from your production environment. A physically separate test environment enables you to use an identical IP configuration and helps ensure that tests conducted in the test environment do not affect the production environment. Using the identical IP address, subnets, and other network configuration information helps to ensure the fidelity of the test environment. However, duplicating IP addresses might not always be the best option when applications do not rely on a hard-coded IP address. You might also pass some network traffic through the router from the production environment to reduce the need for replicating network services. For example, opening the ports for DHCP to pass through eliminates the need for a separate DHCP server in the test lab.
Ensure that your test environment is at the same service-pack and update level as your production environment. Before performing your tests, update your lab environment by applying service packs and updates, or by refreshing the virtual or physical images of your production counterparts. Consider adding the test environment to the change-management process to simplify tracking the updates.
Ensure that you perform all of your tests by using accounts that have similar permissions as the accounts in your production environment. For example, if your organization does not allow users to run as Administrators on their local computers, ensure that similar permissions are not granted to users in the test environment. This process ensures that you can determine potential security issues.

Configuring the Test Environment for Automated Testing

In most instances, you must test the mitigation strategies more than once and be able to revert reliably to a previous test state. Automating your testing process enables you to ensure reproducibility and consistency in your testing process.

Using test automation tools enables you to run your test cases in a standardized, reproducible manner.
Using disk-imaging software for physical images of the servers and using software virtualization features for reversing changes to virtualized hard disks enable you to restore your test environment back to a previous state.

Determining When Virtualization Is Appropriate

You can create and run virtualized servers by using Virtual PC 2004 for Windows Server® 2003, or Virtual Server 2005 R2 for Windows Vista® or Windows 7. There are both advantages and disadvantages to using virtualization, so you must determine whether this is right for your organization.

Advantages	Disadvantages
You can support a large number of servers in a limited amount of physical space. Virtualization enables you to run as many virtual servers as the physical computer’s resources (such as memory, multiple processors, and free disk space) allow. You can more easily share your test environment between teams. For example, your test team can create a virtualized test environment and then provide a copy to your development team for use in its development processes. Virtualization enables multiple users to perform simultaneous testing, mimicking the ability for each user to have a dedicated test environment. You can easily restore your environment to a previous state. For example, you can revert to a previous state by using the Undo Disks option.	You might find that virtualized servers are slower than their physical counterparts. The performance of virtualized servers is significantly reduced, because many of the physical resources (such as disks) are virtualized. You might find that some hardware-specific device drivers and applications are not supported in virtualized servers.

You can support a large number of servers in a limited amount of physical space. Virtualization enables you to run as many virtual servers as the physical computer’s resources (such as memory, multiple processors, and free disk space) allow.
You can more easily share your test environment between teams. For example, your test team can create a virtualized test environment and then provide a copy to your development team for use in its development processes.
Virtualization enables multiple users to perform simultaneous testing, mimicking the ability for each user to have a dedicated test environment.
You can easily restore your environment to a previous state. For example, you can revert to a previous state by using the Undo Disks option.

You might find that virtualized servers are slower than their physical counterparts. The performance of virtualized servers is significantly reduced, because many of the physical resources (such as disks) are virtualized.
You might find that some hardware-specific device drivers and applications are not supported in virtualized servers.

Share via