Protecting a Network from Unmanaged Clients

On This Page

Appendix A: Network Access Protection


Welcome to this document from the Midsize Business Security Guidance collection. Microsoft hopes that the following information will help you create a more secure and productive computing environment.

Executive Summary

In today’s security conscious environment an in-depth approach to protecting a midsize business network and the data that resides therein is a very complex matter. It is no longer sufficient to rely on perimeter defenses and antivirus suites alone to protect network assets and confidential information. Security organizations and professionals now understand that internal network risks, whether intentional or accidental, have the potential to be even more perilous than external threats.

To address the myriad number of vulnerabilities that pose risk to midsize businesses, significant investments of time and resources have been made in areas such as patch management, anti-malware solutions, and identity management initiatives. To ensure that these investments are used universally within the business and to maximize the effectiveness of the investment, the business must find ways to efficiently enforce security policies.

Rogue computers are systems that are not considered substantial risks because there is no way to determine whether they comply with established security policies. Rogue computers can be a problem for system administrators and security professionals. Noncompliant systems pose a number of risks, from being vulnerable to malware infection to being potential platforms for an attack. Traditionally, they have been difficult to manage and bring into compliance.

This document discusses some effective methods that can be used to help enforce compliance with security policies. This approach maximizes the benefits of risk management efforts and adds an extra layer of security to midsize business networks that will help reduce the risks associated with untrusted and unmanaged computers.


This document consists of four main sections that focus on details to help the reader better understand concepts and principles associated with helping to protect a midsize business network from unmanaged computers. These four main sections are summarized as follows:


This section provides an executive summary of the document along with an overview of the document’s framework and some information regarding the intended audience for the document.


This section provides some details and definitions that will be useful for understanding the solutions that are discussed in the document.


This section describes some of the common issues that a midsize business might contend with when determining how to help protect against unmanaged and untrusted computers. It serves as a general background about the issues this document addresses.


This section discusses solutions that help address the challenge presented by unmanaged computers by assessing the possible approaches and discussing issues around developing plans to address the problem. It also provides some information about deployment and management methods.

Who Should Read This Guide

This technical document is intended to help technology professionals and technical managers understand and make informed decisions about helping to protect a midsize business network against the threats posed by unmanaged devices. Although non-technical audiences may be able to use this document to gain a better understanding of this particular security issue, basic knowledge of the following is useful to fully understand and apply the subject matter discussed in this document.

Specific technologies discussed include:

  • IEEE 802.1X Authentication

  • IPsec Policies

  • Network Security

  • Microsoft® Systems Management Server 2003

  • Microsoft Windows Server™ 2003

  • Active Directory® directory service

  • Microsoft Operations Management Server

  • Microsoft Internet Authentication Service

  • RADIUS Authentication


This guide is designed to use the Microsoft Operations Framework (MOF) process model in addition to the MOF Security Administration service management functions (SMFs).

In particular, the solutions presented in this document recommend the use of a continual process approach instead of a linear deployment approach to helping to improve network security against threats posed by unmanaged computers.

Specifically, the application of these solutions should entail the steps shown in the following figure:

Figure 1. Applying MOF

Figure 1. Applying MOF

As the figure shows, any responses to the security issues presented by unmanaged systems should be continual processes of testing and adjustment, not just issues of planning and deployment. The threats that can affect a midsize business network are always changing, and the systems that protect a business network must continually adapt to those threats.

Applying the solutions discussed in this document aligns with the Security Management SMF, which is outlined as follows:

  • Assess business exposure and identify which assets to secure.

  • Identify ways to reduce risks posed by unsecured devices to acceptable levels.

  • Design a plan to mitigate security risks.

  • Monitor the efficiency of security mechanisms.

  • Re-evaluate effectiveness and security requirements regularly.

To read more about MOF, refer to the Microsoft Operations Framework Web site on Microsoft TechNet at

For more information about the Security Management SMF, see the Security Management Functions page at

Risk management is the process of determining an organization’s level of acceptable risk, assessing current risks, finding ways to reach that acceptable risk level, and managing risk. Although this document deals with some risk management concepts, an in-depth discussion of risk management is a subject in its own right and thus deserves its own dedicated focus. For more information about risk analysis and assessments, see the Security Risk Management Guide at


Some of the challenges involved with protecting against unmanaged computers include:

  • How to prevent unmanaged computers from accessing network resources.

  • How to keep roaming computers compliant with updates and policies.

  • How to enforce established security policies on computers that connect to the network remotely.

  • How to protect a network against rogue wireless devices.

  • How to detect unmanaged systems, like vendor laptops or personal devices, when they connect to the network.

  • How to protect against other mobile devices like palmtops and handhelds.


The "Challenges" section listed a number of factors that must be accounted for when considering how to protect against the threats that unmanaged rogue computers pose to a network. In addition to defending the traditional internal wired network, steps must also be taken to help secure the wireless network and any remote access paths as well. To cover all the possible ways a rogue device can connect to a network, the solution needs to be comprehensive in scope.

The discussions in the following section deal with the listed challenges by addressing the following approaches:

  • Using IPsec domain and server isolation to prevent unmanaged computers from accessing network resources.

  • Using VPN quarantine to enforce security policies on computers that connect to the network remotely.

  • Using 802.1X authentication to protect against untrusted wireless devices.

  • Using SMS to detect unmanaged computers when they connect to the network.

Note   This document assumes that the reader has already implemented processes to ensure that member computers residing in the midsize business’ domain structures are compliant with any established security and patching requirements, and is researching methods to enforce compliance and protect against non-compliant devices. It is beyond the scope of this document to discuss how to bring computers into compliance or use automatic security update mechanisms like WSUS or the patch management features of SMS.


This section introduces you to some potential responses to help address the problem that unmanaged computers present to midsize businesses. It also discusses some of the decisions that must be made before committing to any of these responses. Any one of the discussed responses would help to enhance the level of network security in a midsize business network. However, when implemented as part of a comprehensive defense-in-depth approach, the combined responses create a more sophisticated response to the problems discussed in the "Challenges" section.

Using IPsec to Isolate Domains and Servers

Physically isolating computers and networks is not a new idea. It has been practiced in many forms through the years, most commonly where development and test networks are concerned. However, the older physical isolation approaches do not lend themselves well to the task of protecting managed systems from potentially compromised unmanaged devices. This is especially true in modern business network environments where mobile computing is gaining dominance and the demand for automated and flexible solutions is increasing. The following table outlines the common physical isolation methods that have been used in the past, along with the difficulties associated with their use in this capacity.

Table 1. Network Isolation Approaches

OSI layer



Layer 1

Use separate cabling for segments that need to be isolated from the trusted network.

  • Costs associated with running new cable for each new connection.

  • Increased administrative overhead to manage new cable drops for user moves.

  • Lack of flexibility and difficult to manage as business grows.

  • Increased likelihood of oversight and error due to high management requirements.

Layer 2

Use VLANs to create logical subnets isolated from the trusted network.

  • Increased cost associated with updating switch fabric to support VLANs.

  • Higher administrative overhead for tracking network changes, user moves, and responding to guest connection requests.

  • Prone to oversight and error when multiple user moves occur or mobile devices are used.

As this table demonstrates, the modern business network requires a more flexible solution to help enforce compliance with security standards that help protect against unmanaged systems. Although there are some third-party solutions to this problem, they involve client side installations on all managed workstations and add a layer of complexity to the network. Fortunately, current versions of Microsoft Windows have built-in functionality that helps to address this problem without the need for additional software installations and administrative learning curves.

Microsoft server and domain isolation allows IT administrators to restrict TCP/IP communications by leveraging built-in Active Directory Group Policy and IPsec, which results in the following benefits:

  • Uses the Network layer of the OSI model, which means that it can span switch and router boundaries.

  • Independent of switching fabric and physical network limitations.

  • Lowers cost by leveraging the built-in authentication features of Windows-based computers.

  • Automated, and does not require ongoing maintenance associated with network changes and user moves.

  • Provides the capability to not only authenticate trusted computers but also encrypt the communications between trusted systems.

  • Enforces security policies by refusing connections from unmanaged devices.

  • Improved auditing and resource management.

  • The ability to rapidly isolate specific resources if an attack occurs.

The typical concerns surrounding the use of IPsec to secure communications have been addressed by recent developments made in response to those issues. Specifically, network address translation (NAT) is no longer a concern due to NAT traversal capabilities available with current versions of Microsoft Windows, such as Windows Server 2003 and Microsoft Windows XP with Service Pack 2 (SP2). In addition, support for non-IPsec capable platforms is possible via configurable exemption lists and/or a fall back exception to allow clear text communication with those systems.

Domain and server isolation solutions, as presented in this document, are even used within Microsoft’s own internal network. In addition to recommending these solutions to customers, Microsoft depends on the additional level of security that this solution provides and has a long-term commitment to support this method of making networks highly secure and manageable for trustworthy computing.

Understanding Domain and Server Isolation

Simply put, the creation of an isolated network involves separating the various types of computers in a business network according to the type of access they should have. In this case, the various types are in two sets: managed and unmanaged. Two basic conditions must be met to achieve a functional level of network isolation. These conditions are:

  • Computers that reside in the isolated network can initiate communication with all computers on the entire network, including those that reside outside of the trusted network.

  • Computers that reside outside of the isolated network can only initiate communication with other computers outside of the trusted network; they cannot initiate communication with computers inside the trusted network.

IPsec domain and server isolation leverages existing domain structures by using Group Policy settings. Everything needed to create an isolated network already exists on computers that use Windows XP, Windows 2000 Server, and Windows Server 2003. When the necessary Group Policy settings have been configured, the process of inserting a new computer into the isolated network merely involves adding that computer to the domain.

Figure 2. Network Isolation Using Active Directory Domains

Figure 2. Network Isolation Using Active Directory Domains

As illustrated in the preceding figure, any computer that is not a member of the domain is considered to be untrusted and therefore resides outside of the isolated network. Any system residing outside of the isolated network cannot initiate IPsec communication and thus cannot establish a connection with any system inside the isolated domain. However, members of the isolated domain are free to establish clear text communication with devices outside of the isolated network.

Of course, many organizations have computers and servers that, while managed and trusted, do not exist within an Active Directory domain or are not capable of using IPsec for any number of reasons but still have a business need to communicate with systems inside of the isolated network. To resolve this dilemma, another isolated group (or boundary group) can be made by using exemption lists as shown in the following figure.


Figure 3. Network Isolation and Boundary Groups

A network isolation model such as this makes it possible to help reduce the security footprint of a business network. However, this solution by itself cannot guarantee that trusted systems will remain compliant with the standards that make them trusted. This solution is not a comprehensive network security solution in and of itself. It is a useful part of a larger comprehensive security process that, in conjunction with other solutions, can help reduce risks that modern midsize business networks contend with. Specifically, when used with other security solutions such as WSUS, SMS, MOM, and others, it is possible to use this isolation method to help enforce security policy compliance within the isolated network.

Other Active Directory–based security solutions make it possible to automate tasks that help ensure continued security policy compliance within the trusted isolated network. However, a boundary group will have to rely on different methods to ensure that computers within it do not become the weak points in the isolated network solution. Such computers must be validated as compliant before being added to the boundary group, and must have specific policies and methods associated with them that guarantee their continued compliance with trusted systems security policy requirements.

Understanding IPsec

IPsec is an Internet Protocol security standard that provides a general, policy–based IP layer security mechanism that is ideal for providing host-by-host authentication. IPsec policies are defined as having security rules and settings that control the flow of inbound and outbound traffic on a host system. These policies are managed centrally in Active Directory using Group Policy objects (GPOs) for policy assignments to domain members. They provide the ability to help establish secure communications between domain members, which is the basis for this solution.

IPsec uses the Internet Key Exchange negotiation protocol to negotiate options between two hosts to determine how to communicate securely with IPsec. The agreements made between two hosts and the various parameters that define this negotiation method are called security associations, or SAs. Microsoft Windows is capable of using one of the three following IKE methods:

  • Kerberos version 5 authentication protocol

  • X.509 digital certificates with corresponding RSA key pairs

  • Preshared keys using passphrases

Note   Although PKI can be used as an authentication method, the integration of Windows 2000 domain authentication (Kerberos) in the IKE negotiation protocol makes PKI an unrecommended approach.

Windows IKE negotiation can be configured to allow communication with computers that do not respond to the IKE negotiation request. This functionality is called fall back to clear, and in addition to being essential during a rollout it is also useful in this context because it allows systems in the Boundary Group to communicate with isolated network members. Any communication that IPsec cannot protect is referred to as a soft security association and is recorded as a Security Log success audit event.

IPsec can perform other useful functions besides authentication, including the ability to address the integrity and the encryption of network traffic by its use of Authentication Headers (AH) and Encapsulating Security Payload (ESP) options. Although AHs can be used to provide the ability to help ensure that a packet was not modified in transit, the use of headers to perform this task makes it incompatible with network address translation (NAT). The use of ESP, which is typically used to encrypt traffic, can be used in null encryption mode (ESP/null), which allows for data integrity checking that is NAT compatible.

IPsec is also capable of operating in two different modes, transport or tunnel. IPsec transport mode is the recommended method to secure traffic between two hosts. It works by simply inserting a header after the original IP header, and then encrypting the remainder of the packet with either AH or ESP. Tunnel mode is typically used for gateway-to-gateway VPN tunnels in which the original packets and IP headers are encapsulated entirely and a new IPsec header replaces the original IP header.

For more information, refer to the Microsoft Server and Domain Isolation page at

Defense in Depth

Figure 4. Defense-in-depth model with logical isolation

Figure 4. Defense-in-depth model with logical isolation

The preceding figure demonstrates how logical isolation fits into the defense-in-depth model. Although domain and server isolation appear to focus on securing the host computer, as would a host-based firewall, it resides within the realm of network communications with its use of IPsec. Although this solution spans the gap between host and internal network, it does not reside entirely within either set because it is a ‘logical isolation’ solution. Therefore the following functionality is outside of its scope:

  • Logical isolation cannot secure network devices, such as routers.

  • Logical isolation cannot secure network links, such as what 802.11i WPA for wireless encryption and 802.1x EAP-TLS for wireless access control do.

  • Logical isolation cannot provide security for all hosts on the network, because it only governs those systems with a trusted machine credential and domain–based IPsec policy.

  • Logical isolation is not well suited for and has no automatically configurable method that can secure application level paths, such as what HTTPS and SSL do for Web applications.

It is important to understand and consider what part logical isolation plays within a comprehensive defense-in-depth solution. By itself, this solution cannot secure all aspects of a midsize business infrastructure. However, when used as a part of a comprehensive solution, it can play a very valuable role.

Using VPN Quarantine Services to Protect Against Unmanaged Remote Computers

Remote systems that use VPN solutions to access a business network pose unique problems from a security policy perspective. Most remote access solutions only validate the credentials of a remote user but do not verify that the remote computer complies with security policies. This validation approach is a problem because it potentially negates efforts to protect a network from threats associated with issues like outdated anti-malware signature files, out-of-date security update levels, improper routing settings, and a lack of adequate firewall protection.

Microsoft Windows Server 2003–based Network Access Quarantine Control helps address this problem by delaying remote access to a VPN until the configuration state of the connecting computer can be verified as being compliant with current security policy standards.

Understanding VPN Quarantine Services

Network Access Quarantine Control is a feature provided with the Windows Server 2003 family that delays normal remote access services until connecting remote computers are examined and validated by an administrator-configured script. Typically, when a remote session is initiated the user is authenticated and the remote computer is provided an IP address, but at this point the connection is placed in quarantine mode in which network access is limited until the administrator-provided script is run on the remote computer. After the script has run and notifies the remote access server that the remote computer complies with the configured network policies, the quarantine mode is removed and the remote computer is granted access to the network.

While a remote connection is in quarantine mode the following restrictions can be put on that connection:

  • A set of quarantine packet filters can be configured to restrict the type of traffic that the client can initiate or receive.

  • A quarantine session timer can be configured to restrict the amount of time that a client can remain connected while in quarantine mode.

Network Access Quarantine Control can be used as part of a comprehensive security solution that assists with the enforcement of security policies and helps ensure that untrusted systems cannot connect to a trusted network. However, it is not a security solution in and of itself and cannot prevent connections from malicious users who have obtained a valid set of credentials.

Note   Network Access Quarantine Control is not the same as Network Access Protection (NAP), a new policy enforcement platform that will be included in the upcoming Windows Server Longhorn operating system family. For more information about NAP please refer to "Appendix A: Network Access Protection" at the end of this document.

Network Access Quarantine Control Components


Figure 5. Windows Network Access Quarantine Control Components

The preceding figure depicts the typical components of a Windows remote access solution that uses Network Access Quarantine Control. These components include:

  • Quarantine compatible remote access client. This component is a computer running a Windows version that is capable of supporting CM profiles created with the Connection Manager Administration Kit (CMAK), which is provided with Windows Server 2003. Windows versions that provide this support include Windows 98 Second Edition and newer. These client computers need to have a notifier component installed, a network policy requirements script, and be configured to run the script as a post-connect action.

  • Quarantine compatible remote access server. This component is a Windows Server 2003 computer running Routing and Remote Access, which supports the use of a listener component and the MS-Quarantine-IPFilter and MS-Quarantine-Session-Timeout RADIUS vendor specific attributes (VSAs) along with an installed listener component.

  • Quarantine compatible RADIUS server. This component is optional, and is used when RADIUS authentication is configured on the remote access server that runs Windows Server 2003 and IAS to support the MS-Quarantine-IPFilter and MS-Quarantine-Session-Timeout RADIUS VSAs.

  • Quarantine resources. These components are the servers and services that the remote client can access while in quarantine mode. Typically, they consist of servers that provide DNS services, CM profiles, or security update providers to bring clients into compliance.

  • Account database. Typically, this component would be the Active Directory domain where the remote user’s account is stored and dial-in properties are located.

  • Quarantine remote access policy. This policy is necessary to provide the required conditions for remote access connections to occur and to specify the MS-Quarantine-IPFilter or MS-Quarantine-Session-Timeout attributes.

Defense in Depth

Figure 6. Defense in depth and Network Access Quarantine Control

Figure 6. Defense in depth and Network Access Quarantine Control

As this adaptation of Microsoft’s defense in depth model shows, VPN quarantine spans three layers of the model; host, internal network, and the perimeter. Although this solution does not directly secure the client, it does help secure servers from threats that could be introduced by remote clients that do not meet security guidelines. It also indirectly secures remote clients by ensuring that they meet policy requirements to work effectively, thereby providing encouragement to keep updates current.

Also, this solution improves the protection of the network itself from the disruptive effects that rogue remote computers can create when not configured correctly, including the propagation of malware, which can have deleterious effects on network performance. Although the perimeter may seem unrelated because this solution does not directly prevent brute force attacks on the VPN (which does reside at the perimeter), it does offer another layer of security that an attacker would have to contend with to gain direct access to resources residing in the internal network—even if that attacker gained authentication credentials.

Using 802.1X Authentication to Protect Against Unmanaged Wireless Clients

The use of the Institute of Electrical and Electronic Engineers (IEEE) 802.1X standard is well suited to the task of helping to defend wireless networks from unauthorized usage. To put it simply, unless a computer is configured to be authorized to communicate on the network it simply will not communicate. Although this solution works well on wireless networks, it is not as effective on wired networks even though it is possible to use this standard in such a manner. Its limited effectiveness is why this prescriptive document recommends a combined approach for securing a midsized business network. By using a combination of the most effective solutions that help address each aspect of a typical business network, a robust defense-in-depth solution can be implemented.

Understanding IEEE 802.1X Authentication

IEEE 802.1X authentication is a port–based access control mechanism that can be configured to require mutual authentication between clients and a network. When implemented, any device that fails to authenticate cannot participate in any communication with the network in question.

802.1X supports the Extensible Authentication Protocol (EAP) which has a number of variants including three that are supported by current versions of Microsoft Windows out of the box:

  • EAP-MD5. With MD5, an authentication server sends a challenge to a client (supplicant) and the supplicant combines this challenge request with its own identifier and passphrase. This response is turned into an MD5 hash and sent back to the authentication server, which decrypts and matches the response to the original challenge. If this response matches, authentication occurs. This method is not secure in most implementations because the password itself is sent and can be intercepted and decrypted by third parties.

  • Protected EAP. Protected EAP (PEAP) uses a server side certificate and authentication information from the client (such as user names and passwords) to establish mutual authentication. The Microsoft implementation of this standard uses MS-CHAPv2, which relies on traditional domain account and password information and a RADIUS server to establish this authentication. This method is usually considered sufficiently secure for smaller environments that cannot afford the additional infrastructural and administrative requirements that come with the EAP-TLS method.

  • EAP-TLS. With this method, an authentication server sets up a transport layer security (TLS) session with the supplicant to send the client a digital certificate. The supplicant validates this certificate when it is received and then sends its own certificate in response, which in turn is validated by the authentication server. As long as each side trusts each other’s certificates, authentication occurs. This approach is best suited for networks that can support a PKI infrastructure and RADIUS authentication servers. It is also the most secure option available for wireless networks, especially when combined with the use of the 802.11i WPA2 standard.

Before a client authenticates, it must rely on the authenticator (a wireless access point or network switch) until it is authenticated by the authenticating server. Therefore, during the initial authentication handshake all communication is forwarded by the authenticator between the supplicant and the authentication server. When authentication is complete the supplicant can access the network directly.

Why 802.1X Is Not Effective for Wired Networks

Although 802.1X generally helps address the security of wireless networks, there are some problems associated with implementing this method to secure wired networks. The first problem is that although 802.1X has several Active Directory Group Policy objects that support its implementation on wireless networks, it does not support 802.1X authentication on wired networks. Therefore, 802.1X would require a significant degree of management overhead for such an implementation. In addition, many switches, network print devices, and other wired network infrastructure devices do not support 802.1X, so it is very likely that costly upgrades would have to be implemented to support this implementation on most midsize business networks.

Besides the administrative and infrastructure costs associated with such an implementation, there are also some security flaws that are introduced when this wireless focused standard is used on wired networks. For example, since 802.1X authentication only occurs when a connection is established and hubs are invisible to 802.1X, all an attacker would have to do is attach a hub to the internal network along with an authenticated computer and a “shadow” computer with which to conduct an attack.

Note   These specific disadvantages and vulnerabilities do not affect wireless networks. These issues with 802.1X security are only introduced when it is used on wired networks.

These issues are the reasons why IPsec is recommended for domain and server isolation instead of 802.1X. However, this recommendation does not mean that there is no place for 802.1X in a midsize business network. As stated earlier, 802.1X is a valuable tool that helps address wireless security issues and is only made more secure with the addition of an IPsec–based network isolation solution.

Defense in Depth

Figure 7. Defense in depth with 802.1X wireless security

Figure 7. Defense in depth with 802.1X wireless security

As the preceding figure illustrates, wireless security using 802.1X authentication occupies the network layer of the defense-in-depth model. Although it might seem as though wireless security would span the perimeter as well, wireless networks are actually a portion of the local network while the perimeter deals more with external networks such as the Internet. Some portions of wireless security methods have host components, but wireless security is not designed to protect the host directly.

Using SMS to Detect and Remediate Unmanaged Systems

Microsoft Systems Management Server (SMS) 2003 is often used to manage computers on the network, maintain asset inventory information, and deliver software and updates to maintain compliance with security policies and maintain platform and software installation commonality. The network discovery and inventory functionality of SMS 2003 are within the scope of this document because they provide a method that can help detect untrusted systems when they connect to the network. This functionality also provides a way to help with remediation, depending on what policies are implemented as a response to unmanaged computer usage.

This document anticipates that the reader has used SMS and other methods to bring domain member computers into compliance with security policies, which would include being current on all security updates. Therefore, discussions concerning patch management using SMS 2003 and other tools are beyond the scope of this document. For more information regarding these topics and other SMS-related information, refer to the Patch Management Using Systems Management Server 2003 solution accelerator at or the Microsoft Systems Management Server home page at

Discover and Document Existing Assets

For businesses that have already implemented SMS, the documentation of existing assets has probably already been completed and an inventory of managed and unmanaged systems is complete. Such an inventory should contain information about the security update requirements and procedures for computers that are not managed by SMS, whether by design or because there are no working agents for that client type. These unmanaged computers can include devices in a security perimeter (also known as DMZ, demilitarized zone, and screened subnet), test computers, or stand-alone devices.

It is important to have a well-documented list of devices that are not managed by SMS, not only because of security requirements, but because such a list needs to be used to compare with any new inventory information to determine whether newly discovered systems are authorized or unknown. To keep such lists current, it is important to have a process in place for the addition of new unmanageable systems to the business network that includes not only identifying information but also the process involved with keeping those systems secure along with ownership assignments.

Although SMS might not be able to gather much information on such unmanaged systems beyond just an IP address and a name, this information alone is often sufficient to determine whether new network devices are authorized or not.


The "Assessment" section has shown that a blend of different solutions that address specific functions on a midsize business network is the most comprehensive approach to help defend that network. Domain and server isolation covers the wired network to ensure that only known and managed computers can access trusted resources. VPN Quarantine ensures that remote systems that only connect occasionally to the local network remain compliant with security policies. 802.1X authentication secures the wireless network so that only authorized machines can establish connections. Finally, SMS is used to help keep trusted computers compliant and discover untrusted rogue computers when they connect to the network. All of these solutions combine to help protect a network against unauthorized connections and rogue computers.

In addition to discussing the requirements for implementing these solutions, this section will also address some of the potential issues that need to be addressed to help ensure that each solution is the most appropriate approach for each midsize business environment.

Using IPsec to Isolate Domains and Servers

Developing a plan to implement IPsec–based domain and server isolation involves determining whether or not this solution is feasible for a given network environment as well as gathering the prerequisite information required to develop an implementation plan. The concept of IPsec network isolation was introduced in the "Assessment" section of this document. Now that the benefits of this approach are known, it is possible to weigh them against any possible issues that can be associated with such an implementation. This section will discuss some issues involved with the implementation of IPsec–based network isolation along with the requirements of such an implementation to assist with that decision-making process.

Identifying Network Architecture

Because IPsec is layered on the Internet Protocol itself, a detailed understanding of the current network infrastructure and traffic flow is essential to a successful deployment of this solution. Although information regarding the IP addressing schemes, subnet layouts, and network traffic patterns are important components that need identification, a detailed documentation of network segmentation and a current network traffic flow model is absolutely essential to successfully developing and deploying an effective network isolation plan.

Note   It is beyond the scope of this document to discuss the details of documenting the network environment. The creation of such documentation is vital to the success of any comprehensive network security initiative, including one such as this. Therefore, if no such documentation exists, projects such as this should be considered an undue risk to implement effectively until such documentation initiatives have been completed.

The network architecture documentation should accurately reflect the current details with regard to the following information:

  • Firewall and load balancer details and locations

  • Network device information including RAM size, make/model, and MTU settings

  • Network traffic and utilization reports

  • NAT usage

  • VLAN segmentation

  • Device network links

  • Intrusion detection system (IDS) information

Identifying Network Traffic Flow

In addition to information about devices and configurations that can affect IPsec–based network isolation, it is important to examine network traffic patterns within the network. When gathering this type of information it is important to consider factors such as how non-Windows devices (such as Linux computers) or devices that use versions of Windows older than Windows 2000 SP4 communicate with other Windows-based systems. Additional considerations include:

  • Are there subnets devoted to specific types of traffic, such as mainframe communications?

  • Does the traffic between locations need to be separated?

  • Are there some types of traffic that require the high level of security that encryption provides, such as communications with an HR database?

  • Are there any security devices or firewall rules that may impact implementation, such as blocking UDP port 500?

  • How are applications communicating? For example, do they use Remote Procedure Call (RPC), which may need additional consideration?

  • How are hosts and servers communicating with each other? Do they use port/protocol level connections or do they establish multiple sessions across a wide array of protocols?

Identifying Active Directory Structure

To understand the impact that IPsec implementation may have on an Active Directory structure, it is important to gather information about the Active Directory forest structure, domain layout, organizational unit (OU) design, and the site topology in addition to the network information identified earlier. This information should include the following:

  • Number of forests

  • Number and names of domains

  • Number and types of trust relationships

  • Number and names of sites

  • OU and Group Policy structure

  • Any existing IPsec policy information (if IPsec is currently in use)

Identifying Hosts

Some of the relevant host information that needs to be gathered includes:

  • Computer names

  • IP addresses, especially static addresses

  • MAC addresses

  • Operating system version, including service pack and hotfix revision levels

  • Domain membership

  • Physical location

  • Hardware type and role

Identifying IPsec Capacity Considerations

There are some capacity planning considerations that need to be made when developing a plan to implement IPsec, particularly when contemplating the use of IPsec encryption because it requires a certain amount of host utilization and network overhead. Some considerations include:

  • Server memory utilization. Each session can consume as much as 5 KB of RAM.

  • Server , workstation , and infrastructure device CPU utilization. This information is especially significant for servers. Ensure that devices are not already over-utilized.

  • Network latency and utilization. Latency will be increased when using IPsec. When Microsoft deployed IPsec, network utilization increased by 1% to 3%.

  • Use of NAT and encryption type. AH communication cannot traverse NATs. Therefore, ESP would have to be used to encrypt communications instead. ESP has a higher utilization overhead than AH encryption unless ESP null encryption is used.

  • Group Policy impact. IPsec GPOs have an impact on computer startup and logon times. Baselines should be taken before and after test implementations to determine the actual effect on those times.

Identifying Levels of Trust

Another important consideration involves determining which hosts participate in this solution and in what capacity. To determine this it is necessary to establish a clear definition of what conditions must be met to be trusted and what degrees of trust may exist. The following information can help this process by offering some clear definitions of trust, the criteria necessary to meet given levels of trustworthiness, and how these affect the planning process.

There are four basic states that can be applied to a host with regard to levels of trust. In order of trustworthiness from high to low, these states are:

  • Trusted. A trusted status implies that a host’s security risks are managed and that other trusted hosts should be able to reasonably assume that a malicious act will not be launched from that host. Although this state does not necessarily mean that this host is completely secure, it does mean that this host’s state is the responsibility of an IT department and is managed by some mechanism, whether automatic or a documented process.

    Because this host is trusted, the communication between this host and other trusted hosts should not be impeded by the network infrastructure. Because it is safe to assume that no malicious act would be committed by this host, there is no need to block or filter traffic from the host by using firewalls or similar measures.

  • Trustworthy. A trustworthy state indicates that although a computer is capable of becoming a trusted device, it still requires either some additional configuration changes or upgrades of some sort to be certified as trusted. Identifying such computers is especially important during the design phase, because they provide some indication about the amount of effort and cost that may be required to establish an isolated network.

  • Known , Untrusted. There are a number of reasons why a known computer might not be capable of becoming a trusted asset, ranging from financial restraints, business requirements, or even functional requirements. For example, the project funding may be insufficient to upgrade all computers, some business units may have ownership of their own domains, or an older application may be incompatible with currently supported operating systems and therefore cannot run on a secured computer.

    Regardless of the reason, business owners of known but untrusted computers should be consulted to determine whether there is anything that can be done to bring a computer into compliance and to determine how to address this situation while maintaining an acceptable risk profile.

  • Unknown , Untrusted. The unknown, untrusted state should be the default state for all hosts. Hosts in this state are not managed and their configurations are unknown, therefore no other trust state can be assigned to them. It should be assumed that these hosts are capable of being compromised and can be used as platforms for malicious activity, and therefore pose an unacceptable level of risk to the company. This solution is designed to reduce the potential impact of such systems.

Using the Gathered Information

When all of the relevant information has been gathered it will be possible to develop an implementation plan and determine whether the IPsec–based network isolation solution is appropriate for the present network environment. Some of the considerations that may influence the implementation of this solution include the following:

  • Cost of upgrades needed to bring hosts into compliance with a trusted state and be compatible with IPsec requirements.

  • Cost of infrastructure upgrades or replacements if current network devices do not support IPsec.

  • Balancing the possible loss of router–based QOS and the security benefits of network isolation, because port–based prioritization will not work when IPsec is implemented but IP address–based QOS will continue to function.

  • Network monitors and IDS devices may fail when IPsec is implemented, especially when ESP encryption is used. Parsers can be used to overcome this problem if the device has an IPsec parser available.

  • Implementation of IPsec may require hardware upgrades because of the increased overhead and demands on CPU and memory for servers and network devices.

  • Expectation management may be an issue, because network latency may increase.

  • Vendor and guest user management may also generate excessive overhead because these users have untrusted devices and may have business reasons to access network resources on the isolated network. Such exceptions can become difficult to manage as their numbers and frequency increase, but can be mitigated by use of a boundary zone between trusted and untrusted networks.

These issues illustrate why careful planning and testing is necessary before implementing changes that can affect the entire network, such as the implementation of IPsec and network isolation. Careful consideration should be given to determine not only how to implement this solution, but also whether it is feasible to implement because of the network’s current state and the financial and political costs of bringing the network into compliance. In addition, limited or incremental deployments that only isolate targeted resources should be discussed as a possible mitigation strategy to help address such concerns.

Again, IPsec–based network isolation is but one part of a comprehensive solution. Each solution in this guide helps defend against untrusted devices that can connect to the network, but each part of this solution can stand on its own and still increase the security level of any given network. Therefore, even if IPsec domain and server isolation is not an acceptable solution at this time, it can still be useful to implement the other solutions listed here and perhaps come back to network isolation after a network has become more mature and compatible with the requirements outlined in this document.

Using VPN Quarantine Services to Protect Against Unmanaged Remote Computers

In a standard Windows–based remote access server session, the server will perform the following actions when a remote access client initiates a session:

  1. The remote access server performs checks against configured remote access policies and allows the connection.

  2. The server verifies the user’s remote access permissions.

  3. The server then authenticates the user credentials against Active Directory or some other authentication service.

  4. The server assigns an IP address to the remote client.

At this point the remote client has full access to the network, and no checks have been run to verify update levels, the existence or update status of antivirus software, or any other security policy related information. This functionality is less than optimal, because it sometimes means that updates, configuration changes, or patches do not get applied to remote users or roaming computers.

A VPN Quarantine connection is different, however, as shown in the following figure and numbered list:


Figure 8. VPN Quarantine process

  1. The remote client runs a Connection Manager pre-connection script that checks basic connection prerequisites (such as security update level and virus signature file date) and stores the results. After running this script, the client establishes a remote access session through a VPN tunnel.

  2. The remote access server authenticates the user credentials via RADIUS, which checks Active Directory to verify the credentials.

  3. After Active Directory has authenticated the user, the remote access server places the client in a quarantine state by applying the remote access policy. While in quarantine, network access is limited to what the quarantine policy specifies. This limited access may be achieved through an IP filter that restricts access to specific resources that can be used to bring a client into compliance or by specifying a timeout period that will disconnect the client after a period of time.

  4. A post-connection script notifies the remote access server whether or not the client complies with specified requirements. If the connection does not meet requirements within a time-out period, the user is notified of the failure details and the connection is dropped.

  5. If the post-connection script indicates that the client does meet specified requirements, then the remote access server removes the client from quarantine mode by removing the IP filter restrictions and thereby granting permission to access network resources specified by the remote access policy.

A remote access client can only access resources that are located within the specified quarantine network, whether that network is a separate subnet or a defined set of Internet-facing servers. A quarantine network should provide resources that would enable a remote client to perform activities to bring a remote computer into compliance with security standards. Generally, these resources would include access to a DNS server for name resolution, a file server to retrieve updates from, and perhaps a Web server for instructions or Web–based updates.

The use of a quarantine subnet in this process would require longer session time-out settings to ensure that there was sufficient time to download and install updates to the remote client computer. To avoid this problem, the client computer could be directed to use other sources or use Internet-facing update servers outside of the VPN session, although this approach would require a more complex script and could present other security issues. In either case, it is the script that determines the quarantine behavior, not the quarantine network itself.

Note   IPsec policies can be scripted if the quarantine solution has to accommodate non-domain joined systems. In these cases, netsh or lpseccmd.exe can be used to apply simple policies with certificates to support IPsec authentication. IPsec can also work for domain-joined systems in VPN quarantine configurations as well as a layered solution.

VPN Quarantine Client Components

As noted in the previous section, the first step in the VPN quarantine process begins with the client, specifically with the Connection Manager pre-connection script. Connection Manager is a part of the Connection Manager Administration Kit (CMAK) that centralizes and automates the establishment and management of network connections and supports several key areas of a VPN quarantine configuration, including:

  • Pre-connection security checks to manage client computer configurations automatically.

  • Post-connection security checks and logon validations.

Connection Manager allows administrators to set custom actions via profiles that can be distributed to multiple computers. This capability simplifies the connection process for remote users by limiting the number of settings they can change. Some of the settings that can be changed include:

  • Establishing lists of phone numbers that can be used for dial-up connections.

  • Customizing graphics, icons, messages, and Help text.

  • Executing custom pre-connect and post-connect actions, which can include activities such as resetting the dialer profile or configuring Windows Firewall packet filter rules.

Another CMAK component is the client agent (RQC.exe), which communicates with the Remote Access Quarantine Service (RQS.exe) on the remote access server using TCP port 7250. When a quarantine connection is established, the RQS sends the RQC a secret shared key. If the client meets the necessary conditions, the RQC sends the shared key to release the client from quarantine.

The Connection Manager profiles are self-extracting customized Connection Manager client dialer packages that can be created and configured by using CMAK. The CMAK wizard guides administrators through the process of configuring the profile, which can then be distributed via a number of mechanisms ranging from Group Policy to Microsoft Systems Management Server (SMS) 2003 software distribution. When the created executable is run on the client computer, it installs the profile on the local computer along with the settings necessary to establish communication with remote access servers. When the installation is complete, the user only needs to initiate the profile name in the Connect To menu in Windows XP to establish a connection.

For more information about CMAK, refer to the Connection Manager Administration Kit page on Microsoft TechNet at

VPN Quarantine Server Components

The VPN remote access server is the central part of this solution and fulfills the following functions:

  • Runs the Remote Access Quarantine Service (RQS.exe).

  • Applies remote access policies for quarantine access settings.

  • Negotiates communication with the client agent.

  • Receives policy compliance information from the client agent.

  • Applies remote access policies to determine access permissions to network resources.

Remote Access Quarantine Service is an optional component in Windows Server 2003 with Service Pack 1 and supports the necessary APIs to place remote client computers into quarantine and then remove them from quarantine mode after the client agent sends notification of policy compliance. This service (RQS.exe) listens on TCP port 7250 for notification from the client-side component (RQC.exe) that the client computer meets policy requirements. This functionality means that any firewalls, including host-based firewalls, must allow traffic on TCP port 7250 for this solution to function.

VPN Quarantine Network Components

The following network components include both required and optional pieces as noted that can be part of a VPN quarantine network solution.

  • Internet Authentication Service (IAS) RADIUS server (Recommended). Although IAS servers are only needed if RADIUS is used as the authentication provider, they do have some compelling advantages over other authentication solutions including support for Extensible Authentication Protocol to support certificate and smart card based two-factor authentication. If RADIUS is used, it is recommended to use IAS that comes with Windows Server 2003 as the RADIUS provider because it supports MS-Quarantine Session Timeout and MS-Quarantine-IPFilter VSAs; other RADIUS servers may not.

  • Active Directory (Recommended). Active Directory serves as the user account authentication database for RADIUS and integrates with IAS and other services. When used in combination with IAS, it can support two-factor authentication with certificates or other methods.

  • DHCP server (Recommended). DHCP servers are used to provide IP address provisioning to remote clients.

  • DNS server (Recommended). A DNS server provides name resolution services so that clients in the quarantine network can connect to other servers (if provided) that can help bring remote client computers into compliance.

  • Windows Software Update Service (WSUS) server (Optional). WSUS servers can supply the necessary security updates and hotfixes that remote computers may need to meet the requirements for being released from a quarantined state. Other methods could be used as well, including SMS 2003 or even standard Windows Update services if computers need to be directed to external sources for updates instead.

  • File server (Optional). File servers can be used to provide shares where quarantined computers can access software updates and antivirus signature files to meet policy requirements.

  • Web server (Optional). Web servers can be used in quarantine to provide user instructions for removing their computers from a quarantine state. Some update packages also use Web components for delivery, such as some antivirus products.

Using 802.1X Authentication to Protect Against Unmanaged Wireless Clients

The previous section introduced 802.1X authentication as the most effective option available to help secure wireless networks from untrusted computers. The various 802.1X methods that are natively supported in current versions of Microsoft Windows were described, and an explanation of why 802.1X authentication is not so effective for wired networks was provided (although it is very effective for wireless networks). With that information, it is possible to discuss the differences between the supported methods and determine which might be more suited to different types of midsize business network environments.

802.1X Comparisons

As discussed, there are three 802.1X methods that are supported in current versions of Microsoft Windows. These methods are:

  • EAP-MD5

  • Protected EAP (PEAP)


Of these three methods, the first (EAP-MD5) is considered the least secure method because the passphrases used in this approach are transmitted over the air and therefore could be intercepted. The other two options (PEAP and EAP-TLS) are worthy of consideration for deployment in midsize business networks.

The Microsoft implementation of PEAP uses Microsoft Challenge Handshake Authentication Protocol version 2 (MS-CHAP v2), which was originally designed as a dialup VPN authentication method but lends itself well to PEAP because it can support the strong user password policies available through Active Directory. This approach is simpler than EAP-TLS to implement and costs less in terms of resources and initial cost, because there aren’t as many additional servers and services required for implementation. Although this solution does require certificates for the authentication servers, these certificates could be generated by third-party certificate providers because there aren’t too many that are needed for implementation.

However, EAP-TLS is considered the most secure implementation of 802.1X authentication available, because it requires certificates on both the client and the authentication server for mutual authentication. Because a large number of certificates would be required for such an implementation, it is suggested that a public key infrastructure be implemented to support it if one does not already exist. Therefore, EAP-TLS is viewed to have higher implementation and support costs when compared to PEAP with MS-CHAP v2.

The 802.1X Decision Process

To ease the decision process with regard to which implementation of 802.1X is best for any given network environment, Microsoft designed the following 802.1X authentication decision tree.

Figure 9. The 802.1X decision tree

Figure 9. The 802.1X decision tree

This flowchart can cause some confusion by its use of the term “large business.” This term should be described as generally meaning any business with at least 500 employees and a supporting IT staff of at least 5 technology practitioners. Given this information, it is clear that either solution would be appropriate for a midsize business environment. Therefore the decision process evolves more along lines of risk appetite and cost effectiveness.

The key difference between PEAP and EAP-TLS involves the need for a certificate infrastructure, which is why EAP-TLS is generally considered to be the option more in line with larger business needs and PEAP is considered to be generally sufficient for smaller businesses if no regulatory requirements exist that demand more stringent security standards. Aside from cost considerations, sometimes organizations have additional needs for a certificate infrastructure, which provides additional justification for an investment in PKI. After all, if many certificates are needed for secure Web content or code signing purposes, it is only reasonable to use that infrastructure investment to implement the most secure form of 802.1X authentication as well.

802.1X PEAP Requirements

A Microsoft–based PEAP with MS-CHAP v2 implementation would require at least two IAS RADIUS servers to act as authentication servers to authenticate wireless clients against an Active Directory account database. The number of RADIUS servers may require adjustments based on the number of remote sites or to accommodate larger numbers of user authentication attempts.

The existence of remote locations does not automatically require additional IAS RADIUS servers. However, if remote locations do not have redundant high-bandwidth connections or already have their own domain controllers and other local resources, then additional authentication servers should be added for such sites.

802.1X EAP-TLS Requirements

As mentioned earlier, EAP-TLS requires more resources than PEAP implementations because of the additional certificate requirements. Of course, it is possible to use third-party providers to issue the requisite certificates for all wireless clients and authentication servers, but this approach is more costly than implementing a certificate infrastructure except when the number of wireless clients is strictly limited to a handful of users.

In total, a simple EAP-TLS implementation will require at least four servers, more if required for larger companies or for geographically distributed networks. Two of the four servers will act as redundant IAS RADIUS servers and the other two will function as the certificate infrastructure. Of the two certificate servers, it is recommended that one (the root certificate server) be built off the network and remain disconnected from the network, which can mean that the number of servers can be reduced by one in certain circumstances.

Using WEP or WPA

Another issue with regard to wireless security that should be addressed is whether or not to use wireless encryption, and if so, what standard should be used. For a midsize business environment there are very few reasons why the lack of wireless encryption should be considered—reasons that are associated with businesses that provide public access wireless Internet connections. Otherwise, to help ensure the security of a wireless network it is imperative that some form of wireless encryption be implemented along with 802.1X based authentication.

Wireless encryption comes in the following two main forms, with one additional variant. These forms are:

  • WEP. The Wired Equivalent Privacy standard was part of the original IEEE 802.11 standard that uses 64-bit or 128-bit RC4 encryption to secure wireless communications. Serious flaws were discovered that make this encryption method extremely vulnerable to passive listening attacks. A number of tools are readily available that are capable of decoding WEP transmissions in a relatively short period of time, sometimes in a matter of minutes. Generally, the use of WEP is not recommended for business environments, but it can be secured somewhat by using various methods, including the use of 802.1X authentication.

  • WPA. As a response to the weaknesses inherent in the WEP standard, a consortium of industry leaders that included Microsoft and the Wi-Fi Institute created the Wi-Fi Protected Access (WPA) standard as a partial implementation of the 802.11i standard that was still under development at the time. This standard provides much stronger encryption that uses Temporal Key Integrity Protocol for data encryption, which is far superior to the flawed WEP standard. Most wireless access points (WAPs) on the market today support the WPA standard, as do all current versions of Microsoft Windows operating system.

  • WPA2. The Wireless Protected Access 2 (WPA2) standard was established in September of 2004 by the Wi-Fi Alliance. It is certified as a full implementation of the IEEE 802.11i specification, which was ratified in June 2004. This standard supports 802.1X–based EAP authentication methods or PSK (sometimes referred to as personal mode WPA) technology, but introduces the advanced encryption mechanism that uses Counter-Mode/CBC-MAC Protocol (CCMP) that is called Advanced Encryption Standard (AES). This implementation of wireless encryption is extremely secure. Most vendors support WPA2 in some form, including current versions of the Microsoft Windows operating system.

If at all possible, WPA2 or WPA should be used to help secure wireless data. However, it may not be possible to do so because these standards are newer, especially if a significant wireless investment was already made that does not support them. As mentioned earlier, it is possible to increase the level of security offered by the WEP standard by configuring 802.1X authentication and frequent key-pair changes. Even so, however, the use of WEP should be reserved for wireless networks that are in a state of transition to the WPA or WPA 2 standard.

Using SMS to Detect Unmanaged Systems

Microsoft Systems Management Server (SMS) 2003 is a very versatile management tool in a Microsoft network. With SMS it is possible to centralize several management functions, including software delivery, security update deployment, and systems auditing. Central to these functions is the ability to have an automated and centralized systems inventory from which to deploy these incredibly useful tools, which is where the detection of unmanaged systems comes into play.

SMS 2003 provides different discovery methods that administrators can use to build the computer collection database, which holds information about every computer on the network. These discovery methods range from the Active Directory–based discovery method (which updates the collection with details provided by the Active Directory computer accounts list) to the network discovery method (which actively probes the network for any connected devices). These discovery methods are configurable to automatically occur at preordained intervals and update the collections databases when finished.

To fulfill the need to detect unmanaged systems when they are connected to the network, it is possible to configure the network discovery to occur more frequently and then review the findings at regular intervals to determine when a new system was connected to the network. When such system connections occur they can be checked against any established list of new computer builds or network connection requests to confirm that the new system is authorized to use the network.

Network Discovery Process

One of the drawbacks to this approach of discovering unmanaged systems is that SMS may have difficulty discovering the details of computers that use operating systems other than Microsoft Windows. In fact, even versions older than Microsoft Windows 98 SE may go undetected by SMS 2003 discovery methods, because they do not support the Windows Management Interface (WMI) implementations. Accordingly, it is necessary to add scripts to this solution process so that any new device is detected when it has been attached to the network. The following figure illustrates how such a process would work.

Figure 10. SMS unmanaged computer discovery process

Figure 10. SMS unmanaged computer discovery process

This figure shows the discovery methods that could be used to add different computer types into the SMS collection database. There are three different types of computers that have to be contended with, including:

  • SMS accessible managed systems. These computers are capable of being managed by SMS and have already been placed under SMS management. They already exist in the SMS collection database and should be configured for automatic management. They would likely be considered part of the trusted network and don’t require further attention.

  • SMS accessible unmanaged systems. These computers are capable of being managed by SMS but have not been placed under the management of SMS. They may or may not be detectable by SMS, depending on their placement on the network (for example, behind firewalls or not), and may include computers that are managed by other means. These computers should exist in an exempted list, complete with management details. If they do not, then they should be considered untrusted and may require further investigation. They can be discovered by SMS, barring any network device that blocks such activity.

  • SMS inaccessible unmanaged systems. These computers are not capable of being managed by SMS and are not under the control of SMS. They may or may not include known systems that are managed by other means. These computers should exist in an exempted list that includes management details. If they do not then they should be considered unknown and untrusted and require further investigation. These computers can only be detected by using processes other than SMS discovery, including customized discovery scripts.

At this point it should be clear that the process of discovering unmanaged systems on the network depends on the establishment of processes that documents the details of every authorized computer connection on the network. Such documentation should include details about the computer, whether or not it is managed by automated methods (such as SMS), and if not then what process is used to keep the computer in compliance with established security policies, if any. Of course, an authorized computer could exist on a network but not be required to meet security standards, but such computers should be forced to remain in an untrusted network segment that is isolated from the trusted network.

When a documented “computer add” process is in place that documents all computers known to exist on the network in an authorized manner, it is possible to use this information to determine what is authorized and what is not. Any computers discovered on the network that do not exist on this list should be considered suspect and investigated immediately.

Deployment and Management

The "Development" section provided information about what details are needed to create plans for deployment. This section details some of the common deployment issues and processes that can assist technology implementers when implementing these solutions or help decision makers better understand what these solutions entail so that they can make an informed decision about whether they are appropriate for a given environment.

Using IPsec to Isolate Domains and Servers

Because of the complexity associated with IPsec–based network isolation, it is suggested that implementers take a phased approach to this solution’s deployment in a production network. There are a couple of ways to accomplish phased deployment, and they are based on how IPsec policies are deployed—by groups or by policy.

Regardless of which method is used, it is important to use pilot samples from each group during the initial stages of deployment in addition to testing the impact of this solution in a test environment if at all possible. The deployment of IPsec should follow a formal change control request process that details rollback plans along with informed buy-in from technology and affected business process owners.

Deploying IPsec by Group

The method of deploying IPsec network isolation by group uses fully defined IPsec policies but controls implementation by using ACLs on the GPOs that deliver the policies. All IPsec policies will have all exemptions and secure subnets completely defined with all appropriate filter actions enabled prior to deployment.

When configuration is complete, administrators create GPOs for each IPsec policy and computer groups to manage those GPOs. When created, the appropriate IPsec policies are assigned to their corresponding GPO and the GPOs are linked to the appropriate objects in Active Directory. None of the policies should be delivered at this point in the process, because the ACLs that control GPO assignment are empty.

When pilot computers are identified, the corresponding computer accounts can be placed in the appropriate computer groups and the IPsec policy will be applied after the next GPO polling interval has passed. As more computers are added to the deployment list in phases, their corresponding accounts can be placed in the appropriate groups for policy delivery.

This method requires some careful planning to ensure that communications are not disrupted. For example, if a server that hosted an application used by all hosts was placed in a secured group that required IPsec–encrypted communication, then the computers that were not yet added to groups and any host that couldn’t use IPsec encryption would be disrupted.

Deploying IPsec by Policy

This method approaches deployment by building the IPsec policies from scratch during deployment instead of prior to the actual production deployment. When using this method, initial IPsec policies only include exception lists with no rules set for computers to negotiate security. After the initial policy is delivered, administrators can create a security rule with a filter that only affects a single subnet. As the deployment progresses, additional subnets can be added to the secure rule until the policy reaches the end deployment state.

The advantages to using this phased deployment method are that IPsec is negotiated only for a small percentage of the total TCP/IP traffic instead of for all subnets, as is the case with a group-by-group approach. Also, this method allows for testing all network paths from a single subnet, thereby reducing the number of affected clients if any problems occur. However, the problem with this approach is that it is applied to all computers in the isolation domain or group, not to specific computers as with the group-by-group approach. Also, all computers will have to contend with the three-second delay for fall back to clear at some point when initiating connections with specified subnets.

This approach works well for complex networks with multiple subnets, but has very limited appeal for smaller networks with relatively few subnets.

Exemption Lists

Even the best designed network isolation model will encounter some constraints when it is implemented in a production environment. Typically, key components of a network environment like DNS servers and domain controllers present some challenges that will need to be addressed. Such systems need to be secured but also need to be available to all systems on a network. Therefore they must be open to inbound connection requests. Other problems can also become manifest when translating the plan into production, but there is a method to resolve such issues: exemption lists.

An exemption list is a list of computers that cannot be secured with IPsec and will need to be implemented in every designed IPsec policy. Because IPsec only supports static filtering, any host added to an exemption list will allow both outbound and inbound traffic from all parts of the network, trusted or not. For this reason the exemption list should be kept as small as possible, because these hosts will need additional protection through server hardening and host-based stateful firewall filtering to mitigate the risks they face from unmanaged devices. It might also be useful to consider placing exemption list hosts on a single subnet or VLAN for easier management, or consolidating server functions to reduce the number of exempted hosts.

Some of the hosts that may need to be added to an exemption list include:

  • Domain controllers do not support IPsec communication with domain members even though they do provide the IPsec policies to domain members and perform the Kerberos authentication that network isolation is based on.

  • Hosts that may suffer unacceptable performance impact, which may occur on hosts that have to manage thousands of clients simultaneously as well as hosts that are impacted by the three-second fall back to clear delay, such as DHCP servers.

  • Hosts that do not have a compatible IPsec implementation, or that will provide services to trusted and untrusted computers, but will not use IPsec.

As with the boundary group, there should be a formal approval process for hosts that require addition to the exemption lists.

Note   There is a “Simple Policy” fix for Microsoft Windows XP and Windows Server 2003 that makes exemption lists unnecessary. For more information, see Knowledge Base article 914841 at and Knowledge Base article 914842 at

IPsec Management Considerations

Once implemented, IPsec policies affect the communication between many hosts on the network. Therefore, changes that occur after deployment can have a profound effect on many users. For this reason, all changes made to an IPsec–based isolation network should follow a standardized MOF change management process that includes the following:

  • Change Request (RFC)

  • Change Classification

  • Change Authorization

  • Change Development Plan

  • Change Release Plan

  • Change Review

Microsoft recommends that Windows Server 2003 be used as the IPsec management platform. To facilitate management tasks, administrators can either use the IP Security Policy MMC snap-in or the Netsh command-line tool at the Windows Server 2003 console. Otherwise, managing IPsec policies is a fairly straightforward endeavor because they are delivered via Group Policy. Therefore, adding new computers to the trusted network can be automated if an established "computer add" process is used.

However, some things will have to be accounted for when changes are being considered, including:

  • Any changes in a filter action that was being used for an IPsec SA will cause established SAs under those policy settings to be deleted. This functionality creates a new quick mode attempt if any traffic is en route, which can result in dropped network traffic.

  • Changing the authentication method or main mode security method will cause IKE to delete existing main modes. In some circumstances, this functionality can cause server side IKE main mode negotiations with clients to fail.

  • When IPsec policies in a GPO are changed, certain delays (including Active Directory replication delay and the GPO polling delay) will occur that can range from minutes to hours, depending on the size and complexity of the environment.

Another management consideration deals with how to isolate suspect computers when infections occur. There are a number of different ways to deal with malicious activity, including:

  • Isolating the isolation domain. By modifying the IPsec – Secure Request Mode filter to disable unsecured communication, the isolated network can be completely cut off from the untrusted network when an untrusted device is suspected of being a platform for malicious activity.

  • Blocking suspect ports. By creating filter lists with two one-way filters, it is possible to block traffic by ports. This approach should be used with caution, because it can block necessary traffic, such as DNS, that would make further IPsec changes or rollbacks difficult.

  • Isolation to child domains. By changing the policies for a domain to use a preshared key rather than the Kerberos protocol for IPsec authentication, it is possible to isolate a domain from other domains in a forest.

  • Isolation to predefined groups. By using preshared keys or certificates for IPsec authentication instead of the Kerberos protocol, it is possible to create isolation groups that use different keys or certificates that act to perform the same type of isolation functionality.

Using VPN Quarantine Services to Protect Against Unmanaged Remote Computers

The actual deployment of VPN quarantine control basically involves six steps beyond any other change management request process and pre-deployment test processes that may exist in a given organization. These six steps include:

  1. Create quarantine resources.

  2. Create scripts to validate client configurations.

  3. Install the listener component Rqs.exe on remote access servers.

  4. Create quarantine Connection Manager (CM) profiles with Windows Server 2003 CMAK.

  5. Distribute the CM profiles to remote access client computers.

  6. Configure the quarantine remote access policy.

1. Create Quarantine Resources

If quarantined clients are going to use internal resources to bring themselves into a state that complies with established security policies, there will need to be a set of accessible resources that they can use to install the requisite updates and other software needed to make them secure. As detailed in the previous section, these resources usually include a name server, file servers, and sometimes Web servers as well. A couple of approaches can be used when designating these quarantine resources, including designating existing servers that reside on the internal network and placing the required resources in their own separate subnet.

The first approach of designating individual servers may reduce the costs associated with adding such servers to deliver updates, because existing servers will be used, However, this approach requires complex packet filters because each resource will need its own set of filters in the MS-Quarantine-IPFilter attribute of the quarantine remote access policy.

Placing all quarantine resources in a single subnet devoted to quarantine services may involve adding more servers to the network environment, but it only requires a single input and output packet filter for all quarantine resources.

2. Create Validation Scripts

Quarantine scripts perform a set of tests that ensure remote access computers comply with security policies. When these tests are complete, the script will need to either launch the RQC.exe client component (if successful) or direct the client computer to appropriate resources to bring it into compliance. Of course, the script could also simply enforce a network timeout along with a failure message if network policies dictate that approach instead. These scripts can be as simple as a command line batch file or complex executables.

Some sample scripts are available for review and download from the* * VPN Quarantine Sample Scripts page on the Microsoft Download Center at\&displaylang=en.

3. Install RQS.exe on Remote Access Servers

The process of installing the Remote Access Quarantine Agent service (Rqs.exe) on a Microsoft Windows Server 2003 computer is different for servers that have Service Pack 1 (SP1). For the sake of brevity, this section only describes installation on Windows Server 2003 with SP1, because it is assumed that all systems have all recent service packs and security updates.

To install Rqs.exe on a Windows Server 2003 with SP1 server

  1. Click Start and then click Control Panel.

  2. Click Add or Remove Programs.

  3. Click Add/Remove Windows Components.

  4. Select Components and then click Network Services.

  5. Click Details.

  6. Select Subcomponents of Networking Services, click Remote Access Quarantine Service, and then click OK.

This process installs but does not start the quarantine service, which must be started manually by an administrator after the solution is completely configured and the environment is prepared for implementation.

An additional step in the installation process includes modifying the script version strings so that they match the version string configured for the Rqc.exe in the quarantine script. Rqs.exe can be configured to accept multiple script version strings which are, in turn, written to the registry of the remote access server under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\rqs\AllowedSet, which is where these settings need to be applied via a REG_MULTI_SZ value type. Typically, the Allowed Set value is set to RASQuarantineConfigPassed, but additional string values can be added.

4. Create Quarantine CM Profiles

A quarantine CM profile is actually just a typical remote access CM profile that has the following additions:

  • A post-connect action that runs the script which has been created to check network policy compliance and the actual script itself. This is done through the Custom Actions page of the CMAK Wizard.

  • A notification component must also be added to the profile as done through the Additional Files screen of the CMAK Wizard.

5. Distribute CM Profiles

The newly created quarantine CM profiles must be distributed and installed on all remote access client computers. A profile comes in the form of an executable file that has to be run on each remote access client computer for it to install the profile and configure the quarantine network connection.

The distribution of these profiles can be a manual process, one in which they are sent as attachments to users along with instructions, for example. Or distribution could be automated with tools such as SMS 2003 software distribution. There are a number of methods that could be used, and the best approach for each environment is different.

6. Configure a Quarantine Remote Access Policy

The procedure for configuring a quarantine remote access policy differs depending on whether or not it is configured on an IAS server or another authentication provider as follows:

  • If configured on an IAS RADIUS server, then use the Internet Authentication Service snap-in.

  • If configured on a Windows authentication provider, then use the Routing and Remote Access snap-in.

When creating a policy, a normal mode remote access policy should be made first. Then MS-Quarantine-Session-Timeout and MS-Quarantine-IPFilter attributes should be added via the Advanced tab of the remote access policy profile properties.

If RADIUS is being used, the quarantine remote access policy will need to be replicated on the other RADIUS servers either by copying the configuration or manually creating it on each server.

Using 802.1X Authentication to Protect Against Unmanaged Wireless Clients

As described in the previous sections, the most to least secure options that are recommended by Microsoft for securing a midsized business wireless network are:

  • WPA2/AES and EAP-TLS

  • WPA2/AES and PEAP-MS-CHAP v2



Although EAP-TLS or PEAP-MS-CHAP v2 can make WEP more secure, using WEP as a part of either solution is only recommended for use during a transition to the more secure WPA or WPA2 standards.

The processes of implementing EAP-TLS and PEAP-MS-CHAP v2 share some similarities. Both require the use of Microsoft IAS RADIUS servers as part of the solution and both require certificates in some form, but EAP-TLS requires certificates on both the client computer and the server while PEAP-MS-CHAP v2 only needs certificates for the authentication servers. This is the reason why a certificate infrastructure is highly recommended when implementing EAP-TLS, because the cost of purchasing certificates from third-party providers for all clients may be prohibitive.

Wireless Authentication with EAP-TLS and PEAP-MS-CHAP v2

The implementation processes for EAP-TLS and PEAP-MS-CHAP v2 involve a number of details which are beyond the scope of this document. However, several excellent resources already exist to help design and implement solutions to secure wireless networks, such as:

Using SMS to Detect and Remediate Unmanaged Systems

As established in the "Development" section, the SMS discovery process is very useful for finding newly connected computers on the network when they are manageable by SMS. Also mentioned was the fact that SMS has difficulty discovering some unmanageable systems during the discovery process, and so another mechanism is needed to fill this gap.

There can be a few reasons why SMS might not discover computers attached to a network. These reasons include:

  • The computer is attached but not running at the time of discovery and does not have a computer account on the domain.

  • The computer is attached to a subnet where a firewall or other network device prevents communication between the SMS server and that subnet.

  • The computer is attached to an accessible network but is using an operating system that SMS cannot discover.

To deal with such eventualities, a custom solution is required that combines the usefulness of SMS with scripts that cover the exceptions to what SMS can manage. Custom scripts can be used to scan the network at regular intervals with a scheduler process, which means they can discover devices that may only be on for brief periods of time. Such scripts can also be run from workstations or servers placed on otherwise isolated subnets, which means they can detect when unmanaged systems connect to these otherwise unmonitored network segments. Such scripts don’t attempt WMI–based discovery processes, and are therefore unaffected by what operating system might be in use on unmanaged clients.

For more information about how to use SMS to discover devices on the network, refer to the Patch Management Using Systems Management Server 2003 solution accelerator at or the Microsoft Systems Management Server home page at for more links, including links to resources on SMS scripting solutions.


This document has shown that it is possible to use a comprehensive approach to help manage the concerns associated with unmanaged systems. IPsec network isolation was used to help prevent unmanaged systems from gaining access to trusted network resources, and to assist with compliance enforcement through denial of service to noncompliant computers. VPN Quarantine services can be used to help enforce compliance on mobile computers that use a remote access solution but rarely connect to the local network. IEEE 802.1X and WPA/WPA2 can be used to help defend against rogue devices on wireless LANs. Finally, SMS and detection scripts can be used to assist with the detection of unmanaged systems as well as helping to keep managed computers up to date with all patches and security updates.

Although these technical solutions help resolve many issues associated with unmanaged and rogue devices, these solutions are only improved when a sound security policies are created and strict change management processes are established as well. When all of these solutions, policies, and processes are combined, the result is a manageable midsize business infrastructure that can lower administrative costs and reduce security risks to acceptable levels.

Appendix A: Network Access Protection

Network Access Protection (NAP) is a new feature that will be a part of the next generation of Windows (Microsoft Windows Server Longhorn and Windows Vista™) that will provide the capability to help enforce compliance with computer health policies. With NAP, administrators can set health validation policy thresholds through an application programming interface (API) that limit network access for computers that do not meet health requirements when they connect to the network.

The flexibility of NAP allows administrators to configure custom solutions for their environment’s requirements, whether just logging the health state of computers attaching to the network, automatically deploying software updates, or limiting the connectivity of systems that do not meet health requirements. In addition, NAP is flexible enough to allow incorporation with third-party applications and custom program solutions so that enforcement of computer health policies will be comprehensive. NAP will also be comprehensive; its enforcement components will enable administrators to force compliance to health policies via DHCP, VPN, and 802.1X wireless networks, and it is compatible with IPsec.

For more information about NAP, refer to the Network Access Protection home page at


Get the Protecting a Network from Unmanaged Clients paper