Share via

Connection Security and IPsec

  • Article

Updated: December 7, 2009

Applies To: Windows 7, Windows Server 2008, Windows Server 2008 R2, Windows Vista

Internet Protocol Security (IPsec) is a framework of open standards for protecting communications over TCP/IP networks by using cryptographic security services. IPsec supports network-level peer authentication, data origin authentication, data integrity, data confidentiality (encryption), and replay protection. The Microsoft implementation of IPsec is based on standards developed by the Internet Engineering Task Force (IETF) IPsec working group.

The implementation of IPsec included with Windows Vista and later versions of Windows is fully integrated into the Network layer (layer 3) of the Open Systems Interconnection (OSI) network reference model. This lets it provide protection to any IP-based protocol in a manner that is transparent to the programs that are running on the computer. It fully supports both IPv4 and IPv6.

IPsec is an important layer in a defense-in-depth strategy to protect your organization's network-accessible resources.

How IPsec works

IPsec provides a variety of connection security services to network traffic. You can configure each service to apply to specific network traffic by creating a connection security rule in Windows Firewall with Advanced Security that identifies the characteristics of the network traffic to protect, and the nature of the protection to be applied.

  • Source authentication. Source authentication ensures that each computer participating in a connection receives proof that the remote computer (and optionally the user on the remote computer) is truly the entity that it claims to be.

    Authentication involves each computer providing some form of credentials to the other computer that can be proved to be from the claimed source. Kerberos tokens, which can be checked with a domain controller, or a computer or user certificate which can be cryptographically checked against its trusted root certificate, are among the authentication methods generally used.

  • Data integrity. Data integrity ensures that the packet that is received is identical to the packet that was transmitted, and provides an assurance that it was not damaged or modified in transit.

    A network packet that is sent through an IPsec protected connection includes a cryptographic hash of the packet. The hash is calculated by the sending computer, encrypted, and included in the packet. The receiving computer calculates its own hash on the received packet, and after decrypting the included hash, compares the two hash values. If they match, the packet is accepted and processed. If they do not match, then the packet was damaged or modified in transit, and is dropped.

  • Data confidentiality. Data confidentiality ensures that the information included in the network connection cannot be accessed or read by non-authorized computers or users.

    When enabled, every network packet sent through the protected connection has its data payload encrypted. Various strength encryption protocols are available for use. Protocols that provide a greater level of protection generally require more computer resources to process the more complex cryptographic algorithms.

Transport versus Tunnel Mode

IPsec operates in one of two modes: transport or tunnel mode.

Transport mode. In transport mode, the network traffic is IPsec-protected by the originating computer and it stays protected all of the way through the network to the destination computer. The IPsec-protected packets are routed to their destination as standard IP datagrams by the routers along the way. Transport mode provides end-to-end security. A transport mode connection security rule requires only two IP addresses: the source and destination computers. The domain and server isolation scenarios discussed in this guide use transport mode IPsec rules.

The following diagram illustrates transport mode, where each computer establishes an iPsec protected connection from itself to a remote server.

Tunnel mode. In tunnel mode, the network traffic is IPsec-protected only for a part of the trip between the origin and destination computers, typically as it traverses an untrusted network. For example, an organization with two geographically separated private intranets that are separated by Internet, can use an IPsec tunnel mode connection to treat the two separate intranets as one logical network. To do this, a computer on each intranet is designated as the IPsec gateway, or tunnel endpoint. Clients and servers in each intranet treat their local gateway as a router. But instead of routing packets across the Internet like a typical router, the gateway creates an IPsec-protected connection, or tunnel, and then sends the packets through the tunnel. Traffic is sent plain text between the client and the local gateway, then sent IPsec-protected from the local gateway to the remote gateway, and then finally sent plain text again between the remote gateway and the destination computer. Unlike a transport mode rule, a tunnel mode rule requires four IP addresses: the two tunnel endpoints, or gateway computers, and the two sets of computers that are accessible to each other through the tunnel, referred to as endpoints. The endpoints are typically sets of subnet addresses.

The following diagram illustrates tunnel mode, where computers on each intranet can communicate with computers on the other intranet by routing packets through the IPsec tunnel between the two gateways.

Tunnel mode can also be used in a scenario where the originating computer is the local gateway. This is a typical VPN scenario, where the client computer is attached to an untrusted network, but communicates securely with computers on a private intranet through an IPsec tunnel established with a remote gateway. In this case, one of the endpoints and its corresponding tunnel endpoint are one and the same.

The following diagram illustrates a tunnel mode connection between a remote client and an IPsec gateway to a private network.

You can also combine the tunnel and transport mode rules, to have a client establish a tunnel mode connection to a remote site, and transport mode end-to-end protected connections to servers in that site. The transport mode traffic is sent through the tunnel as is all other traffic between the client and the site.

For more information about IPsec, see:

For more information about the IPsec functionality in Windows Firewall with Advanced Security and its use in supporting serer and domain isolation, see:

Next topic: Group Policy