IPsec Configuration

Windows Filtering Platform (WFP) is the underlying platform for Windows Firewall with Advanced Security. WFP is used to configure network filtering rules, which include rules that govern securing network traffic with IPsec. Application developers may configure IPsec directly using the WFP API, in order to take advantage of a more granular network traffic filtering model than the model exposed through the Microsoft Management Console (MMC) snap-in for Windows Firewall with Advanced Security.

What is IPsec

Internet Protocol Security (IPsec) is a set of security protocols used to transfer IP packets confidentially across the Internet. IPsec was formerly mandatory for all IPv6 implementations (but see IPv6 Node Requirements; and optional for IPv4.

Secured IP traffic has two optional IPsec headers, which identify the types of cryptographic protection applied to the IP packet and include information for decoding the protected packet.

The Encapsulating Security Payload (ESP) header is used for privacy and protection against malicious modification by performing authentication and optional encryption. It can be used for traffic that traverses Network Address Translation (NAT) routers.

The Authentication Header (AH) is used only for protection against malicious modification by performing authentication. It cannot be used for traffic that traverses NAT routers.

For more information on IPsec, see also:

IPsec Technical Reference

What is IKE

Internet Key Exchange (IKE) is a key exchange protocol that is part of the IPsec protocol set. IKE is used while setting up a secure connection and accomplishes the safe exchange of secret keys and other protection-related parameters without the intervention of the user.

For more information on IKE, see also:

Internet Key Exchange

What is AuthIP

Authenticated Internet Protocol (AuthIP) is a new key exchange protocol that expands IKE as follows.

While IKE only supports computer authentication credentials, AuthIP also supports:
  • User credentials: NTLM, Kerberos, certificates.
  • Network Access Protection (NAP) health certificates.
  • Anonymous credential, used for optional authentication.
  • Combination of credentials; for example, a combination of machine and user Kerberos credentials.

AuthIP has an authentication-retry mechanism that verifies all configured authentication methods before failing the connection.
AuthIP can be used with secure sockets to implement application-based IPsec secured traffic. It provides:

  • Per-socket authentication and encryption. See WSASetSocketSecurity for more information.
  • Client impersonation. (IPsec impersonates the security context under which the socket is created.)
  • Inbound and outbound peer name validation. See WSASetSocketPeerTargetName for more information.

For more information on AuthIP, see also:

AuthIP in Windows Vista

What is an IPsec Policy

An IPsec policy is a set of rules that determine which type of IP traffic needs to be secured using IPsec and how to secure that traffic. Only one IPsec policy is active on a computer at one time.

To learn more about implementing IPsec policies, open the Local Security Policy MMC snap-in (secpol.msc), press F1 to display the Help, and then select Creating and Using IPsec Policies from the table of contents.

For more information on IPsec policies, see also:

Overview of IPsec Policy Concepts
Description of an IPsec Policy

How to Use WFP to Configure IPsec Policies

The Microsoft implementation of IPsec uses Windows Filtering Platform to setup IPsec policies. IPsec policies are implemented by adding filters at various WFP layers as follows.

  • At the FWPM_LAYER_IKEEXT_V{4|6} layers add filters that specify the negotiation policies used by the keying modules (IKE/AuthIP) during Main Mode (MM) exchanges. Authentication methods and cryptographic algorithms are specified at these layers.

  • At the FWPM_LAYER_IPSEC_V{4|6} layers add filters that specify the negotiation policies used by the keying modules during Quick Mode (QM) and Extended Mode (EM) exchanges. IPsec headers (AH/ESP) and cryptographic algorithms are specified at these layers.

    A negotiation policy is specified as a policy provider context associated with the filter. The keying module enumerates the policy provider contexts based on the traffic characteristics and obtains the policy to use for the security negotiation.


    The WFP API can be used to specify the Security Associations (SAs) directly and therefore to ignore the keying module negotiation policy.


  • At the FWPM_LAYER_INBOUND_TRANSPORT_V{4|6} and FWPM_LAYER_OUTBOUND_TRANSPORT_V{4|6} layers add filters that invoke callouts and determine which traffic flow should be secured.

  • At the FWPM_LAYER_ALE_AUTH_RECV_ACCEPT_V{4|6} layers add filters that implement identity filtering and per-application policy.

The following diagram illustrates the interaction of the various WFP components, with respect to IPsec operation.ipsec configuration using windows filtering platform

Once IPsec is configured, it integrates with WFP and extends the WFP filtering capabilities by providing information to be used as filtering conditions at the Application Layer Enforcement (ALE) authorization layers. For example, IPsec provides the remote user and remote machine identity, which WFP exposes at the ALE connect and accept authorization layers. This information can be used for fine-grained remote identity authorization by a WFP-based firewall implementation.

Below is a sample isolation policy that may be implemented using IPsec:

  • FWPM_LAYER_IKEEXT_V{4|6} layers – Kerberos authentication.
  • FWPM_LAYER_IPSEC_V{4|6} layers – AH/SHA-1.
  • FWPM_LAYER_INBOUND_TRANSPORT_V{4|6} and FWPM_LAYER_OUTBOUND_TRANSPORT_V{4|6} layers - negotiation discovery for all network traffic.
  • FWPM_LAYER_ALE_AUTH_RECV_ACCEPT_V{4|6} layers - IPsec required for all network traffic.

WFP Layers

Filtering Layer Identifiers

ALE Layers

IPsec Policy Scenarios Implemented using WFP API:

Transport Mode

Negotiation Discovery Transport Mode

Negotiation Discovery Transport Mode in Boundary Mode

Tunnel Mode

Guaranteed Encryption

Remote Identity Authorization

Manual IPsec SAs

IKE/AuthIP Exemptions

IPsec Solutions:

Server and Domain Isolation