Share via

Verify That Network Traffic Is Authenticated

  • Article

Applies To: Windows 7, Windows Essential Business Server, Windows SBS 2003, Windows SBS 2008, Windows Server 2000, Windows Server 2003, Windows Server 2003 R2, Windows Server 2003 with SP1, Windows Server 2003 with SP2, Windows Server 2008, Windows Server 2008 R2, Windows Vista

After you have configured your domain isolation rule to request, rather than require, authentication, you must confirm that the network traffic sent by the computers on the network is being protected by IPsec authentication as expected. If you switch your rules to require authentication before all of the computers have received and applied the correct GPOs, or if there are any errors in your rules, then communications on the network can fail. By first setting the rules to request authentication, any network connections that fail authentication can continue in clear text while you diagnose and troubleshoot.

In these procedures, you confirm that the rules you deployed are working correctly. Your next steps depend on which zone you are working on:

  • Main domain isolation zone. Before you convert your main domain isolation IPsec rule from request mode to require mode, you must make sure that the network traffic is protected according to your design. By configuring your rules to request and not require authentication at the beginning of operations, computers on the network can continue to communicate even when the main mode authentication or quick mode integrity and encryption rules are not working correctly. For example, if your encryption zone contains rules that require a certain encryption algorithm, but that algorithm is not included in a security method combination on the clients, then those clients cannot successfully negotiate a quick mode security association, and the server refuses to accept network traffic from the client. By first using request mode only, you have the opportunity to deploy your rules and then examine the network traffic to see if they are working as expected without risking a loss of communications.

  • Boundary zone. Confirming correct operation of IPsec is the last step if you are working on the boundary zone GPO. You do not convert the GPO to require mode at any time.

  • Encryption zone. Similar to the main isolation zone, after you confirm that the network traffic to zone members is properly authenticated and encrypted, you must convert your zone rules from request mode to require mode.

Note

In addition to the steps shown in this procedure, you can also use network traffic capture tools such as Microsoft Network Monitor, which can be downloaded from https://go.microsoft.com/fwlink/?linkid=94770. Network Monitor and similar tools allow you to capture, parse, and display the network packets received by the network adapter on your computer. Current versions of these tools include full support for IPsec. They can identify encrypted network packets, but they cannot decrypt them.

Administrative credentials

To complete these procedures, you must be a member of the Domain Administrators group, or otherwise be delegated permissions to modify the GPOs.

In this topic:

  • Procedure for computers running Windows Vista or Windows Server 2008

  • Procedure for computers running earlier versions of Windows

For computers running Windows 7, Windows Vista, Windows Server 2008, or Windows Server 2008 R2

To verify that network connections are authenticated by using the Windows Firewall with Advanced Security MMC snap-in

  1. Click Start, in the Start Search box, type wf.msc, and then press ENTER.

  2. If the User Account Control dialog box appears, confirm that the action it displays is what you want, and then click Continue.

    Windows Firewall with Advanced Security opens.

  3. In the navigation pane, expand Monitoring, and then click Connection Security Rules.

    The details pane displays the rules currently in effect on the computer.

  4. To display the Rule Source column

    1. In the Actions pane, click View, and then click Add/Remove Columns.

    2. In the Available columns list, select Rule Source, and then click Add.

    3. Use the Move up and Move down buttons to rearrange the order. Click OK when you are finished.

      It can take a few moments for the list to be refreshed with the newly added column.

  5. Examine the list for the rules from GPOs that you expect to be applied to this computer.

Note

If the rules do not appear in the list, then troubleshoot the GPO security group and the WMI filters that are applied to the GPO. Make sure that the local computer is a member of the appropriate groups and meets the requirements of the WMI filters.

  1. In the navigation pane, expand Security Associations, and then click Main Mode.

    The current list of main mode associations that have been negotiated with other computers appears in the details column.

  2. Examine the list of main mode security associations for sessions between the local computer and the remote computer. Make sure that the 1st Authentication Method and 2nd Authentication Method columns contain expected values. If your rules specify only a first authentication method, then the 2nd Authentication Method column displays No authentication. If you double-click the row, then the Properties dialog box appears with additional details about the security association.

  3. In the navigation pane, click Quick mode.

  4. Examine the list of quick mode security associations for sessions between the local computer and the remote computer. Make sure that the AH Integrity, ESP integrity, and ESP Confidentiality columns contain expected values.

To verify that network connections are authenticated by using the Netsh command-line tool

  1. Start a command line as an administrator.

  2. Type the command netsh advfirewall monitor show mmsa and then press ENTER.

  3. Examine the output for an entry that includes your local IP address and the IP address of a remote computer that you want to verify. The First Auth and Second Auth lines indicate the authentication method used to establish the SA between these two endpoints.

For computers running earlier versions of Windows

To confirm that network connections are authenticated by using the IP Security Monitor MMC snap-in

  1. Open the IP Security Monitor MMC console. To do so, click Start, click Run, then type mmc and then click OK. In the new console, click File, Add/Remove Snap-in, click Add, select IP Security Monitor, and then click Add. Click Close, and then click OK.

  2. Expand IP Security Monitor, expand Computer Name, and then click Active Policy.

  3. In the details pane, examine the IP Security Policy details. If an unexpected or incorrect policy appears here, then you must investigate the following:

    • Which GPOs were applied to this computer? Did it receive the correct or unexpected GPOs? If unexpected GPOs were applied, then investigate the security group and WMI filters used to control the application of your GPOs.

    • Does more than one GPO have an assigned IP security policy? Windows supports only one assigned IP security policy at a time. You should make sure that only one GPO has an assigned IP security policy. If your design calls for multiple policies, then you must investigate the precedence of the GPOs and the order in which they are applied to determine which GPO’s policy was assigned.

  4. Connect from the remote computer to the local computer using network traffic that is expected to be protected by IPsec.

  5. In the navigation pane, expand Main Mode, and then click Security Associations.

  6. In the details pane, examine the main mode security associations. Look for connections between the local computer (IP address shown in the Me column) and the remote computer (IP address shown in the Peer column). The Authentication, Encryption, Integrity, and Diffie-Hellman columns display details about the security association negotiated by the two computers. The main mode security association is the communications channel used to negotiate the quick mode security associations.

  7. In the navigation pane, expand Quick Mode, and then click Security Associations.

  8. In the details pane, examine the quick mode security associations negotiated between the local computer (Me) and the remote computer (Peer). The Protocol, My Port, and Peer Port columns describe any other filters on the traffic to which this security association applies. The AH integrity column displays the algorithm used if the AH protocol is protecting the data. The ESP Confidentiality column displays the encryption algorithm used if the data is encrypted. The ESP Integrity column displays the integrity algorithm used if ESP is protecting the data.

If you arrived at this page by clicking a link in a checklist, use your browser’s Back button to return to the checklist.