Troubleshooting NIS detection issues

  • Article

Updated: February 1, 2011

Applies To: Forefront Threat Management Gateway (TMG)

This topic contains troubleshooting information for NIS detection issues. For a flowchart showing how to determine that this is a detection issue, and not an issue that is caused by a failure to update NIS signatures, see NIS troubleshooting flow.

Issues in this topic include:

  • An exploit is not detected

  • An exploit is detected but is not blocked

  • NIS blocks legitimate traffic due to incorrect detection

  • NIS blocks legitimate traffic due to incorrect detection of protocol anomaly

An exploit is not detected

An exploit is not detected by NIS, and it infiltrates the internal network. The following sections describe the possible causes for the lack of detection of an apparent exploit, and the corresponding resolutions:

  • File-based exploits are not detected

  • Signature configuration settings prevent detection

  • NIS exception prevents detection

  • Signature set version is not up-to-date

  • Non-standard protocols are not inspected

File-based exploits are not detected

NIS does not provide protection against file-based attacks.

Resolution

Protection against file-based attacks is provided by Forefront TMG e-mail protection. If you are investigating a case where an exploit file penetrated Forefront TMG, make sure that e-mail protection is enabled and updated. For information, see Planning to protect against e-mail threats.

Signature configuration settings prevent detection

NIS configuration settings of the attack’s signature are set to disable inspection of the signature, or to only detect the signature, and not to block it.

Resolution

Change the configuration settings of the signature to block the attack.

To configure a signature to block attacks

  1. In the Forefront TMG Management console, in the tree, click the Intrusion Prevention System node.

  2. On the details pane of the Network Inspection System (NIS) tab, click the signature that you want to modify, and then, on the Tasks tab, click Configure Signature Properties.

  3. On the General tab, click Override, click Enable, and then select Block from the list.

    Note

    To learn more about this signature, click More information about this signature online.

  4. Click OK, and then, on the Apply Changes bar, click Apply.

NIS exception prevents detection

The attack’s signature is defined as a NIS exception, and is therefore excluded from inspection.

Resolution

Remove the signature from the NIS exception list.

To remove a signature from the exception list:

  1. In the Forefront TMG Management console, in the tree, click the Intrusion Prevention System node.

  2. Click the Network Inspection System (NIS) tab, and then, on the Tasks tab, click Define Exceptions.

  3. On the Exceptions tab, select the signature, and then click Remove.

  4. Click OK, and then, on the Apply Changes bar, click Apply.

Signature set version is not up-to-date

Forefront TMG is not updated with the latest signature set.

Resolution

Update Forefront TMG with the latest signature set. For information, see Troubleshooting NIS signature update failure.

Non-standard protocols are not inspected

Only standard protocols, which are predefined by Forefront TMG, are inspected by NIS; non-standard, user-defined protocols are not inspected.

Resolution

Associate the non-standard protocol with a standard protocol.

Note

In order to associate a protocol with an existing protocol, the following conditions must be met:

  1. The custom protocol behavior must match the standard protocol behavior. For example, if your application uses POP3 over port 3389, and you associate this protocol with the standard RDP protocol because it also uses port 3389, NIS blocks the custom POP3 traffic.

  2. The secondary connections of the non-standard protocol must be identical to, or a subset of, the secondary connections of the standard protocol.

  3. The same application filters must be selected for the standard protocol and for the user-defined protocol. For example, because the standard HTTP protocol is handled by the Web Proxy filter, you can only select the HTTP protocol from the list of protocols when associating a non-standard protocol that is configured to use that filter.

You select both the protocol’s secondary connections and the application filters in the Parameters tab of the <protocol>Properties dialog box.

To associate a non-standard protocol with a standard protocol

  1. In the Forefront TMG Management console, in the tree, click the Firewall Policy node.

  2. On the Toolbar tab, click Network Objects, click Protocols, and then expand the User-Defined list.

  3. Right-click the non-standard protocol, and then select Properties.

  4. On the General tab, click the Associate this protocol definition with this standard protocol option, and then, in the list, click a standard protocol.

An exploit is detected but is not blocked

An exploit is detected by NIS but it is not blocked; as a result, the exploit infiltrates the internal network. The alert NIS detected traffic is displayed in the Forefront TMG Management console, on the Alerts tab of the Monitoring node.

Cause

Forefront TMG is configured to detect this signature, and not to block it.

Resolution

  1. In the Forefront TMG Management console, in the tree, click the Intrusion Prevention node.

  2. On the Network Inspection System (NIS) tab, double-click the signature you want to configure. In the General tab of the Signature Information Properties dialog box, under Set the response policy for this signature, click Override, and then, under Response, select Block.

NIS blocks legitimate traffic due to incorrect detection

Legitimate traffic is identified as malicious and is blocked by NIS. The alert NIS blocked traffic is displayed in the Forefront TMG Management console, on the Alerts tab of the Monitoring node.

Cause

Incorrect categorization of the signature by the Microsoft Malware Protection Center.

Resolution

  1. In the Forefront TMG Management console, in the tree, click the Intrusion Prevention node.

  2. On the Network Inspection System (NIS) tab, double-click the signature you want to configure. In the General tab of the Signature Information Properties dialog box, under Set the response policy for this signature, click Override, and then, under Response, select Detect.

  3. Report the issue to Forefront TMG Customer Support. For Customer Support options, see Microsoft Support (https://go.microsoft.com/fwlink/?LinkId=136567).

NIS blocks legitimate traffic due to incorrect detection of protocol anomaly

Legitimate traffic is identified as malicious and is blocked by NIS. The alert NIS blocked traffic because it detected a protocol anomaly is displayed in the Forefront TMG Management console, on the Alerts tab of the Monitoring node.

Cause

An application was not implemented according to formal protocols specifications.

Resolution

  1. In the Forefront TMG Management console, in the tree, click the Intrusion Prevention node.

  2. On the Network Inspection System (NIS) tab, double-click the signature you want to configure, and then, in the Protocol Anomalies Policy tab of the Signature Information Properties dialog box, under Response to protocol anomalies, click Allow, to avoid blocking legitimate traffic.

Concepts

Troubleshooting NIS