Edit

Troubleshoot your CEF or Syslog data connector

  • Article

Caution

This article references CentOS, a Linux distribution that is nearing End Of Life (EOL) status. Please consider your use and planning accordingly. For more information, see the CentOS End Of Life guidance.

This article describes common methods for verifying and troubleshooting a CEF or Syslog data connector for Microsoft Sentinel.

For example, if your logs are not appearing in Microsoft Sentinel, either in the Syslog or the Common Security Log tables, your data source may be failing to connect or there may be another reason your data is not being ingested.

Other symptoms of a failed connector deployment include when either the security_events.conf or the security-omsagent.config.conf files are missing, or if the rsyslog server is not listening on port 514.

For more information, see Connect your external solution using Common Event Format and Collect data from Linux-based sources using Syslog.

If you've deployed your connector using a method different than the documented procedure and are having issues, we recommend that you purge the deployment and install again as documented.

This article shows you how to troubleshoot CEF or Syslog connectors with the Log Analytics agent. For troubleshooting information related to ingesting CEF logs via the Azure Monitor Agent (AMA), review the Common Event Format (CEF) via AMA connector instructions.

Important

On February 28th 2023, we introduced changes to the CommonSecurityLog table schema. Following this change, you might need to review and update custom queries. For more details, see the recommended actions section in this blog post. Out-of-the-box content (detections, hunting queries, workbooks, parsers, etc.) has been updated by Microsoft Sentinel.

How to use this article

When information in this article is relevant only for Syslog or only for CEF connectors, we've organized the page into tabs. Make sure that you're using the instructions on the correct tab for your connector type.

For example, if you're troubleshooting a CEF connector, start with Validate CEF connectivity. If you're troubleshooting a Syslog connector, start below, with Verify your data connector prerequisites.

Validate CEF connectivity

After you've deployed your log forwarder and configured your security solution to send it CEF messages, use the steps in this section to verify connectivity between your security solution and Microsoft Sentinel.

This procedure is relevant only for CEF connections, and is not relevant for Syslog connections.

  1. Make sure that you have the following prerequisites:

    • You must have elevated permissions (sudo) on your log forwarder machine.

    • You must have python 2.7 or 3 installed on your log forwarder machine. Use the python --version command to check.

    • You may need the Workspace ID and Workspace Primary Key at some point in this process. You can find them in the workspace resource, under Agents management.

  2. From the Microsoft Sentinel navigation menu, open Logs. Run a query using the CommonSecurityLog schema to see if you are receiving logs from your security solution.

    It may take about 20 minutes until your logs start to appear in Log Analytics.

  3. If you don't see any results from the query, verify that events are being generated from your security solution, or try generating some, and verify they are being forwarded to the Syslog forwarder machine you designated.

  4. Run the following script on the log forwarder (applying the Workspace ID in place of the placeholder) to check connectivity between your security solution, the log forwarder, and Microsoft Sentinel. This script checks that the daemon is listening on the correct ports, that the forwarding is properly configured, and that nothing is blocking communication between the daemon and the Log Analytics agent. It also sends mock messages 'TestCommonEventFormat' to check end-to-end connectivity. 

    sudo wget -O cef_troubleshoot.py https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/CEF/cef_troubleshoot.py&&sudo python cef_troubleshoot.py [WorkspaceID]

CEF validation script explained

The following section describes the CEF validation script, for the rsyslog daemon and the syslog-ng daemon.

rsyslog daemon

For an rsyslog daemon, the CEF validation script runs the following checks:

  1. Checks that the file
    /etc/opt/microsoft/omsagent/[WorkspaceID]/conf/omsagent.d/security_events.conf
    exists and is valid.

  2. Checks that the file includes the following text:

    <source>
    type syslog
    port 25226
    bind 127.0.0.1
    protocol_type tcp
    tag oms.security
    format /(?<time>(?:\w+ +){2,3}(?:\d+:){2}\d+|\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.[\w\-\:\+]{3,12}):?\s*(?:(?<host>[^: ]+) ?:?)?\s*(?<ident>.*CEF.+?(?=0\|)|%ASA[0-9\-]{8,10})\s*:?(?<message>0\|.*|.*)/
    <parse>
        message_format auto
    </parse>
</source>

<filter oms.security.**>
    type filter_syslog_security
</filter>

  3. Checks that the parsing for Cisco ASA Firewall events is configured as expected, using the following command:

    grep -i "return ident if ident.include?('%ASA')" /opt/microsoft/omsagent/plugin/security_lib.rb

    • If there is an issue with the parsing, the script will produce an error message directing you to manually run the following command (applying the Workspace ID in place of the placeholder). The command will ensure the correct parsing and restart the agent.

      # Cisco ASA parsing fix
sed -i "s|return '%ASA' if ident.include?('%ASA')|return ident if ident.include?('%ASA')|g" /opt/microsoft/omsagent/plugin/security_lib.rb && sudo /opt/microsoft/omsagent/bin/service_control restart [workspaceID]

  4. Checks that the Computer field in the syslog source is properly mapped in the Log Analytics agent, using the following command:

    grep -i "'Host' => record\['host'\]"  /opt/microsoft/omsagent/plugin/filter_syslog_security.rb

    • If there is an issue with the mapping, the script will produce an error message directing you to manually run the following command (applying the Workspace ID in place of the placeholder). The command will ensure the correct mapping and restart the agent.

      # Computer field mapping fix
sed -i -e "/'Severity' => tags\[tags.size - 1\]/ a \ \t 'Host' => record['host']" -e "s/'Severity' => tags\[tags.size - 1\]/&,/" /opt/microsoft/omsagent/plugin/filter_syslog_security.rb && sudo /opt/microsoft/omsagent/bin/service_control restart [workspaceID]

  5. Checks if there are any security enhancements on the machine that might be blocking network traffic (such as a host firewall).

  6. Checks that the syslog daemon (rsyslog) is properly configured to send messages (that it identifies as CEF) to the Log Analytics agent on TCP port 25226:

    Configuration file: /etc/rsyslog.d/security-config-omsagent.conf

    if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then @@127.0.0.1:25226

  7. Restarts the syslog daemon and the Log Analytics agent:

    service rsyslog restart

/opt/microsoft/omsagent/bin/service_control restart [workspaceID]

  8. Checks that the necessary connections are established: tcp 514 for receiving data, tcp 25226 for internal communication between the syslog daemon and the Log Analytics agent:

    netstat -an | grep 514

netstat -an | grep 25226

  9. Checks that the syslog daemon is receiving data on port 514, and that the agent is receiving data on port 25226:

    sudo tcpdump -A -ni any port 514 -vv

sudo tcpdump -A -ni any port 25226 -vv

  10. Sends MOCK data to port 514 on localhost. This data should be observable in the Microsoft Sentinel workspace by running the following query:

    CommonSecurityLog
| where DeviceProduct == "MOCK"

syslog-ng daemon

For a syslog-ng daemon, the CEF validation script runs the following checks:

  1. Checks that the file
    /etc/opt/microsoft/omsagent/[WorkspaceID]/conf/omsagent.d/security_events.conf
    exists and is valid.

  2. Checks that the file includes the following text:

    <source>
    type syslog
    port 25226
    bind 127.0.0.1
    protocol_type tcp
    tag oms.security
    format /(?<time>(?:\w+ +){2,3}(?:\d+:){2}\d+|\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.[\w\-\:\+]{3,12}):?\s*(?:(?<host>[^: ]+) ?:?)?\s*(?<ident>.*CEF.+?(?=0\|)|%ASA[0-9\-]{8,10})\s*:?(?<message>0\|.*|.*)/
    <parse>
        message_format auto
    </parse>
</source>

<filter oms.security.**>
    type filter_syslog_security
</filter>

  3. Checks that the parsing for Cisco ASA Firewall events is configured as expected, using the following command:

    grep -i "return ident if ident.include?('%ASA')" /opt/microsoft/omsagent/plugin/security_lib.rb

    • If there is an issue with the parsing, the script will produce an error message directing you to manually run the following command (applying the Workspace ID in place of the placeholder). The command will ensure the correct parsing and restart the agent.

      # Cisco ASA parsing fix
sed -i "s|return '%ASA' if ident.include?('%ASA')|return ident if ident.include?('%ASA')|g" /opt/microsoft/omsagent/plugin/security_lib.rb && sudo /opt/microsoft/omsagent/bin/service_control restart [workspaceID]

  4. Checks that the Computer field in the syslog source is properly mapped in the Log Analytics agent, using the following command:

    grep -i "'Host' => record\['host'\]"  /opt/microsoft/omsagent/plugin/filter_syslog_security.rb

    • If there is an issue with the mapping, the script will produce an error message directing you to manually run the following command (applying the Workspace ID in place of the placeholder). The command will ensure the correct mapping and restart the agent.

      # Computer field mapping fix
sed -i -e "/'Severity' => tags\[tags.size - 1\]/ a \ \t 'Host' => record['host']" -e "s/'Severity' => tags\[tags.size - 1\]/&,/" /opt/microsoft/omsagent/plugin/filter_syslog_security.rb && sudo /opt/microsoft/omsagent/bin/service_control restart [workspaceID]

  5. Checks if there are any security enhancements on the machine that might be blocking network traffic (such as a host firewall).

  6. Checks that the syslog daemon (syslog-ng) is properly configured to send messages that it identifies as CEF (using a regex) to the Log Analytics agent on TCP port 25226:

    • Configuration file: /etc/syslog-ng/conf.d/security-config-omsagent.conf

      filter f_oms_filter {match(\"CEF\|ASA\" ) ;};destination oms_destination {tcp(\"127.0.0.1\" port(25226));};
log {source(s_src);filter(f_oms_filter);destination(oms_destination);};

  7. Restarts the syslog daemon and the Log Analytics agent:

    service syslog-ng restart

/opt/microsoft/omsagent/bin/service_control restart [workspaceID]

  8. Checks that the necessary connections are established: tcp 514 for receiving data, tcp 25226 for internal communication between the syslog daemon and the Log Analytics agent:

    netstat -an | grep 514

netstat -an | grep 25226

  9. Checks that the syslog daemon is receiving data on port 514, and that the agent is receiving data on port 25226:

    sudo tcpdump -A -ni any port 514 -vv

sudo tcpdump -A -ni any port 25226 -vv

  10. Sends MOCK data to port 514 on localhost. This data should be observable in the Microsoft Sentinel workspace by running the following query:

    CommonSecurityLog
| where DeviceProduct == "MOCK"

Verify your data connector prerequisites

Use the following sections to check your CEF or Syslog data connector prerequisites.

Azure Virtual Machine as a CEF collector

If you're using an Azure Virtual Machine as a CEF collector, verify the following:

  • Before you deploy the Common Event Format Data connector Python script, make sure that your Virtual Machine isn't already connected to an existing Log Analytics workspace. You can find this information on the Log Analytics Workspace Virtual Machine list, where a VM that's connected to a Syslog workspace is listed as Connected.

  • Make sure that Microsoft Sentinel is connected to the correct Log Analytics workspace, with the SecurityInsights solution installed.

    For more information, see Step 1: Deploy the log forwarder.

  • Make sure that your machine is sized correctly with at least the minimum required prerequisites. For more information, see CEF prerequisites.

On-premises or a non-Azure Virtual Machine

If you are using an on-premises machine or a non-Azure virtual machine for your data connector, make sure that you've run the installation script on a fresh installation of a supported Linux operating system:

Tip

You can also find this script from the Common Event Format data connector page in Microsoft Sentinel.

sudo wget -O cef_installer.py https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/CEF/cef_installer.py&&sudo python cef_installer.py <WorkspaceId> <Primary Key>

Enable your CEF facility and log severity collection

The Syslog server, either rsyslog or syslog-ng, forwards any data defined in the relevant configuration file, which is automatically populated by the settings defined in your Log Analytics workspace.

Make sure to add details about the facilities and severity log levels that you want to be ingested into Microsoft Sentinel. The configuration process may take about 20 minutes.

For more information, see Deployment script explained.

For example, for an rsyslog server, run the following command to display the current settings for your Syslog forwarding, and review any changes to the configuration file:

cat /etc/rsyslog.d/security-config-omsagent.conf

In this case, for rsyslog, output similar to the following should display:

if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then @@127.0.0.1:25226

Troubleshoot operating system issues

This section describes how to troubleshoot issues that are certainly derived from the operating system configuration.

To troubleshoot operating system issues:

  1. If you haven't yet, verify that you're working with a supported operating system and Python version. For more information, see CEF prerequisites.

  2. If your Virtual Machine is in Azure, verify that the network security group (NSG) allows inbound TCP/UDP connectivity from your log client (Sender) on port 514.

  3. Verify that packets are arriving to the Syslog Collector. To capture the syslog packets arriving to the Syslog Collector, run:

    tcpdump -Ani any port 514 and host <ip_address_of_sender> -vv

  4. Do one of the following:

    • If you do not see any packets arriving, confirm the NSG security group permissions and the routing path to the Syslog Collector.

    • If you do see packets arriving, confirm that they are not being rejected.

    If you see rejected packets, confirm that the IP tables are not blocking the connections.

    To confirm that packets are not being rejected, run:

    watch -n 2 -d iptables -nvL

  5. Verify whether the CEF server is processing the logs. Run:

    tail -f /var/log/messages or tail -f /var/log/syslog

    Any CEF logs being processed are displayed in plain text.

  6. Confirm that the rsyslog server is listening on TCP/UDP port 514. Run:

    netstat -anp | grep syslog

    If you have any CEF or ASA logs being sent to your Syslog Collector, you should see an established connection on TCP port 25226.

    For example:

    0 127.0.0.1:36120 127.0.0.1:25226 ESTABLISHED 1055/rsyslogd

    If the connection is blocked, you may have a blocked SELinux connection to the OMS agent, or a blocked firewall process. Use the relevant instructions below to determine the issue.

SELinux blocking connection to the OMS agent

This procedure describes how to confirm whether SELinux is currently in a permissive state, or is blocking a connection to the OMS agent. This procedure is relevant when your operating system is a distribution from RedHat or CentOS, and for both CEF and Syslog data connectors.

Note

Microsoft Sentinel support for CEF and Syslog only includes FIPS hardening. Other hardening methods, such as SELinux or CIS are not currently supported.

  1. Run:

    sestatus

    The status is displayed as one of the following:

    • disabled. This configuration is supported for your connection to Microsoft Sentinel.
    • permissive. This configuration is supported for your connection to Microsoft Sentinel.
    • enforced. This configuration is not supported, and you must either disable the status or set it to permissive.

  2. If the status is currently set to enforced, turn it off temporarily to confirm whether this was the blocker. Run:

    setenforce 0

    Note

    This step turns off SELinux only until the server reboots. Modify the SELinux configuration to keep it turned off.

  3. To verify whether the change was successful, run:

    getenforce

    The permissive state should be returned.

Important

This setting update is lost when the system is rebooted. To permanently update this setting to permissive, modify the /etc/selinux/config file, changing the SELINUX value to SELINUX=permissive.

For more information, see RedHat documentation.

Blocked firewall policy

This procedure describes how to verify whether a firewall policy is blocking the connection from the Rsyslog daemon to the OMS agent, and how to disable it as needed. This procedure is relevant for both CEF and Syslog data connectors.

  1. Run the following command to verify whether there are any rejects in the IP tables, indicating traffic that's being dropped by the firewall policy:

    watch -n 2 -d iptables -nvL

  2. To keep the firewall policy enabled, create a policy rule to allow the connections. Add rules as needed to allow the TCP/UDP ports 25226 and 25224 through the active firewall.

    For example:

    Every 2.0s: iptables -nvL                      rsyslog: Wed Jul  7 15:56:13 2021

Chain INPUT (policy ACCEPT 6185K packets, 2466M bytes)
 pkts bytes target     prot opt in     out     source               destination


Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination


Chain OUTPUT (policy ACCEPT 6792K packets, 6348M bytes)
 pkts bytes target     prot opt in     out     source               destination

  3. To create a rule to allow TCP/UDP ports 25226 and 25224 through the active firewall, add rules as needed.

    1. To install the Firewall Policy editor, run:

      yum install policycoreutils-python

    2. Add the firewall rules to the firewall policy. For example:

      sudo firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -p tcp --dport 25226  -j ACCEPT
sudo firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -p udp --dport 25224  -j ACCEPT
sudo firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -p tcp --dport 25224  -j ACCEPT

    3. Verify that the exception was added. Run:

      sudo firewall-cmd --direct --get-rules ipv4 filter INPUT

    4. Reload the firewall. Run:

      sudo firewall-cmd --reload

Note

To disable the firewall, run: sudo systemctl disable firewalld

If the steps described earlier in this article do not solve your issue, you may have a connectivity problem between the OMS Agent and the Microsoft Sentinel workspace.

In such cases, continue troubleshooting by verifying the following:

  • Make sure that you can see packets arriving on TCP/UDP port 514 on the Syslog collector

  • Make sure that you can see logs being written to the local log file, either /var/log/messages or /var/log/syslog

  • Make sure that you can see data packets flowing on port 25226

  • Make sure that your virtual machine has an outbound connection to port 443 via TCP, or can connect to the Log Analytics endpoints

  • Make sure that you have access to required URLs from your CEF collector through your firewall policy. For more information, see Log Analytics agent firewall requirements.

Run the following command to determine if the agent is communicating successfully with Azure, or if the OMS agent is blocked from connecting to the Log Analytics workspace.

Heartbeat
 | where Computer contains "<computername>"
 | sort by TimeGenerated desc

A log entry is returned if the agent is communicating successfully. Otherwise, the OMS agent may be blocked.

Next steps

If the troubleshooting steps in this article have not helped your issue, open a support ticket or use the Microsoft Sentinel community resources. For more information, see Useful resources for working with Microsoft Sentinel.

To learn more about Microsoft Sentinel, see the following articles: