Ingest syslog and CEF messages to Microsoft Sentinel with the Azure Monitor Agent

This article describes how to use the Syslog via AMA and Common Event Format (CEF) via AMA connectors to quickly filter and ingest syslog messages, including messages in Common Event Format (CEF), from Linux machines and from network and security devices and appliances. To learn more about these data connectors, see Syslog and Common Event Format (CEF) via AMA connectors for Microsoft Sentinel.

Note

Container Insights now supports the automatic collection of Syslog events from Linux nodes in your AKS clusters. To learn more, see Syslog collection with Container Insights.

Prerequisites

Before you begin, you must have the resources configured and the appropriate permissions assigned, as described in this section.

Microsoft Sentinel prerequisites

Install the appropriate Microsoft Sentinel solution and make sure you have the permissions to complete the steps in this article.

Log forwarder prerequisites

If you're collecting messages from a log forwarder, the following prerequisites apply:

  • You must have a designated Linux VM as a log forwarder to collect logs.

  • If your log forwarder isn't an Azure virtual machine, it must have the Azure Arc Connected Machine agent installed on it.

  • The Linux log forwarder VM must have Python 2.7 or 3 installed. Use the python --version or python3 --version command to check. If you're using Python 3, make sure it's set as the default command on the machine, or run scripts with the 'python3' command instead of 'python'.

  • The log forwarder must have either the syslog-ng or rsyslog daemon enabled.

  • For space requirements for your log forwarder, refer to the Azure Monitor Agent Performance Benchmark. You can also review this blog post, which includes designs for scalable ingestion.

  • Your log sources, security devices, and appliances, must be configured to send their log messages to the log forwarder's syslog daemon instead of to their local syslog daemon.

Machine security prerequisites

Configure the machine's security according to your organization's security policy. For example, configure your network to align with your corporate network security policy and change the ports and protocols in the daemon to align with your requirements. To improve your machine security configuration, secure your VM in Azure, or review these best practices for network security.

If your devices are sending syslog and CEF logs over TLS because, for example, your log forwarder is in the cloud, you need to configure the syslog daemon (rsyslog or syslog-ng) to communicate in TLS. For more information, see:

Configure the data connector

The setup process for the Syslog via AMA or Common Event Format (CEF) via AMA data connectors includes the following steps:

  1. Install the Azure Monitor Agent and create a Data Collection Rule (DCR) by using either of the following methods:
  2. If you're collecting logs from other machines using a log forwarder, run the "installation" script on the log forwarder to configure the syslog daemon to listen for messages from other machines, and to open the necessary local ports.

Select the appropriate tab for instructions.

Create data collection rule (DCR)

To get started, open either the Syslog via AMA or Common Event Format (CEF) via AMA data connector in Microsoft Sentinel and create a data collection rule (DCR).

  1. For Microsoft Sentinel in the Azure portal, under Configuration, select Data connectors.
    For Microsoft Sentinel in the Defender portal, select Microsoft Sentinel > Configuration > Data connectors.

  2. For syslog, type Syslog in the Search box. From the results, select the Syslog via AMA connector.
    For CEF, type CEF in the Search box. From the results, select the Common Event Format (CEF) via AMA connector.

  3. Select Open connector page on the details pane.

  4. In the Configuration area, select +Create data collection rule.

    Screenshot showing the Syslog via AMA connector page.

    Screenshot showing the CEF via AMA connector page.

  5. In the Basic tab:

    • Type a DCR name.
    • Select your subscription.
    • Select the resource group where you want to locate your DCR.

    Screenshot showing the DCR details in the Basic tab.

  6. Select Next: Resources >.

Define VM resources

In the Resources tab, select the machines on which you want to install the AMA—in this case, your log forwarder machine. If your log forwarder doesn't appear in the list, it might not have the Azure Connected Machine agent installed.

  1. Use the available filters or search box to find your log forwarder VM. Expand a subscription in the list to see its resource groups, and a resource group to see its VMs.

  2. Select the log forwarder VM that you want to install the AMA on. The check box appears next to the VM name when you hover over it.

    Screenshot showing how to select resources when setting up the DCR.

  3. Review your changes and select Next: Collect >.

Select facilities and severities

Be aware that using the same facility for both syslog and CEF messages might result in data ingestion duplication. For more information, see Data ingestion duplication avoidance.

  1. In the Collect tab, select the minimum log level for each facility. When you select a log level, Microsoft Sentinel collects logs for the selected level and other levels with higher severity. For example, if you select LOG_ERR, Microsoft Sentinel collects logs for the LOG_ERR, LOG_CRIT, LOG_ALERT, and LOG_EMERG levels.

    Screenshot showing how to select log levels when setting up the DCR.

  2. Review your selections and select Next: Review + create.

Review and create the rule

After you complete all the tabs, review what you entered and create the data collection rule.

  1. In the Review and create tab, select Create.

    Screenshot showing how to review the configuration of the DCR and create it.

    The connector installs the Azure Monitor Agent on the machines you selected when creating your DCR.

  2. Check the notifications in the Azure portal or Microsoft Defender portal to see when the DCR is created and the agent is installed.

  3. Select Refresh on the connector page to see the DCR displayed in the list.

Run the "installation" script

If you're using a log forwarder, configure the syslog daemon to listen for messages from other machines, and open the necessary local ports.

  1. From the connector page, copy the command line that appears under Run the following command to install and apply the CEF collector:

    Screenshot of command line on connector page.

    Or copy it from here:

    sudo wget -O Forwarder_AMA_installer.py https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/Syslog/Forwarder_AMA_installer.py&&sudo python Forwarder_AMA_installer.py
    
  2. Sign in to the log forwarder machine where you just installed the AMA.

  3. Paste the command you copied in the last step to launch the installation script.
    The script configures the rsyslog or syslog-ng daemon to use the required protocol and restarts the daemon. The script opens port 514 to listen to incoming messages in both UDP and TCP protocols. To change this setting, refer to the syslog daemon configuration file according to the daemon type running on the machine:

    • Rsyslog: /etc/rsyslog.conf
    • Syslog-ng: /etc/syslog-ng/syslog-ng.conf

    If you're using Python 3, and it's not set as the default command on the machine, substitute python3 for python in the pasted command. See Log forwarder prerequisites.

    Note

    To avoid Full Disk scenarios where the agent can't function, we recommend that you set the syslog-ng or rsyslog configuration not to store unneeded logs. A Full Disk scenario disrupts the function of the installed AMA. For more information, see RSyslog or Syslog-ng.

Configure the security device or appliance

Get specific instructions to configure your security device or appliance by going to one of the following articles:

Contact the solution provider for more information or where information is unavailable for the appliance or device.

Test the connector

Verify that logs messages from your linux machine or security devices and appliances are ingested into Microsoft Sentinel.

  1. To validate that the syslog daemon is running on the UDP port and that the AMA is listening, run this command:

    netstat -lnptv
    

    You should see the rsyslog or syslog-ng daemon listening on port 514.

  2. To capture messages sent from a logger or a connected device, run this command in the background:

    tcpdump -i any port 514 -A -vv &
    
  3. After you complete the validation, we recommend that you stop the tcpdump: Type fg and then select Ctrl+C.

  4. To send demo messages, complete of the following steps:

    • Use the netcat utility. In this example, the utility reads data posted through the echo command with the newline switch turned off. The utility then writes the data to UDP port 514 on the localhost with no timeout. To execute the netcat utility, you might need to install another package.

      echo -n "<164>CEF:0|Mock-test|MOCK|common=event-format-test|end|TRAFFIC|1|rt=$common=event-formatted-receive_time" | nc -u -w0 localhost 514
      
    • Use the logger. This example writes the message to the local 4 facility, at severity level Warning, to port 514, on the local host, in the CEF RFC format. The -t and --rfc3164 flags are used to comply with the expected RFC format.

      logger -p local4.warn -P 514 -n 127.0.0.1 --rfc3164 -t CEF "0|Mock-test|MOCK|common=event-format-test|end|TRAFFIC|1|rt=$common=event-formatted-receive_time"
      
  5. To verify that the connector is installed correctly, run the troubleshooting script with one of these commands:

    • For CEF logs, run:

       sudo wget -O Sentinel_AMA_troubleshoot.py https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/Syslog/Sentinel_AMA_troubleshoot.py&&sudo python Sentinel_AMA_troubleshoot.py --cef
      
    • For Cisco Adaptive Security Appliance (ASA) logs, run:

      sudo wget -O Sentinel_AMA_troubleshoot.py https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/Syslog/Sentinel_AMA_troubleshoot.py&&sudo python Sentinel_AMA_troubleshoot.py --asa
      
    • For Cisco Firepower Threat Defense (FTD) logs, run:

      sudo wget -O Sentinel_AMA_troubleshoot.py https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/Syslog/Sentinel_AMA_troubleshoot.py&&sudo python Sentinel_AMA_troubleshoot.py --ftd