Stream CEF logs with the AMA connector

This article describes how to use the Common Event Format (CEF) via AMA connector to quickly filter and upload logs in the Common Event Format (CEF) from multiple on-premises appliances over Syslog.

The connector uses the Azure Monitor Agent (AMA), which uses Data Collection Rules (DCRs). With DCRs, you can filter the logs before they're ingested, for quicker upload, efficient analysis, and querying.

Important

The CEF via AMA connector is currently in PREVIEW. The Azure Preview Supplemental Terms include additional legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

The AMA is installed on a Linux machine that acts as a log forwarder, and the AMA collects the logs in the CEF format.

Important

On February 28th 2023, we will introduce changes to the CommonSecurityLog table schema. This means that custom queries will require being reviewed and updated. Out-of-the-box content (detections, hunting queries, workbooks, parsers, etc.) will be updated by Microsoft Sentinel.

Overview

What is CEF collection?

Many network, security appliances, and devices send their logs in the CEF format over Syslog. This format includes more structured information than Syslog, with information presented in a parsed key-value arrangement.

If your appliance or system sends logs over Syslog using CEF, the integration with Microsoft Sentinel allows you to easily run analytics and queries across the data.

CEF normalizes the data, making it more immediately useful for analysis with Microsoft Sentinel. Microsoft Sentinel also allows you to ingest unparsed Syslog events, and to analyze them with query time parsing.

How collection works with the Common Event Format (CEF) via AMA connector

Diagram showing the CEF log forwarding procedure.

  1. Your organization sets up a log forwarder (Linux VM), if one doesn't already exist. The forwarder can be on-premises or cloud-based.
  2. Your organization uploads CEF logs from your source devices to the forwarder.
  3. The AMA connector installed on the log forwarder collects and parses the logs.
  4. The connector streams the events to the Microsoft Sentinel workspace to be further analyzed.

When you install a log forwarder, the originating device must be configured to send Syslog events to the Syslog daemon on this forwarder instead of the local daemon. The Syslog daemon on the forwarder sends events to the Azure Monitor Agent over UDP. If this Linux forwarder is expected to collect a high volume of Syslog events, its Syslog daemon sends events to the agent over TCP instead. In either case, the agent then sends the events from there to your Log Analytics workspace in Microsoft Sentinel.

Diagram showing the data flow from syslog sources to the Microsoft Sentinel workspace, where the AMA is installed on a separate log-forwarding device.

Set up the Common Event Format (CEF) via AMA connector

Prerequisites

Before you begin, verify that you have:

  • The Microsoft Sentinel solution enabled.
  • A defined Microsoft Sentinel workspace.
  • A Linux machine to collect logs.
    • The Linux machine must have Python 2.7 or 3 installed on the Linux machine. Use the python --version or python3 --version command to check.
  • Either the syslog-ng or rsyslog daemon enabled.
  • To collect events from any system that isn't an Azure virtual machine, ensure that Azure Arc is installed.

Configure a log forwarder

To ingest Syslog and CEF logs into Microsoft Sentinel, you need to designate and configure a Linux machine that collects the logs from your devices and forwards them to your Microsoft Sentinel workspace. This machine can be a physical or virtual machine in your on-premises environment, an Azure VM, or a VM in another cloud. If this machine is not an Azure VM, it must have Azure Arc installed (see the prerequisites).

This machine has two components that take part in this process:

  • A Syslog daemon, either rsyslog or syslog-ng, which collects the logs.
  • The AMA, which forwards the logs to Microsoft Sentinel.

When you set up the connector and the DCR, you run a script on the Linux machine, which configures the built-in Linux Syslog daemon (rsyslog.d/syslog-ng) to listen for Syslog messages from your security solutions on TCP/UDP port 514.

The DCR installs the AMA to collect and parse the logs.

Log forwarder - security considerations

Make sure to configure the machine's security according to your organization's security policy. For example, you can configure your network to align with your corporate network security policy and change the ports and protocols in the daemon to align with your requirements. To improve your machine security configuration, secure your VM in Azure, or review these best practices for network security.

If your devices are sending Syslog and CEF logs over TLS (because, for example, your log forwarder is in the cloud), you need to configure the Syslog daemon (rsyslog or syslog-ng) to communicate in TLS:

Set up the connector

You can set up the connector in two ways:

  • Microsoft Sentinel portal. With this setup, you can create, manage, and delete DCRs per workspace.
  • API. With this setup, you can create, manage, and delete DCRs. This option is more flexible than the UI. For example, with the API, you can filter by specific log levels, where with the UI, you can only select a minimum log level.

Set up the connector in the Microsoft Sentinel portal (UI)

  1. Open the connector page and create the DCR
  2. Define resources (VMs)
  3. Select the data source type and create the DCR
  4. Run the installation script
Open the connector page and create the DCR
  1. Open the Azure portal and navigate to the Microsoft Sentinel service.

  2. Select Data connectors, and in the search bar, type CEF.

  3. Select the Common Event Format (CEF) via AMA (Preview) connector.

  4. Below the connector description, select Open connector page.

  5. In the Configuration area, select Create data collection rule.

  6. Under Basics:

    • Type a DCR name
    • Select your subscription
    • Select the resource group where your collector is defined

    Screenshot showing the DCR details in the Basics tab.

Define resources (VMs)

Select the machines on which you want to install the AMA. These machines are VMs or on-premises Linux machines with Arc installed.

  1. Select the Resources tab and select Add Resource(s).

  2. Select the VMs on which you want to install the connector to collect logs.

    Screenshot showing how to select resources when setting up the DCR.

  3. Review your changes and select Collect.

Select the data source type and create the DCR

Note

Using the same machine to forward both plain Syslog and CEF messages

If you plan to use this log forwarder machine to forward Syslog messages as well as CEF, in order to avoid the duplication of events to the Syslog and CommonSecurityLog tables:

On each source machine that sends logs to the forwarder in CEF format, you must edit the Syslog configuration file to remove the facilities that are being used to send CEF messages. This way, the facilities that are sent in CEF won't also be sent in Syslog.

  1. Select the Collect tab and select Linux syslog as the data source type.

  2. Configure the minimum log level for each facility. When you select a log level, Microsoft Sentinel collects logs for the selected level and other levels with higher severity. For example, if you select LOG_ERR, Microsoft Sentinel collects logs for the LOG_ERR, LOG_CRIT, LOG_ALERT, and LOG_EMERG levels.

    Screenshot showing how to select log levels when setting up the DCR.

  3. Review your changes and select Next: Review and create.

  4. In the Review and create tab, select Create.

Run the installation script
  1. Log in to the Linux forwarder machine, where you want the AMA to be installed.

  2. Run this command to launch the installation script:

    sudo wget -O Forwarder_AMA_installer.py https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/Syslog/Forwarder_AMA_installer.py&&sudo python Forwarder_AMA_installer.py
    

    The installation script configures the rsyslog or syslog-ng daemon to use the required protocol and restarts the daemon.

    Note

    To avoid Full Disk scenarios where the agent can't function, we recommend that you set the syslog-ng or rsyslog configuration not to store unneeded logs. A Full Disk scenario disrupts the function of the installed AMA. Read more about RSyslog or Syslog-ng.

Set up the connector with the API

You can create DCRs using the API. Learn more about DCRs.

Run this command to launch the installation script:

sudo wget -O Forwarder_AMA_installer.py https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/Syslog/Forwarder_AMA_installer.py&&sudo python Forwarder_AMA_installer.py 

The installation script configures the rsyslog or syslog-ng daemon to use the required protocol and restarts the daemon.

Request URL and header 

GET
https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Insights/dataCollectionRules/{dataCollectionRuleName}?api-version=2019-11-01-preview

Request body

Edit the template:

  • Verify that the streams field is set to Microsoft-CommonSecurityLog.
  • Add the filter and facility log levels in the facilityNames and logLevels parameters.
{
    "properties": {
        "immutableId": "dcr-bcc4039c90f0489b80927bbdf1f26008", 
        "dataSources": {
            "syslog": [
                {
                    "streams": [
                        "Microsoft-CommonSecurityLog"
                    ],

                    "facilityNames": [
                        "*"
                    ],
                    "logLevels": [ "*"
                    ],
                    "name": "sysLogsDataSource-1688419672"
                    }
                ]
            },
            "destinations": { 
                "logAnalytics": [
                {
                    "workspaceResourceId": "/subscriptions/{Your-Subscription-
Id}/resourceGroups/{resourceGroupName}/providers/Microsoft.OperationalInsights/workspaces/{SentinelWorkspaceName}", "workspaceId": "123x56xx-9123-xx4x-567x-89123xxx45",
"name": "la-140366483"
                }
            ]
        },
        "dataFlows": [
            {
                "streams": [
                    "Microsoft-CommonSecurityLog"
                ],
                "destinations": [ 
                    "la-140366483"
                ]
            }
        ],
        "provisioningState": "Succeeded"
    },
    "location": "westeurope", 
    "tags": {},
    "kind": "Linux",
    "id": "/subscriptions/{Your-Subscription- Id}/resourceGroups/{resourceGroupName}/providers/Microsoft.Insights/dataCollectionRules/{DCRName}",
    "name": "{DCRName}",
    "type": "Microsoft.Insights/dataCollectionRules", 
    "etag": "\"2401b6f3-0000-0d00-0000-618bbf430000\""
}

After you finish editing the template, use POST or PUT to deploy it:

PUT
https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Insights/dataCollectionRules/{dataCollectionRuleName}?api-version=2019-11-01-preview

Examples of facilities and log levels sections

Review these examples of the facilities and log levels settings. The name field includes the filter name.

This example collects events from the cron, daemon, local0, local3 and uucp facilities, with the Warning, Error, Critical, Alert, and Emergency log levels:

    "syslog": [
    {
            "streams": [
                "Microsoft-CommonSecurityLog"
        ],
        "facilityNames": [ 
            "cron",
            "daemon",
            "local0",
            "local3", 
            "uucp"
        ],
 
        "logLevels": [ 
            "Warning", 
            "Error", 
            "Critical", 
            "Alert", 
            "Emergency"
        ],
        "name": "sysLogsDataSource-1688419672"
    }
]

This example collects events for:

  • The authpriv and mark facilities with the Info, Notice, Warning, Error, Critical, Alert, and Emergency log levels
  • The daemon facility with the Warning, Error, Critical, Alert, and Emergency log levels
  • The kern, local0, local5, and news facilities with the Critical, Alert, and Emergency log levels
  • The mail and uucp facilities with the Emergency log level
        "syslog": [
            {
                "streams": [ 
                    "Microsoft-CommonSecurityLog"
            ],
                "facilityNames": [ 
                    "authpriv", 
                    "mark"
            ],
                "logLevels": [
                    "Info",
                    "Notice", 
                    "Warning", 
                    "Error", 
                    "Critical", 
                    "Alert", 
                    "Emergency"
            ],
                "name": "sysLogsDataSource--1469397783"
        },
        {
                "streams": [ "Microsoft-CommonSecurityLog"
            ],
                "facilityNames": [ 
                    "daemon"
            ],
                "logLevels": [ 
                    "Warning", 
                    "Error", 
                    "Critical", 
                    "Alert", 
                    "Emergency"
            ],
     
                "name": "sysLogsDataSource--1343576735"
        },
        {
                "streams": [ 
                    "Microsoft-CommonSecurityLog"
            ],
                "facilityNames": [ 
                    "kern",
                    "local0",
                    "local5", 
                    "news"
            ],
                "logLevels": [ 
                    "Critical", 
                    "Alert", 
                    "Emergency"
            ],
                "name": "sysLogsDataSource--1469572587"
        },
        {
                "streams": [ 
                    "Microsoft-CommonSecurityLog"
            ],
                "facilityNames": [ 
                    "mail",
                    "uucp"
            ],
                "logLevels": [ 
                    "Emergency"
            ],
                "name": "sysLogsDataSource-1689584311"
        }
    ]
}

Test the connector

  1. To validate that the syslog daemon is running on the UDP port and that the AMA is listening, run this command:

    netstat -lnptv
    

    You should see the rsyslog or syslog-ng daemon listening on port 514.

  2. To capture messages sent from a logger or a connected device, run this command in the background:

    tcpdump -I any port 514 -A vv &
    
  3. After you complete the validation, we recommend that you stop the tcpdump: Type fg and then select Ctrl+C.

  4. To send demo messages, do one of the following:

    • Use the netcat utility. In this example, the utility reads data posted through the echo command with the newline switch turned off. The utility then writes the data to UDP port 514 on the localhost with no timeout. To execute the netcat utility, you might need to install an additional package.

      echo -n "<164>CEF:0|Mock-test|MOCK|common=event-format-test|end|TRAFFIC|1|rt=$common=event-formatted-receive_time" | nc -u -w0 localhost 514
      
    • Use the logger. This example writes the message to the local 4 facility, at severity level Warning, to port 514, on the local host, in the CEF RFC format. The -t and --rfc3164 flags are used to comply with the expected RFC format.

      logger -p local4.warn -P 514 -n 127.0.0.1 --rfc3164 -t CEF "0|Mock-test|MOCK|common=event-format-test|end|TRAFFIC|1|rt=$common=event-formatted-receive_time"
      
  5. To verify that the connector is installed correctly, run the troubleshooting script with this command:

    sudo wget -O cef_AMA_troubleshoot.py https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/CEF/cef_AMA_troubleshoot.py&&sudo python cef_AMA_troubleshoot.py
    

Next steps

In this article, you learned how to set up the Windows CEF via AMA connector to upload data from appliances that support CEF over Syslog. To learn more about Microsoft Sentinel, see the following articles: