Collect logs from text files with the Azure Monitor Agent and ingest to Microsoft Sentinel

• Attiecas uz:

  Microsoft Sentinel in the Azure portal, Microsoft Sentinel in the Microsoft Defender portal

This article describes how to use the Custom Logs via AMA connector to quickly filter and ingest logs in text-file format from network or security applications installed on Windows or Linux machines.

Many applications log data to text files instead of standard logging services like Windows Event log or Syslog. You can use the Azure Monitor Agent (AMA) to collect data in text files of nonstandard formats from both Windows and Linux computers. The AMA can also effect transformations on the data at the time of collection, to parse it into different fields.

For more information about the applications for which Microsoft Sentinel has solutions to support log collection, see Custom Logs via AMA data connector - Configure data ingestion to Microsoft Sentinel from specific applications.

For more general information about ingesting custom logs from text files, see Collect logs from a text file with Azure Monitor Agent.

Important

• The Custom Logs via AMA data connector is currently in PREVIEW. See the Supplemental Terms of Use for Microsoft Azure Previews for additional legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.

• Microsoft Sentinel is now generally available within the Microsoft unified security operations platform in the Microsoft Defender portal. For more information, see Microsoft Sentinel in the Microsoft Defender portal.

Prerequisites

Before you begin, you must have the resources configured and the appropriate permissions assigned, as described in this section.

Microsoft Sentinel prerequisites

• Install the Microsoft Sentinel solution that matches your application and make sure you have the permissions to complete the steps in this article. You can find these solutions in the Content hub in Microsoft Sentinel, and they all include the Custom Logs via AMA connector.

  For the list of applications that have solutions in the content hub, see Specific instructions per application. If there isn't a solution available for your application, install the Custom Logs via AMA solution.

  For more information, see Discover and manage Microsoft Sentinel out-of-the-box content.

• Have an Azure account with the following Azure role-based access control (Azure RBAC) roles:

  Built-in role	Scope	Reason
  - Virtual Machine Contributor - Azure Connected Machine    Resource Administrator	Virtual machines (VM) Virtual Machine Scale Sets Azure Arc-enabled servers	To deploy the agent
  Any role that includes the action Microsoft.Resources/deployments/*	Subscription Resource group Existing data collection rule	To deploy Azure Resource Manager templates
  Monitoring Contributor	Subscription Resource group Existing data collection rule	To create or edit data collection rules

Log forwarder prerequisites

Certain custom applications are hosted on closed appliances that necessitate sending their logs to an external log collector/forwarder. In such a scenario, the following prerequisites apply to the log forwarder:

• You must have a designated Linux VM as a log forwarder to collect logs.

  • Create a Linux VM in the Azure portal.
  • Supported Linux operating systems for Azure Monitor Agent.

• If your log forwarder isn't an Azure virtual machine, it must have the Azure Arc Connected Machine agent installed on it.

• The Linux log forwarder VM must have Python 2.7 or 3 installed. Use the python --version or python3 --version command to check. If you're using Python 3, make sure it's set as the default command on the machine, or run scripts with the 'python3' command instead of 'python'.

• The log forwarder must have either the syslog-ng or rsyslog daemon enabled.

• For space requirements for your log forwarder, refer to the Azure Monitor Agent Performance Benchmark. You can also review this blog post, which includes designs for scalable ingestion.

• Your log sources, security devices, and appliances must be configured to send their log messages to the log forwarder's syslog daemon instead of to their local syslog daemon.

Machine security prerequisites

Configure the log forwarder machine's security according to your organization's security policy. For example, configure your network to align with your corporate network security policy and change the ports and protocols in the daemon to align with your requirements. To improve your machine security configuration, secure your VM in Azure, or review these best practices for network security.

If your devices are sending logs over TLS because, for example, your log forwarder is in the cloud, you need to configure the syslog daemon (rsyslog or syslog-ng) to communicate in TLS. For more information, see:

• Encrypt Syslog traffic with TLS – rsyslog
• Encrypt log messages with TLS – syslog-ng

Configure the data connector

The setup process for the Custom Logs via AMA data connector includes the following steps:

1. Create the destination table in Log Analytics (or Advanced Hunting if you're in the Defender portal).

   The table's name must end with _CL and it must consist of only the following two fields:

   • TimeGenerated (of type DateTime): the timestamp of the creation of the log message.
   • RawData (of type String): the log message in its entirety.
     (If you're collecting logs from a log forwarder and not directly from the device hosting the application, name this field Message instead of RawData.)

2. Install the Azure Monitor Agent and create a Data Collection Rule (DCR) by using either of the following methods:

   • Azure or Defender portal
   • Azure Resource Manager template

3. If you're collecting logs using a log forwarder, configure the syslog daemon on that machine to listen for messages from other sources, and open the required local ports. For details, see Configure the log forwarder to accept logs.

Select the appropriate tab for instructions.

Azure or Defender portal

Create data collection rule (DCR)

To get started, open either the Custom Logs via AMA data connector in Microsoft Sentinel and create a data collection rule (DCR).

1. For Microsoft Sentinel in the Azure portal, under Configuration, select Data connectors.
   For Microsoft Sentinel in the Defender portal, select Microsoft Sentinel > Configuration > Data connectors.

2. Type custom in the Search box. From the results, select the Custom Logs via AMA connector.

3. Select Open connector page on the details pane.

   

4. In the Configuration area, select +Create data collection rule.

   

5. In the Basic tab:

   • Type a DCR name.
   • Select your subscription.
   • Select the resource group where you want to locate your DCR.

   

6. Select Next: Resources >.

Define VM resources

In the Resources tab, select the machines from which you want to collect the logs. These are either the machines on which your application is installed, or your log forwarder machines. If the machine you're looking for doesn't appear in the list, it might not be an Azure VM with the Azure Connected Machine agent installed.

1. Use the available filters or search box to find the machine you're looking for. Expand a subscription in the list to see its resource groups, and a resource group to see its VMs.

2. Select the machine that you want to collect logs from. The check box appears next to the VM name when you hover over it.

   

   If the machines you selected don't already have the Azure Monitor Agent installed on them, the agent is installed when the DCR is created and deployed.

3. Review your changes and select Next: Collect >.

Configure the DCR for your application

1. In the Collect tab, select your application or device type from the Select device type (optional) drop-down box, or leave it as Custom new table if your application or device isn't listed.

2. If you chose one of the listed applications or devices, the Table name field is automatically populated with the right table name. If you chose Custom new table, enter a table name under Table name. The name must end with the _CL suffix.

3. In the File pattern field, enter the path and file name of the text log files to be collected. To find the default file names and paths for each application or device type, see Specific instructions per application type. You don't have to use the default file names or paths, and you can use wildcards in the file name.

4. In the Transform field, if you chose a custom new table in step 1, enter a Kusto query that applies a transformation of your choice to the data.

   If you chose one of the listed applications or devices in step 1, this field is automatically populated with the proper transformation. DO NOT edit the transformation that appears there. Depending on the chosen type, this value should be one of the following:

   • source (the default—no transformation)
   • source | project-rename Message=RawData (for devices that send logs to a forwarder)

5. Review your selections and select Next: Review + create.

Review and create the rule

After you complete all the tabs, review what you entered and create the data collection rule.

1. In the Review and create tab, select Create.

   

   The connector installs the Azure Monitor Agent on the machines you selected when creating your DCR.

2. Check the notifications in the Azure portal or Microsoft Defender portal to see when the DCR is created and the agent is installed.

3. Select Refresh on the connector page to see the DCR displayed in the list.

Resource Manager template

Install the Azure Monitor Agent

Follow the appropriate instructions from the Azure Monitor documentation to install the Azure Monitor Agent on the machine hosting your application, or on your log forwarder. Use the instructions for Windows or for Linux, as appropriate.

• Install the AMA using PowerShell
• Install the AMA using the Azure CLI
• Install the AMA using an Azure Resource Manager template

Create Data Collection Rules (DCRs) using the Azure Monitor Logs Ingestion API. For more information, see Data collection rules in Azure Monitor.

Create the data collection rule

Use the following ARM template to create or modify a DCR for collecting text log files:

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "resources": [
        {
            "type": "Microsoft.Insights/dataCollectionRules",
            "name": "{DCR_NAME}",
            "location": "{DCR_LOCATION}",
            "apiVersion": "2022-06-01",
            "properties": {
                "streamDeclarations": {
                    "Custom-Text-{TABLE_NAME}": {
                        "columns": [
                            {
                                "name": "TimeGenerated",
                                "type": "datetime"
                            },
                            {
                                "name": "RawData",
                                "type": "string"
                            },
                        ]
                    }
                },
                "dataSources": {
                    "logFiles": [
                        {
                            "streams": [
                                "Custom-Text-{TABLE_NAME}"
                            ],
                            "filePatterns": [
                                "{LOCAL_PATH_FILE_1}","{LOCAL_PATH_FILE_2}"
                            ],
                            "format": "text",
                            "name": "Custom-Text-{TABLE_NAME}"
                        }
                    ]
                },
                "destinations": {
                    "logAnalytics": [
                        {
                            "workspaceResourceId": "{WORKSPACE_RESOURCE_PATH}",
                            "workspaceId": "{WORKSPACE_ID}",
                            "name": "workspace"
                        }
                    ]
                },
                "dataFlows": [
                    {
                        "streams": [
                            "Custom-Text-{TABLE_NAME}"
                        ],
                        "destinations": [
                            "DataCollectionEvent"
                        ],
                        "transformKql": "source",
                        "outputStream": "Custom-{TABLE_NAME}"
                    }
                ]
            }
        }
    ]
}

Replace the {PLACE_HOLDER} values with the following values:

Placeholder	Value
{DCR_NAME}	The name you choose for your Data Collection Rule. It must be unique within your workspace.
{DCR_LOCATION}	The region where the resource group containing the DCR is located.
{TABLE_NAME}	The name of the destination table in Log Analytics. Must end with _CL.
{LOCAL_PATH_FILE_1} (required), {LOCAL_PATH_FILE_2} (optional)	Paths and file names of the text files containing the logs you want to collect. These must be on the machine where the Azure Monitor Agent is installed.
{WORKSPACE_RESOURCE_PATH}	The Azure resource path of your Microsoft Sentinel workspace.
{WORKSPACE_ID}	The GUID of your Microsoft Sentinel workspace.

Associate the DCR with the Azure Monitor Agent

If you create the DCR using an ARM template, you still must associate the DCR with the agents that will use it. You can edit the DCR in the Azure portal and select the agents as described in Define VM resources.

Configure the log forwarder to accept logs

If you're collecting logs from an appliance using a log forwarder, configure the syslog daemon on the log forwarder to listen for messages from other machines, and open the necessary local ports.

1. Copy the following command line:

   sudo wget -O Forwarder_AMA_installer.py https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/Syslog/Forwarder_AMA_installer.py&&sudo python Forwarder_AMA_installer.py
   

2. Sign in to the log forwarder machine where you just installed the AMA.

3. Paste the command you copied in the last step to launch the installation script.
   The script configures the rsyslog or syslog-ng daemon to use the required protocol and restarts the daemon. The script opens port 514 to listen to incoming messages in both UDP and TCP protocols. To change this setting, refer to the syslog daemon configuration file according to the daemon type running on the machine:

   • Rsyslog: /etc/rsyslog.conf
   • Syslog-ng: /etc/syslog-ng/syslog-ng.conf

   If you're using Python 3, and it's not set as the default command on the machine, substitute python3 for python in the pasted command. See Log forwarder prerequisites.

   Note

   To avoid Full Disk scenarios where the agent can't function, we recommend that you set the syslog-ng or rsyslog configuration not to store unneeded logs. A Full Disk scenario disrupts the function of the installed AMA. For more information, see RSyslog or Syslog-ng.

Configure the security device or appliance

For specific instructions to configure your security application or appliance, see Custom Logs via AMA data connector - Configure data ingestion to Microsoft Sentinel from specific applications

Contact the solution provider for more information or where information is unavailable for the appliance or device.

Related content

• Data collection rules in Azure Monitor
• Collect logs from a text file with Azure Monitor Agent