Migrate to Innovate Summit:
Learn how migrating and modernizing to Azure can boost your business's performance, resilience, and security, enabling you to fully embrace AI.Register now
This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
Collect logs from text files with the Azure Monitor Agent and ingest to Microsoft Sentinel
Article
Applies to:
Microsoft Sentinel in the Azure portal, Microsoft Sentinel in the Microsoft Defender portal
This article describes how to use the Custom Logs via AMA connector to quickly filter and ingest logs in text-file format from network or security applications installed on Windows or Linux machines.
Many applications log data to text files instead of standard logging services like Windows Event log or Syslog. You can use the Azure Monitor Agent (AMA) to collect data in text files of nonstandard formats from both Windows and Linux computers. The AMA can also effect transformations on the data at the time of collection, to parse it into different fields.
The Custom Logs via AMA data connector is currently in PREVIEW. See the Supplemental Terms of Use for Microsoft Azure Previews for additional legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.
Microsoft Sentinel is generally available within Microsoft's unified security operations platform in the Microsoft Defender portal. For preview, Microsoft Sentinel is available in the Defender portal without Microsoft Defender XDR or an E5 license. For more information, see Microsoft Sentinel in the Microsoft Defender portal.
Prerequisites
Before you begin, you must have the resources configured and the appropriate permissions assigned, as described in this section.
Microsoft Sentinel prerequisites
Install the Microsoft Sentinel solution that matches your application and make sure you have the permissions to complete the steps in this article. You can find these solutions in the Content hub in Microsoft Sentinel, and they all include the Custom Logs via AMA connector.
For the list of applications that have solutions in the content hub, see Specific instructions per application. If there isn't a solution available for your application, install the Custom Logs via AMA solution.
Certain custom applications are hosted on closed appliances that necessitate sending their logs to an external log collector/forwarder. In such a scenario, the following prerequisites apply to the log forwarder:
You must have a designated Linux VM as a log forwarder to collect logs.
If your log forwarder isn't an Azure virtual machine, it must have the Azure Arc Connected Machine agent installed on it.
The Linux log forwarder VM must have Python 2.7 or 3 installed. Use the python --version or python3 --version command to check. If you're using Python 3, make sure it's set as the default command on the machine, or run scripts with the 'python3' command instead of 'python'.
The log forwarder must have either the syslog-ng or rsyslog daemon enabled.
Your log sources, security devices, and appliances must be configured to send their log messages to the log forwarder's syslog daemon instead of to their local syslog daemon.
Machine security prerequisites
Configure the log forwarder machine's security according to your organization's security policy. For example, configure your network to align with your corporate network security policy and change the ports and protocols in the daemon to align with your requirements. To improve your machine security configuration, secure your VM in Azure, or review these best practices for network security.
If your devices are sending logs over TLS because, for example, your log forwarder is in the cloud, you need to configure the syslog daemon (rsyslog or syslog-ng) to communicate in TLS. For more information, see:
The setup process for the Custom Logs via AMA data connector includes the following steps:
Create the destination table in Log Analytics (or Advanced Hunting if you're in the Defender portal).
The table's name must end with _CL and it must consist of only the following two fields:
TimeGenerated (of type DateTime): the timestamp of the creation of the log message.
RawData (of type String): the log message in its entirety.
(If you're collecting logs from a log forwarder and not directly from the device hosting the application, name this field Message instead of RawData.)
Install the Azure Monitor Agent and create a Data Collection Rule (DCR) by using either of the following methods:
If you're collecting logs using a log forwarder, configure the syslog daemon on that machine to listen for messages from other sources, and open the required local ports. For details, see Configure the log forwarder to accept logs.
To get started, open either the Custom Logs via AMA data connector in Microsoft Sentinel and create a data collection rule (DCR).
For Microsoft Sentinel in the Azure portal, under Configuration, select Data connectors. For Microsoft Sentinel in the Defender portal, select Microsoft Sentinel > Configuration > Data connectors.
Type custom in the Search box. From the results, select the Custom Logs via AMA connector.
Select Open connector page on the details pane.
In the Configuration area, select +Create data collection rule.
In the Basic tab:
Type a DCR name.
Select your subscription.
Select the resource group where you want to locate your DCR.
Select Next: Resources >.
Define VM resources
In the Resources tab, select the machines from which you want to collect the logs. These are either the machines on which your application is installed, or your log forwarder machines. If the machine you're looking for doesn't appear in the list, it might not be an Azure VM with the Azure Connected Machine agent installed.
Use the available filters or search box to find the machine you're looking for. Expand a subscription in the list to see its resource groups, and a resource group to see its VMs.
Select the machine that you want to collect logs from. The check box appears next to the VM name when you hover over it.
If the machines you selected don't already have the Azure Monitor Agent installed on them, the agent is installed when the DCR is created and deployed.
Review your changes and select Next: Collect >.
Configure the DCR for your application
In the Collect tab, select your application or device type from the Select device type (optional) drop-down box, or leave it as Custom new table if your application or device isn't listed.
If you chose one of the listed applications or devices, the Table name field is automatically populated with the right table name. If you chose Custom new table, enter a table name under Table name. The name must end with the _CL suffix.
In the File pattern field, enter the path and file name of the text log files to be collected. To find the default file names and paths for each application or device type, see Specific instructions per application type. You don't have to use the default file names or paths, and you can use wildcards in the file name.
In the Transform field, if you chose a custom new table in step 1, enter a Kusto query that applies a transformation of your choice to the data.
If you chose one of the listed applications or devices in step 1, this field is automatically populated with the proper transformation. DO NOT edit the transformation that appears there. Depending on the chosen type, this value should be one of the following:
source (the default—no transformation)
source | project-rename Message=RawData (for devices that send logs to a forwarder)
Review your selections and select Next: Review + create.
Review and create the rule
After you complete all the tabs, review what you entered and create the data collection rule.
In the Review and create tab, select Create.
The connector installs the Azure Monitor Agent on the machines you selected when creating your DCR.
Check the notifications in the Azure portal or Microsoft Defender portal to see when the DCR is created and the agent is installed.
Select Refresh on the connector page to see the DCR displayed in the list.
Install the Azure Monitor Agent
Follow the appropriate instructions from the Azure Monitor documentation to install the Azure Monitor Agent on the machine hosting your application, or on your log forwarder. Use the instructions for Windows or for Linux, as appropriate.
Paths and file names of the text files containing the logs you want to collect. These must be on the machine where the Azure Monitor Agent is installed.
{WORKSPACE_RESOURCE_PATH}
The Azure resource path of your Microsoft Sentinel workspace.
{WORKSPACE_ID}
The GUID of your Microsoft Sentinel workspace.
Associate the DCR with the Azure Monitor Agent
If you create the DCR using an ARM template, you still must associate the DCR with the agents that will use it. You can edit the DCR in the Azure portal and select the agents as described in Define VM resources.
Configure the log forwarder to accept logs
If you're collecting logs from an appliance using a log forwarder, configure the syslog daemon on the log forwarder to listen for messages from other machines, and open the necessary local ports.
Sign in to the log forwarder machine where you just installed the AMA.
Paste the command you copied in the last step to launch the installation script.
The script configures the rsyslog or syslog-ng daemon to use the required protocol and restarts the daemon. The script opens port 514 to listen to incoming messages in both UDP and TCP protocols. To change this setting, refer to the syslog daemon configuration file according to the daemon type running on the machine:
Rsyslog: /etc/rsyslog.conf
Syslog-ng: /etc/syslog-ng/syslog-ng.conf
If you're using Python 3, and it's not set as the default command on the machine, substitute python3 for python in the pasted command. See Log forwarder prerequisites.
Note
To avoid Full Disk scenarios where the agent can't function, we recommend that you set the syslog-ng or rsyslog configuration not to store unneeded logs. A Full Disk scenario disrupts the function of the installed AMA.
For more information, see RSyslog or Syslog-ng.
Learn how to configure data ingestion into Microsoft Sentinel from specific or custom applications that produce logs as text files, using the Custom Logs via AMA data connector or manual configuration.
Ingest syslog messages from linux machines and from network and security devices and appliances to Microsoft Sentinel, using data connectors based on the Azure Monitor Agent (AMA).
Learn about how Azure Monitor's custom log ingestion and data transformation features can help you get any data into Microsoft Sentinel and shape it the way you want.