Collect text logs with Azure Monitor Agent

Many applications log information to text files instead of standard logging services such as Windows Event log or Syslog. This article explains how to collect text logs from monitored machines using Azure Monitor Agent by creating a data collection rule (DCR).

Prerequisites

To complete this procedure, you need:

  • Log Analytics workspace where you have at least contributor rights.

  • Data collection endpoint.

  • Permissions to create Data Collection Rule objects in the workspace.

  • A VM, Virtual Machine Scale Set, or Arc-enabled on-premises server that writes logs to a text file.

    Text file requirements:

    • Store on the local drive of the machine on which Azure Monitor Agent is running.
    • Delineate with an end of line.
    • Use ASCII or UTF-8 encoding. Other formats such as UTF-16 aren't supported.
    • Do not allow circular logging, log rotation where the file is overwritten with new entries, or renaming where a file is moved and a new file with the same name is opened.

Create a custom table

This step will create a new custom table, which is any table name that ends in _CL. Currently a direct REST call to the table management endpoint is used to create a table. The script at the end of this section is the input to the REST call.

The table created in the script has two columns TimeGenerated: datetime and RawData: string, which is the default schema for a custom text log. If you know your final schema, then you can add columns in the script before creating the table. If you don't, columns can always be added in the log analytics table UI.

The easiest way to make the REST call is from an Azure Cloud PowerShell command line (CLI). To open the shell, go to the Azure portal, press the Cloud Shell button, and select PowerShell. If this is your first-time using Azure Cloud PowerShell, you will need to walk through the one-time configuration wizard.

Copy and paste the following script in to PowerShell to create the table in your workspace. Make sure to replace the {subscription}, {resource group}, {workspace name}, and {table name} in the script. Make sure that there are no extra blanks at the beginning or end of the parameters

$tableParams = @'
{
   "properties": {
       "schema": {
              "name": "{TableName}_CL",
              "columns": [
       {
                               "name": "TimeGenerated",
                               "type": "DateTime"
                       }, 
                      {
                               "name": "RawData",
                               "type": "String"
                      }
             ]
       }
   }
}
'@

Invoke-AzRestMethod -Path "/subscriptions/{subscription}/resourcegroups/{resourcegroup}/providers/microsoft.operationalinsights/workspaces/{WorkspaceName}/tables/{TableName}_CL?api-version=2021-12-01-preview" -Method PUT -payload $tableParams

Press return to execute the code. You should see a 200 response, and details about the table you just created will show up. To validate that the table was created go to your workspace and select Tables on the left blade. You should see your table in the list.

Create data collection rule to collect text logs

The data collection rule defines:

  • Which source log files Azure Monitor Agent scans for new events.
  • How Azure Monitor transforms events during ingestion.
  • The destination Log Analytics workspace and table to which Azure Monitor sends the data.

You can define a data collection rule to send data from multiple machines to multiple Log Analytics workspaces, including workspaces in a different region or tenant. Create the data collection rule in the same region as your Log Analytics workspace.

Note

To send data across tenants, you must first enable Azure Lighthouse.

To create the data collection rule in the Azure portal:

  1. On the Monitor menu, select Data Collection Rules.

  2. Select Create to create a new data collection rule and associations.

    Screenshot that shows the Create button on the Data Collection Rules screen.

  3. Enter a Rule name and specify a Subscription, Resource Group, Region, Platform Type, and Data Collection Endpoint:

    • Region specifies where the DCR will be created. The virtual machines and their associations can be in any subscription or resource group in the tenant.
    • Platform Type specifies the type of resources this rule can apply to. The Custom option allows for both Windows and Linux types.
    • Data Collection Endpoint is required to collect custom logs.

    Screenshot that shows the Basics tab of the Data Collection Rule screen.

  4. On the Resources tab:

    1. Select + Add resources and associate resources to the data collection rule. Resources can be virtual machines, Virtual Machine Scale Sets, and Azure Arc for servers. The Azure portal installs Azure Monitor Agent on resources that don't already have it installed.

      Important

      The portal enables system-assigned managed identity on the target resources, along with existing user-assigned identities, if there are any. For existing applications, unless you specify the user-assigned identity in the request, the machine defaults to using system-assigned identity instead.

      If you need network isolation using private links, select existing endpoints from the same region for the respective resources or create a new endpoint.

    2. Select Enable Data Collection Endpoints.

    3. Select a data collection endpoint for each of the resources associate to the data collection rule.

    Screenshot that shows the Resources tab of the Data Collection Rule screen.

  5. On the Collect and deliver tab, select Add data source to add a data source and set a destination.

  6. Select Custom Text Logs.

    Screenshot that shows the Add data source screen for a data collection rule in Azure portal.

  7. Specify the following information:

    • File Pattern - Identifies where the log files are located on the local disk. You can enter multiple file patterns separated by commas.

      Examples of valid inputs:

      • 20220122-MyLog.txt
      • ProcessA_MyLog.txt
      • ErrorsOnly_MyLog.txt, WarningOnly_MyLog.txt

      Note

      Multiple log files of the same type commonly exist in the same directory. For example, a machine might create a new file every day to prevent the log file from growing too large. To collect log data in this scenario, you can use a file wildcard. Use the format C:\directoryA\directoryB\*MyLog.txt for Windows and /var/*.log for Linux. There is no support for directory wildcards.

    • Table name - The name of the destination table you created in your Log Analytics Workspace. For more information, see Prerequisites.

    • Record delimiter - Will be used in the future to allow delimiters other than the currently supported end of line (/r/n).

    • Transform - Add an ingestion-time transformation or leave as source if you don't need to transform the collected data.

  8. On the Destination tab, add one or more destinations for the data source. You can select multiple destinations of the same or different types. For instance, you can select multiple Log Analytics workspaces, which is also known as multihoming.

    Screenshot that shows the destination tabe of the Add data source screen for a data collection rule in Azure portal.

  9. Select Review + create to review the details of the data collection rule and association with the set of virtual machines.

  10. Select Create to create the data collection rule.

Note

It can take up to 5 minutes for data to be sent to the destinations after you create the data collection rule.

Sample log queries

The column names used here are for example only. The column names for your log will most likely be different.

  • Count the number of events by code.

    MyApp_CL
    | summarize count() by code
    

Sample alert rule

  • Create an alert rule on any error event.

    MyApp_CL
    | where status == "Error"
    | summarize AggregatedValue = count() by Computer, bin(TimeGenerated, 15m)
    

Troubleshoot

Use the following steps to troubleshoot collection of text logs.

Check if any custom logs have been received

Start by checking if any records have been collected for your custom log table by running the following query in Log Analytics. If records aren't returned, check the other sections for possible causes. This query looks for entires in the last two days, but you can modify for another time range. It can take 5-7 minutes for new data from your tables to be uploaded. Only new data will be uploaded any log file last written to prior to the DCR rules being created won't be uploaded.

<YourCustomLog>_CL
| where TimeGenerated > ago(48h)
| order by TimeGenerated desc

Verify that you created a custom table

You must create a custom log table in your Log Analytics workspace before you can send data to it.

Verify that the agent is sending heartbeats successfully

Verify that Azure Monitor agent is communicating properly by running the following query in Log Analytics to check if there are any records in the Heartbeat table.

Heartbeat
| where TimeGenerated > ago(24h)
| where Computer has "<computer name>"
| project TimeGenerated, Category, Version
| order by TimeGenerated desc

Verify that you specified the correct log location in the data collection rule

The data collection rule will have a section similar to the following. The filePatterns element specifies the path to the log file to collect from the agent computer. Check the agent computer to verify that this is correct.

"dataSources": [{
            "configuration": {
                "filePatterns": ["C:\\JavaLogs\\*.log"],
                "format": "text",
                "settings": {
                    "text": {
                        "recordStartTimestampFormat": "yyyy-MM-ddTHH:mm:ssK"
                    }
                }
            },
            "id": "myTabularLogDataSource",
            "kind": "logFile",
            "streams": [{
                    "stream": "Custom-TabularData-ABC"
                }
            ],
            "sendToChannels": ["gigl-dce-00000000000000000000000000000000"]
        }
    ]

This file pattern should correspond to the logs on the agent machine.

Screenshot of text log files on agent machine.

Verify that the text logs are being populated

The agent will only collect new content written to the log file being collected. If you're experimenting with the text logs collection feature, you can use the following script to generate sample logs.

# This script writes a new log entry at the specified interval indefinitely.
# Usage:
# .\GenerateCustomLogs.ps1 [interval to sleep]
#
# Press Ctrl+C to terminate script.
#
# Example:
# .\ GenerateCustomLogs.ps1 5

param (
    [Parameter(Mandatory=$true)][int]$sleepSeconds
)

$logFolder = "c:\\JavaLogs"
if (!(Test-Path -Path $logFolder))
{
    mkdir $logFolder
}

$logFileName = "TestLog-$(Get-Date -format yyyyMMddhhmm).log"
do
{
    $count++
    $randomContent = New-Guid
    $logRecord = "$(Get-Date -format s)Z Record number $count with random content $randomContent"
    $logRecord | Out-File "$logFolder\\$logFileName" -Encoding utf8 -Append
    Start-Sleep $sleepSeconds
}
while ($true)

Share logs with Microsoft

If everything is configured properly, but you're still not collecting log data, use the following procedure to collect diagnostics logs for Azure Monitor agent to share with the Azure Monitor group.

  1. Open an elevated PowerShell window.
  2. Change to directory C:\Packages\Plugins\Microsoft.Azure.Monitor.AzureMonitorWindowsAgent\[version]\.
  3. Execute the script: .\CollectAMALogs.ps1.
  4. Share the AMAFiles.zip file generated on the desktop.

Next steps

Learn more about: