Azure Event Hub not sending Linux OS level logs to Splunk

Question

Hello,

I am trying to setup "Diagnostic settings" for the Linux VMs, so I can see the OS level logs and metrics in Splunk.

The Splunk can see "Administration" logs from the console. So that works fine. Also the windows VMs OS logs are visible in the Splunk. The problem is that after I installed the extensions on the Linux VMs, I still can't see the metrics and OS logs in the Splunk.

I created an Event Hub and ran the configuration by following this page: https://learn.microsoft.com/en-us/azure/virtual-machines/extensions/diagnostics-linux?toc=%2Fazure%2Fazure-monitor%2Ftoc.json&tabs=azcli#protected-settings

This is the command that I ran:

az vm extension set --publisher Microsoft.Azure.Diagnostics --name LinuxDiagnostic --version 4.0 --resource-group "xxx-IoT-dev-rg" --vm-name "xxxx26u3txinhdg" --protected-settings "/home/../protectedsettings.json" --settings "/home/../publicSettings.json" --enable-auto-upgrade true

when I verified and ran the status everything is showing 'successful status' as you see below. Can some please advice if there is something wrong with the json files? I assume that the issues lie in those two json configuration files. I tried to follow the instructions and that is what gleaned from the instructions. My public and protected settings files are at the bottom of this post as text files

thank you for your help

Status after vm extentions set:

get-azvm -resourcegroupname 'xxxx-IoT-dev-rg' -Name 'vm-yxxxxx' -Status

Code : ProvisioningState/succeeded

*Level               : Info*

*DisplayStatus       : Provisioning succeeded*

*Time                : 3/10/2023 7:49:43 PM*

Extensions[0] :

Name : AzureNetworkWatcherExtension

Type : Microsoft.Azure.NetworkWatcher.NetworkWatcherAgentLinux

TypeHandlerVersion : 1.4.2573.1

Statuses[0] :

*Code                : ProvisioningState/succeeded*

*Level               : Info*

*DisplayStatus       : Provisioning succeeded*

*Message             : Default configuration is in effect.*

Extensions[1] :

Name : LinuxDiagnostic

Type : Microsoft.Azure.Diagnostics.LinuxDiagnostic

TypeHandlerVersion : 4.1.10

Statuses[0] :

*Code                : ProvisioningState/succeeded*

*Level               : Info*

*DisplayStatus       : Provisioning succeeded*

*Message             : Enable succeeded, extension daemon started*

VMAgent :

VmAgentVersion : 2.9.0.4

ExtensionHandlers[0] :

*Type                : Microsoft.Azure.NetworkWatcher.NetworkWatcherAgentLinux*

*TypeHandlerVersion  : 1.4.2573.1*

*Status              :* 

  *Code              : ProvisioningState/succeeded*

  *Level             : Info*

  *DisplayStatus     : Ready*

  *Message           : Plugin enabled*

ExtensionHandlers[1] :

*Type                : Microsoft.Azure.Diagnostics.LinuxDiagnostic*

*TypeHandlerVersion  : 4.1.10*

*Status              :* 

  *Code              : ProvisioningState/succeeded*

  *Level             : Info*

  *DisplayStatus     : Ready*

  *Message           : Plugin enabled*

ExtensionHandlers[2] :

*Type                : Microsoft.CPlat.Core.RunCommandLinux*

*TypeHandlerVersion  : 1.0.5*

*Status              :* 

  *Code              : ProvisioningState/succeeded*

  *Level             : Info*

  *DisplayStatus     : Ready*

  *Message           : Plugin enabled*

Statuses[0] :

*Code                : ProvisioningState/succeeded*

*Level               : Info*

*DisplayStatus       : Ready*

*Message             : Guest Agent is running*

*Time                : 3/17/2023 7:12:55 PM*

Statuses[0] :

Code : ProvisioningState/succeeded

Level : Info

DisplayStatus : Provisioning succeeded

Time : 3/17/2023 6:40:48 PM

Statuses[1] :

Code : PowerState/running

Level : Info

DisplayStatus : VM running

Public settings.json: (file attached)publicsettings.txt

Protected settings.json (file attached)protectedsettings.txt

Answer 1

Hello Jezo, Libor

Welcome to Microsoft Q&A Platform, thanks for posting your query here.

Based on the JSON configuration you provided earlier, it appears that Azure Event Hub is configured to send logs from the Application and Security event logs, but not from the System event log which contains the Linux OS level logs.

To configure Azure Event Hub to send Linux OS level logs to Splunk, you will need to modify the JSON configuration to include the System event log as a source. You can do this by adding the following block of code to the "inputs" section:

{
    "source": "Syslog",
    "sourcetype": "_json",
    "index": "your_index",
    "disabled": 0,
    "input": {
        "kind": "Syslog",
        "disabled": false,
        "connection_host": "ip",
        "port": 514,
        "restrictToHost": "",
        "useACK": true,
        "queueSize": "10MB",
        "queueSaveInterval": "60",
        "queueType": "memory",
        "host": "your_splunk_host",
        "priority": "normal"
    }
}

Make sure to replace "your_index" with the name of the index where you want to store the logs, and "your_splunk_host" with the hostname or IP address of your Splunk server.

After making these changes, save the modified JSON configuration and apply it to your Azure Event Hub instance. This should allow you to receive and index Linux OS level logs in Splunk.

Hope this helps.