Issue with Syslog server Installed with Azure Arc & AMA

Question

we checked the rsyslog service in this Ubuntu server, getting error, can you please suggest on the best solution for this error.

Answer 1

@Venkatesh - thanks for reaching Microsoft QnA platform

Please do these checks on the Ubuntu Arc log forwarder and you will immediately know where the break is

sudo systemctl status azuremonitoragent

sudo systemctl status rsyslog

sudo systemctl status syslog-ng.service

Use syslog-ng status only if you use syslog-ng instead of rsyslog - https://learn.microsoft.com/ja-jp/azure/sentinel/cef-syslog-ama-troubleshooting

Please confirm the forwarder is receiving syslog traffic on port 514 - sudo tcpdump -i any port 514 -A -vv

So, if you see no packets here the issue is upstream network sender firewall routing or NSG not AMA

Validate rsyslog configuration and find the exact error causing rsyslog to fail

sudo rsyslogd -N1

sudo journalctl -u rsyslog --no-pager -n 200

The rsyslogd -N1 run is the fastest way to detect config errors that prevent rsyslog from starting

Please confirm rsyslog is configured to listen on 514 TCP and UDP - (https://learn.microsoft.com/ja-jp/azure/sentinel/cef-syslog-ama-troubleshooting#initial-verification-steps)

sudo grep -E 'imudp|imtcp' /etc/rsyslog.conf

Later confirm AMA forwarding configuration file exists for rsyslog (https://learn.microsoft.com/ja-jp/azure/sentinel/cef-syslog-ama-troubleshooting#verify-rsyslog-configuration)

sudo cat /etc/rsyslog.d/10-azuremonitoragent-omfwd.conf This file is expected on AMA 1.28 and newer and should begin with a comment stating it forwards logs to azuremonitoragent

If the machine is on AMA 1.27 or older the file name can be different and you should collect both the main rsyslog config and the AMA rsyslog snippet for troubleshooting

sudo ls -l /etc/rsyslog.conf

sudo ls -l /etc/rsyslog.d/

then confirm required ports are listening locally:

sudo ss -lnp | grep -E "28330|514"

Expected result

1. rsyslogd listening on 514 TCP and UDP
2. mdsd listening on 127.0.0.1 28330 TCP

please confirm AMA syslog listen port file exists and check it

sudo cat /etc/opt/microsoft/azuremonitoragent/config-cache/syslog.port

Confirm DCR was downloaded to the agent cache

For CEF specific DCR blobs

sudo grep -i -r "SECURITY_CEF_BLOB" /etc/opt/microsoft/azuremonitoragent/config-cache/configchunks/

Example for Cisco ASA specific DCR blobs

sudo grep -i -r "SECURITY_CISCO_ASA_BLOB" /etc/opt/microsoft/azuremonitoragent/config-cache/configchunks/

If these return nothing, then the DCR is likely not associated or not applied to the Arc machine yet.

Check AMA extension logs on the Linux machine

Check AMA data pipeline errors and disk space issues

df -h

sudo tail -n 200 /var/opt/microsoft/azuremonitoragent/log/mdsd.err

sudo tail -n 200 /var/opt/microsoft/azuremonitoragent/log/mdsd.warn

If mdsd.err shows No space left on device then AMA cannot persist syslog events locally and uploads will stop until disk is freed

Log Analytics validation query in the Sentinel workspace

Syslog

| sort by TimeGenerated desc

| take 50

If you have any further queries, let me know.

Regards

Himanshu