AMA+DCR for Syslog & CEF logs. CEF logs in CommonSecurityLog not parsing .

Question

Referring to this article: https://learn.microsoft.com/en-us/azure/sentinel/connect-cef-syslog

I trying to solution the following scenario:

• Using a single Linux log collector to forward both Syslog and CEF events to your Microsoft Sentinel workspaces using the Azure Monitor Agent (AMA).
• Ingest Syslog events in the Syslog table and CEF events in the CommonSecurityLog table.

According to the above article, there are two options to segregate the Syslog events and CEF events:

(1) Use the source device enables configuration of the target facility.

(2) Use an ingest time transformation to filter out CEF messages from the Syslog stream

I have successfully implemented (1), where i control the Syslog source device logging facility to auth, and CEF source device logging facility to localx. Then in the DCR i create two dataSources, one for Syslog events which allow facility auth only, and output to Syslog table. Another stream for CEF events, which allow facility localx only, and output to CommonSecurityLog table, both table's event is parsed perfectly.

I have not able to implement (2). I created a DCR with single dataSources, and two different dataFlows.

dataFlows stream 1 control the syslog events with this transformKQL: "source | where ProcessName !contains "CEF""

"outputStream": "Microsoft-Syslog"

dataFlows stream 2 control the CEF events with this transformKQL: "source | where ProcessName contains "CEF""

"outputStream": "Microsoft-CommonSecurityLog"

Both Syslog event and CEF event successfully written into Syslog and CommonSecurityLog respectively, but only CEF event are not parsed in the table. 90% of the fields are blank. Syslog event was ok, parsed perfectly.

*As image shown, 24 Aug events are via option 1(control source log facility). Parsing is working fine.

*As image shown, 7 Sep events are via option2(transformKQL). Parsing is not working.

What did i missed here? I implement the setup all according to Microsoft documentation. See attached dcr:

dcr-CEF_SyslogTransformKQL.txt

Answer 1

@Hann, Yap Sheu I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to "Accept " the answer.

Solution: Resolved by @Hann, Yap Sheu

If you have any other questions or are still running into more issues, please let me know.
Thank you again for your time and patience throughout this issue.

Please remember to "Accept Answer" if any answer/reply helped, so that others in the community facing similar issues can easily find the solution.

Answer 2

I suspect this is a schema issue, writing Syslog --> CEF. Marko has a recent article where he resolves this for a custom log, perhaps this is worth a look: https://www.linkedin.com/pulse/filter-split-firewallcef-logs-multiple-sentinel-tables-marko-lauren/

You ultimately may need to modify the transformKQL to do the Column mapping for Syslog --> CEF (i.e. it doesnt know what to put in the DeviceVendor column etc...)