Referring to this article: https://learn.microsoft.com/en-us/azure/sentinel/connect-cef-syslog
I trying to solution the following scenario:
- Using a single Linux log collector to forward both Syslog and CEF events to your Microsoft Sentinel workspaces using the Azure Monitor Agent (AMA).
- Ingest Syslog events in the Syslog table and CEF events in the CommonSecurityLog table.
According to the above article, there are two options to segregate the Syslog events and CEF events:
(1) Use the source device enables configuration of the target facility.
(2) Use an ingest time transformation to filter out CEF messages from the Syslog stream
I have successfully implemented (1), where i control the Syslog source device logging facility to auth, and CEF source device logging facility to localx. Then in the DCR i create two dataSources, one for Syslog events which allow facility auth only, and output to Syslog table. Another stream for CEF events, which allow facility localx only, and output to CommonSecurityLog table, both table's event is parsed perfectly.
I have not able to implement (2). I created a DCR with single dataSources, and two different dataFlows.
dataFlows stream 1 control the syslog events with this transformKQL: "source | where ProcessName !contains "CEF""
"outputStream": "Microsoft-Syslog"
dataFlows stream 2 control the CEF events with this transformKQL: "source | where ProcessName contains "CEF""
"outputStream": "Microsoft-CommonSecurityLog"
Both Syslog event and CEF event successfully written into Syslog and CommonSecurityLog respectively, but only CEF event are not parsed in the table. 90% of the fields are blank. Syslog event was ok, parsed perfectly.
*As image shown, 24 Aug events are via option 1(control source log facility). Parsing is working fine.
*As image shown, 7 Sep events are via option2(transformKQL). Parsing is not working.
What did i missed here? I implement the setup all according to Microsoft documentation. See attached dcr:
dcr-CEF_SyslogTransformKQL.txt