Syslog Transformation DCR not working

Greg Sneed 0

I need assistance troubleshooting a Syslog Transformation DCR used with Microsoft Sentinel. The Transformation DCR looks to work correctly in the Create Transformation wizard, but doesn't actually filter out the records.

I have a few Syslog/CEF forwarders deployed on premises collecting logs and are receiving both CEF and Syslog data. I've followed this guide https://learn.microsoft.com/en-us/azure/sentinel/cef-syslog-ama-overview?tabs=single#data-ingestion-duplication-avoidance but am still getting CEF data in both the CommonSecurityLog and Syslog tables.

My current Transformation DCR applied to the Syslog table:

source 
| where ProcessName !contains "CEF"

However, running the query below in Log Analytics Workspace (or Sentinel) produces results.

Syslog
| where ProcessName contains "CEF"

As stated above, when I run the transformation filter in the Syslog transformation wizard, it does filter the results as expected. But saving and applying the rule has no effect. The Transformation DCR has been in place for months so I don't think it's a timing issue.

Any help is appreciated!

AnuragSingh-MSFT 20,991 Reputation points

2024-05-30T04:34:41.4633333+00:00

@Greg Sneed, thank you for posting this question on Microsoft Q&A.

Based on the information shared in the question, could you please confirm that no other DCRs/source is configured to send logs to these tables in LA workspace. It might be possible that one of the DCR is configured fine, but there are potentially other DCRs which do not have the transformation query in place.

You could check the source (data sources) of DCR and also the source of logs in LA workspace to confirm that they match up.
Greg Sneed 0 Reputation points

2024-05-30T19:36:20.3433333+00:00
@AnuragSingh-MSFT I appreciate the quick reply!

I'm using Microsoft Sentinel on top of Log Analytics.

I have three DCR's in Azure Monitor:

Standard DCR to ingest Syslog (both Performance Counters and Syslog data)

Sentinel DCR to ingest CEF data

WorkspaceTransform DCR to filter out CEF data bound for the Syslog table (this is the one not working)
AnuragSingh-MSFT 20,991 Reputation points

2024-06-03T11:38:35.7866667+00:00

@Greg Sneed, thank you for the reply.

Couldit be possible that the first DCR (for collection of perf counters and Linux syslog) is also forwarding CEF data to the Syslog tables. Thus, while the 3rd DCR is blocking them, there is no transformation query blocking CEF on the first DCR?

In case adding the ingestion time transformation query on the first DCR does not help, I would suggest reaching out to Azure Support to have this environment reviewed along with the backend logs (if required).
Greg Sneed 0 Reputation points

2024-06-03T12:53:59.2+00:00

I think that is what's happening, but I also think that's what is supposed to happen. DCR #1 above is what applies to the syslog forwarder and tells it what logs to forward to Azure. DCR #3 applies in Azure to the Syslog table and filters what gets ingested into the Syslog table.

Please refer to the link in my original post. If I'm understanding it correctly, I should be able to prevent CEF data from hitting the Syslog table with DCR #3.
Greg Sneed 0 Reputation points

2024-06-04T13:55:16.8633333+00:00

@AnuragSingh-MSFT do you have any updates on this?
AnuragSingh-MSFT 20,991 Reputation points

2024-06-10T07:44:38.5366667+00:00
@Greg Sneed, thank you for the additional information and apologies for the delayed response.

A DCR in itself is a complete pipeline, i.e.,

It has configuration for the source about what needs to be collected (in this case the configuration for the syslog forwarder)

And it also contains transformation (query based) and destination (the destination LA workspace)

Also, note that the configuration of 1 DCR is independent of configuration set for another DCR.

Therefore, if indeed the first DCR is collecting syslog and CEF messages, the transformation query needs to be set on the first DCR and 3rd DCR will not even be required.

Hope this helps.
AnuragSingh-MSFT 20,991 Reputation points

2024-06-14T10:23:33.5133333+00:00

@Greg Sneed, Following-up to check if you had a chance to review the answer provided. If the answer did not help, please add more context/follow-up question for it.
Greg Sneed 0 Reputation points

2024-06-17T12:38:47.9666667+00:00

@AnuragSingh-MSFT I don't think the information you provided is correct. Please refer to the link I posted in my original post. I have 1 DCR for syslog data, and 1 DCR for CEF data. The other item is referred to as an "ingest time transformation" to filter out CEF messages from the Syslog stream. Therefore, I don't think the issue is conflicting DCRs.

If I have something wrong, can you please provide instructions on how to properly setup a single Linux appliance that can forward both Syslog and CEF data to Sentinel / LAW?

Share via

Syslog Transformation DCR not working