Prevent the duplicate logs in syslogs table in azure

Question

We have a dcr for the custom table to ingest logs from a device. While logs are successfully appearing, we also receiving same logs in syslogs resulting unnecessary storage kindly assist.

Answer 1

Hello rajesh kumar Thank you for posting your query on Microsoft Q&A platform.

Azure Monitor does not have a deduplication layer across tables or streams.

• Azure Monitor treats each ingestion pipeline as independent
• A DCR is an instruction, not a “routing rule with awareness of other DCRs”
• If the same log event enters AMA once, but matches multiple DCR data sources, it will be materialized multiple times

So when you see:

• Same device
• Same syslog message
• Appearing in:
  • Syslog (built‑in table)
    • <Custom>_CL (custom table)

That means the log was accepted by two independent ingestion paths, not duplicated after ingestion.

There are only three ways a syslog message can reach Log Analytics:

1. Linux Syslog data source in a DCR
2. CEF/Syslog Security stream
3. Custom ingestion (custom table) that still starts from syslog

In your case:

• The device sends logs → rsyslog/syslog‑ng
• AMA reads from syslog socket
• AMA evaluates all DCRs associated to that machine
• If:
  • DCR‑A says “send this message to custom table”
    • DCR‑B says “send syslog messages to Syslog table”
    • Both will succeed
    • Azure does not know they are “the same event”

This is expected behavior.

Azure Monitor cannot dedupe because TimeGenerated, ingestion time, and table schema are different.

Even if two rows look identical:

• They land in different tables
• With different schemas
• Potentially different ingestion timestamps

Deduplication would break audit guarantees — so it is intentionally absent.

As a resolution try below workarounds:

1. One stream → one destination

If logs are meant only for the custom table:

• Do not collect Syslog at all for that device

Practically:

• The device should still emit syslog
• AMA should only have one DCR that:
  • Reads syslog
    • Transforms
      • Writes to the custom table
      • No Syslog stream in any other DCR scoped to that machine

This is the only way to guarantee zero duplication.

2. Ingestion-time dropping: If you must keep Syslog for other messages:

• The Syslog DCR must explicitly drop messages already handled by the custom DCR

Because:

• KQL queries, retention, and purge do not reduce ingestion cost
• Only transformKql prevents storage and billing

You filter based on something inherent to the log, such as:

• Facility
• Program / process name
• Tag inserted by rsyslog
• Message prefix / pattern

This makes the two DCRs mutually exclusive.

3. Facility separation at the source: This is the most correct approach.

• Device sends:
  • Business logs → local4
    • System logs → standard facilities
    • Custom table DCR:
      • Collects only local4
      • Syslog DCR:
        • Explicitly excludes local4

Why this is superior:

• No KQL filtering cost
• No ambiguity
• Clear ownership of data
• Easier troubleshooting
• Scales cleanly

AMA does not send logs “to Azure” — it sends logs to DCRs. If two DCRs accept the same message, Azure stores it twice. Once you think in DCR‑first, the behavior becomes obvious and predictable.

Duplicate syslog entries are not caused by a logging bug or agent issue, they occur because the same syslog message matches multiple Data Collection Rules. Azure Monitor does not deduplicate across tables, so the only way to prevent this is to ensure that each syslog stream is ingested by exactly one DCR, either by disabling Syslog collection for that device, separating facilities at the source, or dropping overlapping messages at ingestion time using DCR transformations.

Thanks,
Suchitra.

Answer 2

The logs are duplicated because the same device/syslog stream is being ingested both via the custom-table DCR and via a Syslog DCR that writes to the built‑in Syslog table. To prevent duplicates in Syslog, remove or filter that second ingestion path.

Use one of these approaches (depending on how the device and DCRs are configured):

1. Stop collecting the device’s logs into Syslog via DCR
   • Open the DCR that has Linux Syslog as Data source type.
   • On the Collect and deliver tab, under Linux Syslog, set the Minimum log level to NONE for the facilities used by this device so that no events are collected for those facilities.
   • Alternatively, remove the device/VM from the scope of that Syslog DCR if it should not send any logs to the Syslog table.
   • Syslog data can only be sent to a Log Analytics workspace where it is stored in the Syslog table, so disabling the relevant facilities or scope will stop the duplicate ingestion there.
2. Filter out the duplicate stream at ingestion time using a DCR transformation If the same agent/device must still send some Syslog data to the Syslog table (for other messages), add an ingestion‑time transformation to exclude the specific logs that are already going to the custom table:
   • Edit the Syslog DCR and add a transformKql clause on the Syslog stream that filters out the unwanted records.
   • For example (pattern taken from the CEF/Syslog duplication guidance):

          "transformKql": "source\n  | where <condition that excludes the custom‑table logs>\n"
     

   • The condition could filter by facility, process name, or any other field that uniquely identifies the logs already ingested into the custom table.
3. Ensure each facility/stream is mapped to only one destination
   • When configuring Syslog in a DCR, select only the facilities and minimum log levels that should go to Syslog.
   • For the custom table DCR, ensure the stream declarations and transformations only cover the subset of messages that should land in that custom table.
   • Avoid having both DCRs ingest the same facility/severity combination from the same device.

These changes will keep the custom table ingestion intact while preventing the same records from being stored again in the Syslog table, reducing unnecessary storage and cost.

---

References:

• Syslog and Common Event Format (CEF) via AMA connectors for Microsoft Sentinel
• Collect Syslog events from virtual machine client with Azure Monitor
• Data collection rule (DCR) samples in Azure Monitor
• Logs Ingestion API in Azure Monitor
• Use Logstash to stream logs with pipeline transformations via DCR-based API