Inconsistent Duplicate Events in Azure Activity Logs API

Question

Inconsistent Duplicate Events in Azure Activity Logs API

Dev Parmar 20

Hi everyone,
We're using the Azure Activity Logs REST API to collect activity logs from our Azure environment, and we've come across some inconsistencies we'd like to understand better.

We're seeing duplicate events in the API response — events with identical content except for a slightly different submissionTimestamp. Screenshot 2025-06-02 at 4.30.36 PM.png 1.txt
We're also using the eventDataId field to filter out duplicates, but we've noticed cases where two events have the same eventDataId yet contain different log content (besides the eventDataId). Screenshot 2025-06-02 at 4.36.47 PM.png
2.txt

Has anyone else experienced similar behavior or inconsistencies? Any insights, explanations, or recommended workarounds would be greatly appreciated.
Thanks in advance!

Anonymous

2025-06-03T16:19:46.8833333+00:00
Hi @Dev Parmar,

It's common to see the same event appear multiple times with slightly different timestamps. These small differences — often just seconds or minutes — usually stem from the gap between when the event actually occurred and when it was recorded or submitted.

For example, in my lab tests, I observed the following three log entries:

Timestamps:

* Begin request: 2025-06-03T14:57:43.8Z

End request: 2025-06-03T14:57:44.4Z (Accepted)

End request: 2025-06-03T14:58:27.8Z (Succeeded)

Note: These logs were filtered using a unique event ID, ensuring no duplicate entries.

In another scenario you mentioned — where logs share the same eventDataId — the entries actually represent two distinct but related sources of logs:

1. Administrative Log Entry

* Category: Administrative

Operation: Microsoft.Authorization/roleAssignments/write (a user initiated a role assignment)

Caller: ******@gmail.com

Status: Started

Timestamp: 2025-05-28T12:18:18Z

2. Alert Trigger Entry

* Category: Alert

Description: "Alert: Create role assignment called on action groups: it_alert"

Operation: Microsoft.Insights/ActivityLogAlerts/Activated/action (an alert fired based on the above event)

Status: Succeeded

Timestamp: 2025-05-28T12:25:21Z

Here, both logs reference the same eventDataId because the alert was triggered by the original administrative action

Please let me know if you have any questions.
Anonymous

2025-06-05T10:21:20.28+00:00

Hi @Dev Parmar,

I just wanted to check if the above provided information worked for you or if you need any further assistance?

Please feel free to let us know, as we are always here to help whenever you need us.

Additionally, if you have a moment, please do upvote it if the information provided has been helpful. This can be beneficial to other community members and would be greatly appreciated.

Thank you.
Anonymous

2025-06-06T10:36:24.0933333+00:00

Hi @Dev Parmar,

Just want to check if the above answer worked for you or else please let us know if any help, we are always here to help whenever you need us.

Please do not forget to “upvote it” wherever the information provided helps you, this can be beneficial to other community members.it would be greatly appreciated and helpful to others.
Dev Parmar 20 Reputation points

2025-06-12T07:27:04.6566667+00:00
Hi @Anonymous ,
We are currently conducting a comparison between the data retrieved from Azure Activity Logs using the REST API:
GET https://management.azure.com/subscriptions/{subscription_id}/providers/Microsoft.Insights/eventtypes/management/values?api-version=2015-04-01

and the data displayed in Microsoft Sentinel. During this comparison, we have noticed discrepancies in the number of records (event counts).

To better understand and troubleshoot the cause, we are looking for insights into the data ingestion flow for both services:

Where does the Azure Activity Logs API pull data from?

How does Microsoft Sentinel collect or ingest Azure Activity log data?

Are there any filters, delays, or processing layers involved that might lead to differences?

We have observed that the Azure Activity Logs API returns duplicate records. Does Microsoft Sentinel apply any mechanism, such as filtering or deduplication, when exporting or ingesting this data?

If available, a reference diagram or flowchart that shows how each system collects and processes activity log data would be very helpful.

Appreciate any documentation links, architecture diagrams, or firsthand experience you can share.

Thank you!
Anonymous

2025-06-16T07:29:24.85+00:00

Hi @Dev Parmar

Just want to check if the above answer worked for you or else please let us know if any help, we are always here to help whenever you need us.

Please do not forget to “upvote it” wherever the information provided helps you, this can be beneficial to other community members.it would be greatly appreciated and helpful to others.

1 answer

Your answer

Anonymous

2025-06-03T16:19:46.8833333+00:00

Hi @Dev Parmar,

It's common to see the same event appear multiple times with slightly different timestamps. These small differences — often just seconds or minutes — usually stem from the gap between when the event actually occurred and when it was recorded or submitted.

For example, in my lab tests, I observed the following three log entries:

Timestamps:

* Begin request: 2025-06-03T14:57:43.8Z

End request: 2025-06-03T14:57:44.4Z (Accepted)

End request: 2025-06-03T14:58:27.8Z (Succeeded)

Note: These logs were filtered using a unique event ID, ensuring no duplicate entries.

In another scenario you mentioned — where logs share the same eventDataId — the entries actually represent two distinct but related sources of logs:

1. Administrative Log Entry

* Category: Administrative

Operation: Microsoft.Authorization/roleAssignments/write (a user initiated a role assignment)

Caller: ******@gmail.com

Status: Started

Timestamp: 2025-05-28T12:18:18Z

2. Alert Trigger Entry

* Category: Alert

Description: "Alert: Create role assignment called on action groups: it_alert"

Operation: Microsoft.Insights/ActivityLogAlerts/Activated/action (an alert fired based on the above event)

Status: Succeeded

Timestamp: 2025-05-28T12:25:21Z

Here, both logs reference the same eventDataId because the alert was triggered by the original administrative action

Please let me know if you have any questions.
Anonymous

2025-06-05T10:21:20.28+00:00

Hi @Dev Parmar,

I just wanted to check if the above provided information worked for you or if you need any further assistance?

Please feel free to let us know, as we are always here to help whenever you need us.

Additionally, if you have a moment, please do upvote it if the information provided has been helpful. This can be beneficial to other community members and would be greatly appreciated.

Thank you.
Anonymous

2025-06-06T10:36:24.0933333+00:00

Hi @Dev Parmar,

Just want to check if the above answer worked for you or else please let us know if any help, we are always here to help whenever you need us.

Please do not forget to “upvote it” wherever the information provided helps you, this can be beneficial to other community members.it would be greatly appreciated and helpful to others.
Dev Parmar 20 Reputation points

2025-06-12T07:27:04.6566667+00:00

Hi @Anonymous ,
We are currently conducting a comparison between the data retrieved from Azure Activity Logs using the REST API:
GET https://management.azure.com/subscriptions/{subscription_id}/providers/Microsoft.Insights/eventtypes/management/values?api-version=2015-04-01

and the data displayed in Microsoft Sentinel. During this comparison, we have noticed discrepancies in the number of records (event counts).

To better understand and troubleshoot the cause, we are looking for insights into the data ingestion flow for both services:

Where does the Azure Activity Logs API pull data from?

How does Microsoft Sentinel collect or ingest Azure Activity log data?

Are there any filters, delays, or processing layers involved that might lead to differences?

We have observed that the Azure Activity Logs API returns duplicate records. Does Microsoft Sentinel apply any mechanism, such as filtering or deduplication, when exporting or ingesting this data?

If available, a reference diagram or flowchart that shows how each system collects and processes activity log data would be very helpful.

Appreciate any documentation links, architecture diagrams, or firsthand experience you can share.

Thank you!
Anonymous

2025-06-16T07:29:24.85+00:00

Hi @Dev Parmar

Just want to check if the above answer worked for you or else please let us know if any help, we are always here to help whenever you need us.

Please do not forget to “upvote it” wherever the information provided helps you, this can be beneficial to other community members.it would be greatly appreciated and helpful to others.

Answer 1

@Dev Parmar

Where does the Azure Activity Logs API pull data from?

The Azure Activity Logs API pulls data from the Activity Log stored in the Azure Resource Manager (ARM) control plane.

User's image

How does Microsoft Sentinel collect or ingest Azure Activity log data?

Microsoft Sentinel collects or ingests Azure Activity log data using a native connector that pulls logs directly from your Azure subscription.When enabled, Sentinel sets up a diagnostic setting behind the scenes that routes the Activity logs to the Log Analytics workspace that Sentinel is connected to.

reference

https://learn.microsoft.com/en-us/azure/sentinel/connect-services-diagnostic-setting-based

Are there any filters, delays, or processing layers involved that might lead to differences?

Azure Activity Logs can take up to 5–15 minutes to appear in Microsoft Sentinel after an event occurs.

This delay is due to Azure’s internal processing and eventual delivery to the Log Analytics workspace. There is no transformation or data reduction applied by Sentinel — logs are ingested as-is

We have observed that the Azure Activity Logs API returns duplicate records. Does Microsoft Sentinel apply any mechanism, such as filtering or deduplication, when exporting or ingesting this data?

Microsoft Sentinel itself does not automatically deduplicate logs at ingestion. However, deduplication can be implemented at the query level using Kusto Query Language (KQL).

The discrepancy might be due to differences in ingestion, enrichment, and filtering. The Azure Activity Logs REST API gives raw control plane logs, while Sentinel ingests them via Azure Monitor, often with enrichment and possible delays or filtering.

reference:

https://learn.microsoft.com/en-us/azure/sentinel/audit-sentinel-data

Please let me know if you need any further assistance

Please do not forget to "Accept the answer” and “upvote it” wherever the information provided helps you, this can be beneficial to other community members.it would be greatly appreciated and helpful to others.

Share via

Inconsistent Duplicate Events in Azure Activity Logs API

1 answer

Your answer