Share via

Is it possible to create an Id field for idempotency in a Log Analytics custom log table?

Darren 1 Reputation point
2023-08-01T12:35:16.0266667+00:00

Is it possible to create a unique Id field in a custom log table that would enforce idempotency?

I am using the LogAnalytics.Client library to push data into a custom log and need to be sure there is no duplicate data being sent in the batches. I have a unique I field which is populated with a Guid, I was looking to see if I could set that as some unique or primary key on the table.

Azure Monitor
Azure Monitor
An Azure service that is used to collect, analyze, and act on telemetry data from Azure and on-premises environments.
3,037 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Ben Gimblett 3,845 Reputation points Microsoft Employee
    2023-08-08T08:53:55.4266667+00:00

    Hey - Thanks for the question

    The only thing that I could think of was possibly using Data Collection Rules (DCRs) however whilst this offers transformation it's row by row (not even on an incoming batch) . Also the KQL available for transformations is limited at this time

    The thing is with Log Analytics is that it's engineered for fast inserts and fast reads - it isn't like a regular referential / tabular database

    You could de-duplicate on read using KQL (e.g. distinct) based on field(s) that have values individually or together that provide uniqueness but it wouldn't actually be removing the duplicates it's a view on the existing data

    0 comments