Log Analytics - HTTP Data Collector API – Prevent uploading duplicates

Question

I want to upload custom data Log Analytics using the HTTP Data Collector-API (which is in preview).
I am using PowerShell to upload the data. I used the sample on https://learn.microsoft.com/en-us/azure/azure-monitor/logs/data-collector-api

I want to query a Rest Api (Cisco Meraki) and send this to Log Analytics. This works as expected
I also want to prevent duplicates in Log Analytics. I thought that Timegenerated is used as primary key. How do I prevent duplicates?

# Replace with your Workspace ID  
$CustomerId = ""    
  
# Replace with your Primary Key  
$SharedKey = ""  
  
# Specify the name of the record type that you'll be creating  
$LogType = "Custom"  
  
# You can use an optional field to specify the timestamp from the data. If the time field is not specified, Azure Monitor assumes the time is the message ingestion time  
$TimeStampField = "ts"  
  
$Array = @()  
$Array+= New-Object -type psobject -Property @{  
"ts"             = "2021-09-28T10:13:18.676095Z"  
"eventType"      = "IDS Alert"  
"deviceMac"      = "ac:17:c8:24:f5:70"  
"clientMac"      = "d0:7e:35:17:6b:be"  
"srcIp"          = "88.221.144.104:80"  
"destIp"         = "172.28.0.163:54305"  
"protocol"       = "tcp/ip"  
"priority"       = "1"  
"classification" = "9"  
"blocked"        = "True"  
"message"        = "BROWSER-IE Microsoft Edge App-v vbs command attempt"  
"signature"      = "1:48053:1"  
"sigSource"      = "ids-vrt-balanced"  
"ruleId"         = "meraki:intrusion/snort/GID/1/SID/48053"  
}  
$Array+= New-Object -type psobject -Property @{  
"ts"             = "2021-09-28T10:13:21.872426Z"  
"eventType"      = "IDS Alert"  
"deviceMac"      = "ac:17:c8:24:f5:70"  
"clientMac"      = "d0:7e:35:17:6b:be"  
"srcIp"          = "88.221.144.104:80"  
"destIp"         = "172.28.0.163:54308"  
"protocol"       = "tcp/ip"  
"priority"       = "1"  
"classification" = "9"  
"blocked"        = "True"  
"message"        = "BROWSER-IE Microsoft Edge App-v vbs command attempt"  
"signature"      = "1:48053:1"  
"sigSource"      = "ids-vrt-balanced"  
"ruleId"         = "meraki:intrusion/snort/GID/1/SID/48053}"  
}  
$json = $Array | ConvertTo-Json  
  
# Create the function to create the authorization signature  
Function Build-Signature ($customerId, $sharedKey, $date, $contentLength, $method, $contentType, $resource)  
{  
    $xHeaders = "x-ms-date:" + $date  
    $stringToHash = $method + "`n" + $contentLength + "`n" + $contentType + "`n" + $xHeaders + "`n" + $resource  
  
    $bytesToHash = [Text.Encoding]::UTF8.GetBytes($stringToHash)  
    $keyBytes = [Convert]::FromBase64String($sharedKey)  
  
    $sha256 = New-Object System.Security.Cryptography.HMACSHA256  
    $sha256.Key = $keyBytes  
    $calculatedHash = $sha256.ComputeHash($bytesToHash)  
    $encodedHash = [Convert]::ToBase64String($calculatedHash)  
    $authorization = 'SharedKey {0}:{1}' -f $customerId,$encodedHash  
    return $authorization  
}  
  
  
# Create the function to create and post the request  
Function Post-LogAnalyticsData($customerId, $sharedKey, $body, $logType)  
{  
    $method = "POST"  
    $contentType = "application/json"  
    $resource = "/api/logs"  
    $rfc1123date = [DateTime]::UtcNow.ToString("r")  
    $contentLength = $body.Length  
    $signature = Build-Signature `  
        -customerId $customerId `  
        -sharedKey $sharedKey `  
        -date $rfc1123date `  
        -contentLength $contentLength `  
        -method $method `  
        -contentType $contentType `  
        -resource $resource  
    $uri = "https://" + $customerId + ".ods.opinsights.azure.com" + $resource + "?api-version=2016-04-01"  
  
    $headers = @{  
        "Authorization" = $signature;  
        "Log-Type" = $logType;  
        "x-ms-date" = $rfc1123date;  
        "time-generated-field" = $TimeStampField;  
    }  
    Write-Host $uri  
    $response = Invoke-WebRequest -Uri $uri -Method $method -ContentType $contentType -Headers $headers -Body $body -UseBasicParsing  
    return $response.StatusCode  
  
}  
  
# Submit the data to the API endpoint  
Post-LogAnalyticsData -customerId $customerId -sharedKey $sharedKey -body ([System.Text.Encoding]::UTF8.GetBytes($json)) -logType $logType    

Answer 1

Hi,
The API does not offer such functionality. TimeGenerated cannot be primary key as you can send the same data at the same exact time. One option is to query the data and compare it before sending it but of course that could result in a lot of problems. Other option is to write your mechanism in a way that you store the data locally and when you send it to clear it from that local place. Something like cache mechanism. In general this is for example how the agent woks.

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

Answer 2

Thank you stan.

This is not the answer I was hoping for but at least its clear.