Capture Event Hub without duplication

Question

What is the best way to capture event hubs without duplication. Since event hubs delivery at least once. We can have the same event multiple times.
As you can imagine I don't want to store the same event hub twice. I just want to store each event hub once.

What is the best way to do this?

Answer 1

hi @Davi Sena

Regarding duplication, there is no solution that will guarantee that because of the concept of at-least once delivery:

Better read the official documentation to understand why:
https://learn.microsoft.com/en-us/azure/architecture/serverless/event-hubs-functions/resilient-design#duplicate-events
https://learn.microsoft.com/en-us/azure/architecture/serverless/event-hubs-functions/resilient-design#deduplication-techniques
https://learn.microsoft.com/en-us/azure/azure-functions/functions-idempotent

You can tune it up to reduce the chances of duplication. As this article mentions, you can reduce batch size, azure function parallelism, but those actions will slow down performance and still not guarantee the end of duplications

https://medium.com/@jeffhollan/in-order-event-processing-with-azure-functions-bb661eb55428

in other words, it's impossible to avoid duplications 100% and changing settings will compromise performance. the best approach is to have a field with some unique value like a pk or correlation id and add code to detect duplicates and prevent it from happening

if your scenario is to complicated to implement a duplication detection and you don't need to process a massive amount of messages in a short period of time , check azure service bus : https://learn.microsoft.com/en-us/azure/service-bus-messaging/duplicate-detection

Hope this helps!