We are using DLQ functionality for ADT Event routes. This can easily be set up and there is no issues. We are also using Event historization (that creates, we assume, an event route in the background) that is going over a different EventHub and into ADX. Documentation states: " Azure Digital Twins data history builds on the existing ingestion mechanism provided by Azure Data Explorer." The question are: What happens in case of ADX or any other type of failure for historical connection? Is there some built in DLQ and how exactly the historization mechanism works? What exactly is the guarantee that event produced by ADT will be delivered to ADX? Is it at-least-once? Can there be data loss?

Hello @Bernard Pletikosa , Azure Data Explorer uses the concept of database data connections for ingesting event data. One of the connections is for Event Hubs : Check this blog post for a background story. Regarding the guarantee, this is based on the standard behavior of the used tooling like the EventHub. If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

DLQ functionality for data historization events

Accepted answer

Sander van de Velde 28,386 Reputation points MVP

2023-07-12T17:13:49.01+00:00

Hello @Bernard Pletikosa ,

Azure Data Explorer uses the concept of database data connections for ingesting event data.

One of the connections is for Event Hubs:

Check this blog post for a background story.

Regarding the guarantee, this is based on the standard behavior of the used tooling like the EventHub.

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.
Please sign in to rate this answer.
Bernard Pletikosa 105 Reputation points

2023-07-13T12:34:38.32+00:00

Hi @Sander van de Velde , thanks for the quick answer.

I'm aware that guarantees build on each other but the question is what is the result. We are also using exactly this data path and ADT+ADX queries

Namely, in the step where DT property changes are put on EventHub. What happens if EventHub is not available for prolonged time (let's say couple of hours) because of some infrastructure problems. Would DT events be:

stored somewhere and wait for EHub availability (this is not mentioned in the docs)?

stored in some external DLQ (not aware that this exists)?

dropped after some time?

The main question is can some business critical data (like events about assets being onboarded) be lost or there is at-least-once guarantee?

Sander van de Velde 28,386 Reputation points MVP

2023-07-13T14:40:04.38+00:00

Hello @Bernard Pletikosa ,

I understand your concern.

Let me check with someone within Microsoft to answer this question.

Sander van de Velde 28,386 Reputation points MVP

2023-07-14T09:54:47.72+00:00

Hello @Bernard Pletikosa ,

check out this documentation:

Once the connection is set up, Azure Digital Twins implements at least once delivery for data emitted to egress services.

If the Event hub is unavailable, ADT will try to send the message for up to a day.

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

Bernard Pletikosa 105 Reputation points

2023-07-14T13:29:29.74+00:00

Thanks you very much @Sander van de Velde !

That's exactly what I was wondering, at-least-once I've heard but couldn't find in the docs.
I assume that 1 day timeframe is coming from the product team and would be super useful piece of information in the documentation

Sander van de Velde 28,386 Reputation points MVP

2023-07-14T14:19:49.8833333+00:00

Hello @Bernard Pletikosa ,

yes, I have asked the product team to mention the 1-day timespan too.
Sign in to comment

DLQ functionality for data historization events

Regarding the guarantee, this is based on the standard behavior of the used tooling like the EventHub.

0 additional answers