How does checkpoints are maintained between Azure Data Explorer and Azure IoT Hub

Question

How does checkpoints are maintained between Azure Data Explorer and Azure IoT Hub

Anonymous

Hi,

How does the check points are being maintained between ADX ad Azure IoT Hub.

Assume, an ADX database is connected to an Azure IoT Hub, for ingestion.

When an ADX is down, for due to some reason and is up after some time, how does ADX knows, from which point(which data) onwards, it should consider?

Regards,

Viswanath

Anonymous

2023-08-16T10:59:48.3033333+00:00

Hi Sander,

Appreciate the prompt reply.

Please let me know, whether my understanding is correct.

Within Azure IoTHub's internal workings, a duplicate of every incoming message is retained for each consumer group.

In case an output linked to a consumer group experiences a failure, the messages will be retained according to the configured timeframe; subsequent to this period, they will be purged from the IoT hub.

Alternatively, if the output has successfully read the message from the consumer group, it will then be erased.

Now, consider a situation where two outputs are connected to the same consumer group, but one is inactive while the other remains functional. In such a scenario, Azure IoT Hub operates to ensure data preservation without any loss.
Anonymous

2023-08-16T11:01:04.6766667+00:00

Hi Sander,

Appreciate the prompt reply.

Please let me know, whether my understanding is correct.

Within Azure IoTHub's internal workings, a duplicate of every incoming message is retained for each consumer group.

In case an output linked to a consumer group experiences a failure, the messages will be retained according to the configured timeframe; subsequent to this period, they will be purged from the IoT hub.

Alternatively, if the output has successfully read the message from the consumer group, it will then be erased.

Now, consider a situation where two outputs are connected to the same consumer group, but one is inactive while the other remains functional. In such a scenario, Azure IoT Hub operates to ensure data preservation without any loss.

Answer accepted by question author

0 additional answers

Your answer

Anonymous

2023-08-16T10:59:48.3033333+00:00

Hi Sander,

Appreciate the prompt reply.

Please let me know, whether my understanding is correct.

Within Azure IoTHub's internal workings, a duplicate of every incoming message is retained for each consumer group.

In case an output linked to a consumer group experiences a failure, the messages will be retained according to the configured timeframe; subsequent to this period, they will be purged from the IoT hub.

Alternatively, if the output has successfully read the message from the consumer group, it will then be erased.

Now, consider a situation where two outputs are connected to the same consumer group, but one is inactive while the other remains functional. In such a scenario, Azure IoT Hub operates to ensure data preservation without any loss.
Anonymous

2023-08-16T11:01:04.6766667+00:00

Hi Sander,

Appreciate the prompt reply.

Please let me know, whether my understanding is correct.

Within Azure IoTHub's internal workings, a duplicate of every incoming message is retained for each consumer group.

In case an output linked to a consumer group experiences a failure, the messages will be retained according to the configured timeframe; subsequent to this period, they will be purged from the IoT hub.

Alternatively, if the output has successfully read the message from the consumer group, it will then be erased.

Now, consider a situation where two outputs are connected to the same consumer group, but one is inactive while the other remains functional. In such a scenario, Azure IoT Hub operates to ensure data preservation without any loss.

Answer 1

Sander van de Velde | MVP 37,066 MVP

Hello @Peter Prabhat,

the answer is quite simple due to the way the ADX Database Data connection is created.

The ADX Database Data connection is relying on the message stream coming from the IoT Hub.

This is the Event Hub compatible endpoint of the IoT Hub, probably enforced with the default event route.

This endpoint is also instrumented with a separate consumer group (other than $default) so there is a copy for Azure Data Explorer for each incoming message.

If ADX stops, the data connection stops and ingestion stops.

The IoT Hub has a certain retention time for messages not being picked up:

User's image

So, depending on the retention time, you have one or more days to stop and restart the Azure Data Explorer without information loss.

Check out more details here.

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

Anonymous

2023-08-16T11:01:57.16+00:00

Hi Sander,

Appreciate the prompt reply.

Please let me know, whether my understanding is correct.

Within Azure IoTHub's internal workings, a duplicate of every incoming message is retained for each consumer group.

In case an output linked to a consumer group experiences a failure, the messages will be retained according to the configured timeframe; subsequent to this period, they will be purged from the IoT hub.

Alternatively, if the output has successfully read the message from the consumer group, it will then be erased.

Now, consider a situation where two outputs are connected to the same consumer group, but one is inactive while the other remains functional. In such a scenario, Azure IoT Hub operates to ensure data preservation without any loss.
Sander van de Velde | MVP 37,066 Reputation points MVP

2023-08-16T13:25:38.9066667+00:00

Hello Peter,

Yes, each consumer group gets its own copy of a message:

It enables multiple consuming applications to read the same streaming data in an event hub independently at their own pace with their offsets.

Do not use the same consumer group multiple times; Just create more consumer groups!

We recommend that there's only one active receiver on a partition within a consumer group.

Check the documentation for the effect you experience if you reuse resource groups.
Anonymous

2023-08-16T13:28:08.65+00:00

Thanks Sander

Share via

How does checkpoints are maintained between Azure Data Explorer and Azure IoT Hub

0 additional answers

Your answer