Share via

IoT Hub lose data

Anonymous
2023-07-13T03:04:56.2333333+00:00

I'm worried that IoTHUB will lose data.

Which is better in terms of data loss, adding or not adding Kafka, IoTHUB or Functions?

And why?image

Azure IoT Hub
Azure IoT Hub

An Azure service that enables bidirectional communication between internet of things (IoT) devices and applications.

0 comments

Answer accepted by question author

  1. Sander van de Velde | MVP 37,061 Reputation points MVP
    2023-07-13T11:32:15.2766667+00:00

    Hello @Anonymous ,

    The Azure IoT Hub is a perfect (public) cloud gateway to connect devices to for D2C messages if you want to have every device using its own credentials. The IoT Hub also offers C2D communication. Another important feature is conditional routing.

    The Azure IoT Hub guarantees at least once delivery of incoming messages:

    IoT Hub implements at least once delivery guarantees for both device-to-cloud and cloud-to-device messaging.

    The Azure IoT Hub guarantees delivery messages to all its endpoints, without duplicates through duplicate routing:

    Each message is routed to all endpoints whose routing queries it matches. In other words, a message can be routed to multiple endpoints. If a message matches multiple routes that point to the same endpoint, IoT Hub delivers the message to that endpoint only once.

    According to this, the reliability is shifted towards the receiving solution.

    I'm not sure what the need is for an (extra) Kafka resource.

    It's a common pattern to use IoTHub -> Azure Function -> DB

    You could add an extra EventHub (IoTHub -> Event Hub -> Azure Function -> DB) which gives you more control eg:

    • listen to the event hub to see what messages are passed to the function
    • Temporarily stop the egress of the event hub for whatever reason
    • Adding archive functionality to the Event Hub

    If the Azure Function is stopped, the ingest will pause. Next time the Azure Function starts, it will proceed with the next, not yet received, message.

    Please try out these patterns first en see what fits best to your situation.

    If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

    Was this answer helpful?

    1 person found this answer helpful.
    0 comments

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Sedat SALMAN 14,530 Reputation points MVP
    2023-07-13T10:15:44.4133333+00:00

    In distributed systems, data loss is generally prevented by having redundancy, reliable delivery mechanisms, and robust error and retry policies in place. But for your architecture at first you need know these tools capabilities

    1. IoT Hub has built-in support for data persistence, so if a message fails to be delivered, it will retain the message for a period of time. (if this is ok for you
    2. Apache Kafka, when used as an Event Hubs, provides high-throughput, distributed messaging system. Kafka includes a commit log that can store data for a configurable amount of time to handle data loss. If a consumer fails, it can read back from the log at its own pace, providing a buffer against data loss.

    Also you can check the following link you can see the differences

    https://learn.microsoft.com/en-us/azure/event-hubs/azure-event-hubs-kafka-overview

    If the differences of Kafka is important for you you can add otherwise there is no need to add Kafka to your architecture

    Was this answer helpful?

    1 person found this answer helpful.
    0 comments

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.