What is the good design to handle messages in a same process and send it to cloud without a data loss?

Question

I am working on designing a solution for sending data from windows based devices to the cloud

Device Software: The device software is currently written in C, C++ or C# languages based on the device model and legacy build.

IoT Service: This is a proposed service which might run as a windows service and this might use C# language as well as Azure SDK for IoT Hub connectivity.

Now question is, What is the most efficient way to handle a large number of messages from Device Software to IoT Service and successfully send them to the cloud without any data loss? I mean do I have to use any Queue concept or MQTT libraries which help to handle messages and retry logics?

Should a consumer be implemented to retry failed messages or is an Remote procedure callsufficient for this use case?

I have tried my best to explain and draw diagrams. Kindly do not downvote if you can not help with ideas.

This is a real-time use case in this IoTera.

Answer 1

Hello @DEEPAK KUMPALA ,

Your planned architecture is possible and while a few things need more clarity the keyword seems to be "without data loss".
To build your IoT Service in .Net you should consider the Azure IoT Device SDK what provides built-in functionality for resiliency like re-connection and retry, explained in the Manage connectivity and reliable messaging by using Azure IoT Hub device SDKs article. Although these features are useful, if your IoT Service was shut down while offline you would lose all non-processed messages. Therefore, you would need some way to persist the messages and mark them as processed when your device client receives the acknowledge from IoT Hub. As you identified already, one option would be Mosquitto MQTT broker what provides a "persistence" option to write all connection, subscription and message data to disk, see the mosquitto man page and consider settings for persistent_client_expiration, max_queued_bytes and max_queued_messages options.

When designing your solution, you should have clarity about few things

• What is "a large number of messages"?
• What is the min/avg/max message frequency and size?
• What is the accepted/expected delivery time from source to destination when talking about near real-time (NRT)?
• Do you need all messages, or do they get useless after a certain time?
• Is it suitable to upload buffered messages on a different than the hot path, e.g., directly writing it to storage?
• What are the overall uptime goals and how to ensure resiliency for each solution component, great starting point.

---

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

Answer 2

Hello @DEEPAK KUMPALA ,

The answer provided by Christian is indeed correct for an 'direct internet connected device'.

As an alternative, I recommend checking out Azure IoT Edge offline support.

Yes, you are not using an Azure IoT edge solution but you can benefit by adding an IoT edge device into your architecture.

Azure IoT Edge offers persisting d2c messages on disk by default. Messages from modules are stored by the EdgeHub and disposed of if sent to the cloud or expired (TTL). It even offers priorities.

Azure IoT Edge supports the concept of child devices. these child devices are expecting their messages to be sent to the cloud but these are ending up at the edge device first.

These messages are then part of the original routing plan on the edge.

You also benefit by only needing one firewall entry for the edge device if needed.

Of course, if the device can not connect to the edge, messages are not delivered by the edge.

Check out this blog post for an example.