Hello @OMEP IoT Developer , This is one of the most helpful questions we see on getting started with IoT projects as well as migrating from another IoT provider. Thanks for posting this question on this community forum. We tried to address each query section, sorry for the too lengthy answer, We hope this helps other community members as with similar queries.
So let us take Question 1.
What is a simple way to store this data in a database on Azure?
- Hot path where the most recent data that is coming from sensors need to be placed in storage layer as soon as possible. In this example, events are sent via Azure Event Hub and Azure Functions into Azure SQL Database.
- Cold path where IoT data is stored on some low-price storage layer, such as Azure Blob Storage. Cold data can be loaded later in some system for analysis to query the data.
IoT Hub can write messages to multiple endpoints, so with various feature improvements customers can both send the message through a hot-path analytics pipeline, as well as push the message to storage for cold-path analytics or long-term archival. Cold-path analytics are used to process data that requires more complex processing than simple windowing or thresholding, and cold-path analytics often uses data from devices over a longer period of time. With this feature update, you can set up your hot- and cold-path analytics easily in IoT Hub routes.
With an Azure Storage container as a custom endpoint, IoT Hub will write messages to a blob based on the batch frequency and block size specified by the customer. After either the batch size or the batch frequency are hit, whichever happens first, IoT Hub will then write the enqueued messages to the storage container as a blob. You can also specify the naming convention you want to use for your blobs, as shown below.
Let us take a Question 2.
What then, is the best way to visualize this data in a dashboard?
Well, once the data is captured as we discussed in the above section, it now the time to visualize the data. Below are some of the tutorials which demonstrate the visualization of real-time sensor data from your Azure IoT hub. Please give it a try and let us know your feedback, we are always happy to help and discuss further in this regard!
- Also, look at the IoT Central, which is an IoT application platform that reduces the burden and cost of developing, managing, and maintaining enterprise-grade IoT solutions.
Let us look into the query
I cannot see the original incoming messages & their data anywhere in the iot hub interface.
Yes, We can not read the raw messages directly on the Azure IoTHub portal.
Reason: Azure IoTHub: By default, messages are routed to the built-in service-facing endpoint (messages/events) that is compatible with Event Hubs. IoT Hub exposes the messages/events built-in endpoint for your back-end services to read the device-to-cloud messages received by your hub. This endpoint is Event Hub-compatible, which enables you to use any of the mechanisms the Event Hubs service supports for reading messages.
let us look at the concern, on Latency
Concern(s): 1) I am seeing a discrepancy between timestamps when viewing transmissions using the CLI vs in the Device twin on azure. 2) This leads me to question whether I am accessing real time data with my stream analytics job when I query the device twin.
Latency
IoT Hub strives to provide low latency for all operations. However, due to network conditions and other unpredictable factors it cannot guarantee a certain latency. When designing your solution, you should:
- Avoid making any assumptions about the maximum latency of any IoT Hub operation.
- Provision your IoT hub in the Azure region closest to your devices.
- Consider using Azure IoT Edge to perform latency-sensitive operations on the device or on a gateway close to the device.
- Multiple IoT Hub units affect throttling as described previously, but do not provide any additional latency benefits or guarantees.
**Have a look at the document - IoT Hub quotas and throttling.. For test purpose, try to increase the number of the units. The other option is to change the tier to S2 where the Device connections is guaranteed for 120 devices/sec/unit.
If you see unexpected increases in operation latency, contact Microsoft Support.
Ref: Borrowed from Microsoft tech Community & Official blogs.
https://azure.microsoft.com/en-us/blog/route-iot-device-messages-to-azure-storage-with-azure-iot-hub/
https://techcommunity.microsoft.com/t5/azure-sql/working-with-azure-iot-data-in-azure-sql-database/ba-p/386175
Please comment in the below section and let us know your feedback, we are always happy to help and discuss further in this regard!