Ideal Means of Collecting Real Time IoT Sensor data

OMEP IoT Developer 41 Reputation points
2021-03-16T19:15:05.4+00:00

Hello!

Situation: I have a sensor setup, sending data to a gateway device, which is transmitting to Azure. I can access this data with a Stream Analytics job & I see messages coming in using the Azure CLI.

Concern(s): 1) I am seeing a discrepancy between timestamps when viewing transmissions using the CLI vs in the Device twin on azure. 2) This leads me to question whether I am accessing real time data with my stream analytics job when I query the device twin.

Essentially, I would like to determine the best way to query real time data from my sensors in Azure.

Bonus questions:

What then, is the best way to visualize this data in a dashboard?
What is a simple way to store this data in a database on Azure?

Thank you for your time. Please let me know if there's anything I can clarify.

Azure IoT
Azure IoT
A category of Azure services for internet of things devices.
373 questions
Azure Digital Twins
Azure Digital Twins
An Azure platform that is used to create digital representations of real-world things, places, business processes, and people.
214 questions
Azure IoT Hub
Azure IoT Hub
An Azure service that enables bidirectional communication between internet of things (IoT) devices and applications.
1,093 questions
{count} votes

Accepted answer
  1. Sander van de Velde 27,186 Reputation points MVP
    2021-03-16T22:56:17.447+00:00

    Hello @OMEP IoT Developer ,

    If I understand your (first) question correctly, you are looking at the same incoming telemetry both with the CLI and with Stream Analytics.

    Azure Stream Analytics is a very powerful and scalable 'rule engine' that can transform, filter, and group messages based on time window patterns. It can also join with other data streams in-flight.

    You probably have noticed that the Stream analytics job is capable of picking up IoT Hub messages within a second. So this is near-real-time.

    I'm not sure what you mean by "when I query the device twin". The device twin is not the most efficient way to pass on data from edge to cloud. The Devicetwin limited in size and speed. Reading Device twin is also calculated as IoT Hub message consumption.

    Most of the time it's more efficient to put information in the message body.

    The answer to your first bonus question relies on several factors. If you want to store incoming data in a cheap way because there is no value in there at this moment, Azure blob storage is fine. A Data Scientist can check it out later on.

    If you are familiar with SQL Server, SQL Azure is a good choice for persisting data. If you are interested in No-SQL kinds of storage, or you need to distribute data all around the world, CosmosDB is your friend.

    Keep in mind that IoT Devices can generate a lot of messages and therefore rows in tables. So you need to think about the lifetime of the persisted messages. Do you still want to pay for millions of records with data nobody is looking for anymore? So you could remove records after eg. 6 months to keep the size and costs of the database in an optimal condition.

    Regarding the answer to the second bonus question, check out the Stream Analytics output types:

    78443-image.png

    You could write DB access code in an Azure Function but Stream Analytics already has some nice DB support.

    2 people found this answer helpful.

1 additional answer

Sort by: Most helpful
  1. QuantumCache 20,021 Reputation points
    2021-03-29T22:56:36.797+00:00

    Hello @OMEP IoT Developer , This is one of the most helpful questions we see on getting started with IoT projects as well as migrating from another IoT provider. Thanks for posting this question on this community forum. We tried to address each query section, sorry for the too lengthy answer, We hope this helps other community members as with similar queries.

    So let us take Question 1.

    What is a simple way to store this data in a database on Azure?

    1. Hot path where the most recent data that is coming from sensors need to be placed in storage layer as soon as possible. In this example, events are sent via Azure Event Hub and Azure Functions into Azure SQL Database.
    2. Cold path where IoT data is stored on some low-price storage layer, such as Azure Blob Storage. Cold data can be loaded later in some system for analysis to query the data.

    IoT Hub can write messages to multiple endpoints, so with various feature improvements customers can both send the message through a hot-path analytics pipeline, as well as push the message to storage for cold-path analytics or long-term archival. Cold-path analytics are used to process data that requires more complex processing than simple windowing or thresholding, and cold-path analytics often uses data from devices over a longer period of time. With this feature update, you can set up your hot- and cold-path analytics easily in IoT Hub routes.

    82476-image.png

    With an Azure Storage container as a custom endpoint, IoT Hub will write messages to a blob based on the batch frequency and block size specified by the customer. After either the batch size or the batch frequency are hit, whichever happens first, IoT Hub will then write the enqueued messages to the storage container as a blob. You can also specify the naming convention you want to use for your blobs, as shown below.

    82358-image.png

    Let us take a Question 2.

    What then, is the best way to visualize this data in a dashboard?

    Well, once the data is captured as we discussed in the above section, it now the time to visualize the data. Below are some of the tutorials which demonstrate the visualization of real-time sensor data from your Azure IoT hub. Please give it a try and let us know your feedback, we are always happy to help and discuss further in this regard!

    82380-image.png

    82487-image.png

    82534-image.png

    • Also, look at the IoT Central, which is an IoT application platform that reduces the burden and cost of developing, managing, and maintaining enterprise-grade IoT solutions.

    Let us look into the query

    I cannot see the original incoming messages & their data anywhere in the iot hub interface.

    Yes, We can not read the raw messages directly on the Azure IoTHub portal.
    Reason: Azure IoTHub: By default, messages are routed to the built-in service-facing endpoint (messages/events) that is compatible with Event Hubs. IoT Hub exposes the messages/events built-in endpoint for your back-end services to read the device-to-cloud messages received by your hub. This endpoint is Event Hub-compatible, which enables you to use any of the mechanisms the Event Hubs service supports for reading messages.

    let us look at the concern, on Latency

    Concern(s): 1) I am seeing a discrepancy between timestamps when viewing transmissions using the CLI vs in the Device twin on azure. 2) This leads me to question whether I am accessing real time data with my stream analytics job when I query the device twin.

    Latency

    IoT Hub strives to provide low latency for all operations. However, due to network conditions and other unpredictable factors it cannot guarantee a certain latency. When designing your solution, you should:

    • Avoid making any assumptions about the maximum latency of any IoT Hub operation.
    • Provision your IoT hub in the Azure region closest to your devices.
    • Consider using Azure IoT Edge to perform latency-sensitive operations on the device or on a gateway close to the device.
    • Multiple IoT Hub units affect throttling as described previously, but do not provide any additional latency benefits or guarantees.

    **Have a look at the document - IoT Hub quotas and throttling.. For test purpose, try to increase the number of the units. The other option is to change the tier to S2 where the Device connections is guaranteed for 120 devices/sec/unit.

    If you see unexpected increases in operation latency, contact Microsoft Support.

    Ref: Borrowed from Microsoft tech Community & Official blogs.
    https://azure.microsoft.com/en-us/blog/route-iot-device-messages-to-azure-storage-with-azure-iot-hub/
    https://techcommunity.microsoft.com/t5/azure-sql/working-with-azure-iot-data-in-azure-sql-database/ba-p/386175

    Please comment in the below section and let us know your feedback, we are always happy to help and discuss further in this regard!

    1 person found this answer helpful.
    0 comments No comments