Store MQTT messages IoT Hub

Question

Store MQTT messages IoT Hub

Andrea Previtali 106

Hi all,

I'm moving my first steps into Azure world.

I'm trying to edit some of samples provided by Microsoft documentation.
I'm able to send MQTT messages to IoT Hub from different platforms.

Now I would to store this messages into a table.
In messages routing i created a rule to direct messages to a storage account.
Is this the correct way to proceed? Create a storage account and route messages to it?
I don't understand exactly what I have to do, do I have to create a function to store this messages?

Andrea

1 answer

Your answer

Answer 1

Matthijs van der Veer 4,376 MVP Volunteer Moderator

You're on the right path! If you want to store the messages in blob storage, you can follow this tutorial. It will store the messages in your Azure Storage Account. This is a built-in feature of IoT Hub.

However, you also mentioned:

Now I would to store this messages into a table

If by that you mean that you want to use table storage (available in CosmosDB or Storage Accounts), you will need to take a different approach. One such approach could be to use an Azure Function to listen to the IoT Hub built-in endpoint and use the CosmosDB/Storage Account SDK to put the messages in table storage.

Andrea Previtali 106 Reputation points

2022-06-06T10:51:26.893+00:00

Hi @Matthijs van der Veer !

Thanks for answering me and sorry if maybe my question was not clear.
I'm able to store messages in blob storage and also to read them with no particoular touble.

Now, as you suggested, I would like to use table storage. My final goal is to realize a """realtime""" telemetry application so I need to read the data very fast (once per second ?! ), I guess Table Storage is the right choice.
According to your experience, what's the best way to proceed? Can I continue to use my Storage Account or do you suggest switching to CosmosDB?
Do I have to implement special function or there is something like a "pre-built" solution which allows me to store directly my messages into table storage?

Thanks for your help,

Andrea
Matthijs van der Veer 4,376 Reputation points MVP Volunteer Moderator

2022-06-07T07:00:25.833+00:00

If you want to achieve something in real-time, you might consider not reading from storage/database every second but instead ingesting the data as a stream. There are many different options to consider, and without more details about your application, there are no recommendations to be made. You can look into Event Hubs or Service Bus if you're reading the data in a cloud application or perhaps even a desktop application. A good choice for a web app would be to push the telemetry to SignalR. I would avoid storing and reading to storage on a high frequency.

You probably still want to store the data to be able to refer to it later. For a breakdown between Azure Storage Tables and CosmosDB tables, check this link. There is no pre-built solution to store telemetry in table storage, so you must build your own. One way to achieve this would be to create an Azure Function with an Event Hub (IoT Hub) input binding and an Azure Storage output binding.

In IoT architectures, we usually make a distinction between hot/warm/cold data storage. If you're building a (near) real-time application you're definitely dealing with data on the hot path (stored briefly, for quick querying), but blob storage could still be a valuable option for your cold storage. There's a ton to say about these architectural decisions, a good place to start is here.
Andrea Previtali 106 Reputation points

2022-06-08T08:33:27.77+00:00

Thanks for your consderation.

I guess the problem with SignalR is that it uses HTTP, I'm going to use MQTT.

Very shortly:
I'm trying to realizing a king of telemetry for our devices. The tipical use case is a customers who have some issue with the device and call our assistance service.
Our technician asks the customer to perform certain operations on the device and remotely checks that those operations are performed. This is why my goal is to implement a realtime system.
During the operations performed by the customer we would be able to check for any anomalies in the machinery.

What I am interested in is sending a string containing this information, it will be up to the webapp to make use of this data, interpret it, etc...

I chose MQTT since i read it is the "standard de facto" for IoT.

I will take a look on what you linked in your comment and I will be back.

Thanks you,

Andrea
Andrea Previtali 106 Reputation points

2022-06-10T13:57:19.643+00:00

Hi all,

Keep trying different strategies to store my messages into Storage Table.

I'm trying to store received messages into Table Storage using function and also using stream analytics.

I think everything is well configured but it still doesn't work.
I feel a bit silly but I'm not experienced with this stuff and surely there is something stupid I'm doing wrong.

For example using stream analytics I can see the following error:
Could not deserialize the input event(s) from resource 'Partition: [1], Offset: [8589973744], SequenceNumber: [228], DeviceId: [2B384451]' as Json

I'm still using the sample obtained from Azure documentation to send messages to my IoT Hub( https://learn.microsoft.com/it-it/azure/iot-hub/iot-hub-mqtt-support).

The project I'm working on seems to be very simple, I guess finding a solution to this issue will keep me very close to the "end".

Andrea

Share via

Store MQTT messages IoT Hub

1 answer

Your answer