Azure IoT Hub/IoT Central messages using multiple time stamps

Andirly 16

Hello,

I'm going to have the following scenario where I'm ingesting data every 5s into IoT Central (IoT Hub). The containing data block consists of multiple data points of which each is supposed to have their own time stamp.

Is there a certain format I have to follow so IoT Central is correctly interpreting the individually time-stamped telemetry in the block? If so, can someone point that out to me.

Thank you in advance.

Andy

AshokPeddakotla-MSFT 30,241 Reputation points

2022-06-06T12:38:00.833+00:00

@Andirly Wanted to follow up with you to understand whether the below information helped. Please let us know if there are any other queries.
If below suggestions answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.
Roman Kiss 2,246 Reputation points

2022-06-07T07:59:14.297+00:00

Hi Andy,

have a look at the issue: Send telemetry data into batches.

also, please make a vote for this feature here.

Thanks
Roman

1 answer

Sander van de Velde | MVP 31,116 Reputation points MVP

2022-05-19T12:12:31.287+00:00

Hello @Andirly ,

you are sending a custom message containing multiple 'sub' messages to the cloud.

Compared to the IoT Hub offers IoT Central some additional mapping but I'm not sure if this can be used to 'split' your message.

The IoT Hub handles an incoming message as-is so if you want to split your message into multiple messages, you need to add extra logic after the IoT Hub (for example, Azure Stream Analytics is a great solution for this).

Next to that, the Azure IoT Hub offers message batching using the Azure IoT Device SDKs (features can differ per programming language eg. C#).

Then two or more messages can be batched on the device and sent to the IoT Hub. This will slow down the ingest (only when the required batch is collected, the application will send the messages) but makes more efficient use of the number of daily messages supported by your hub.

The IoT HUb will do the splitting of the individual messages as part of this batching solution.
Please sign in to rate this answer.
Andirly 16 Reputation points

2022-05-19T12:38:15.723+00:00

Thanks for your reply. Batching seems an interesting solution.

Then two or more messages can be batched on the device and sent to the IoT Hub. This will slow down the ingest (only when the required batch is collected, the application will send the messages) but makes more efficient use of the number of daily messages supported by your hub.

You're suggesting that IoT Hub is treating batched messages not as individual messages but as messages of Batch size/4kB (max message size). The answer (iot-edge-sendeventbatchasync) on StackOverflow claims that even if batched IoT Hub handles these messages as individual messages which would significantly reduce my Hub quota.

Regarding a solution using Stream Analytics: Is it possible to set up a Stream Analytics job which handles data before further data is further processed by an IoT Central application?

Looking here, at least AMQP seems to support batching and treating messages as one. azure-iot-sdk-csharp

Sander van de Velde | MVP 31,116 Reputation points MVP

2022-05-19T15:38:42.063+00:00

Hello @Andirly ,

A message send using the Batch option will be split into separate messages on a functional level. From a user perspective, the user is unaware the messages were batched, and individual messages arrive.

Technically, the message will be a bigger message (more bytes). An IoT Hub counts the number of 4Kb chunks. So, the consumption of the daily amount of chunks (very confusing but these are called messages too by the IoT Hub) can drastically drop based on your choices (because you statistically fill more chucks up to the max thus fewer chunks are needed).

The architecture of IoT Hub -> ASA -> IoT Central seems over complicated.

If you still want to work with your own logic, check out the IoT Central bridge. This (sample) Azure Function is demonstrating how 3rd party cloud messages are ingested and sent to IoT Central. It's based on an HTTP trigger but you should be able to alter this if needed.

The code is not that complex and that way perfect for integration purposes.

----------

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. By doing so, all community members who have a similar issue will benefit. Your contribution is highly appreciated.

Andirly 16 Reputation points

2022-05-19T19:48:28.703+00:00

Thank you for the clarification.

I modified my C# script to test the # of consumed messages when using SendBatchAsync via AMQP and the IoT Hub tells me just random message counts. I sent 10 messages of size 38Bytes each which should be shown as one message in IoT Hub? (I'm using our free tier Hub for testing which grants 0.5kB Message size, should be enough anyway)

Maybe the more important question is: Will IoT Central count a batch message as single message or multiple? That would definitely be not economic for our purpose. We'd end up sending millions of messages per month.

Edit: One of the diagrams in the IoT Hub overview seems to show the right count (batch vs single messages):

Sander van de Velde | MVP 31,116 Reputation points MVP

2022-05-19T20:37:33.483+00:00

Hello @Andirly ,

regarding the IoT Hub, it seems to count as expected.

Regarding IoT Central, if you are able to send messages directly to IoT Central using the batched solution, it will probably work the same as the IoT Hub (only the chunks are counted).

If you want to use the bridge, you need to batch it yourself.

The bridge is written in NodeJS so this method has to change:

await client.sendEvent(message);

I'm not sure if batching is supported by the NodeJS device SDK though.

I recommend performing a live test with a limited number of devices. Then you know the impact. This is the proper procedure of every well architected framework.
Sign in to comment

Share via

Azure IoT Hub/IoT Central messages using multiple time stamps

1 answer