Is there a better way to stream data?

Question

Hello Support,

We currently have a system where there are 10 devices per iot hub, and each of those devices are routed to their own event hub. From there they are ingested and streamed in ADX, each having there own table in the database. I tried using stream analytics to filter, but the watermark delay is too high and defeats the purpose of streaming. Is there a better way to do this? Maybe another way to filter by device id?

Thank you for your support,

Nathan

Answer 1

Hello @Nathan Sommer ,

welcome to this moderated Azure community forum.

The obvious way would be enriching the messages in the IoT Hubs while being routed to the EventHub.

You can add the device identifier as enrichment if you repeat it eg. in the tags of a device.

When messages are enriched, the key/value pairs are added to the application settings, outside the body.

In the past, the Azure Data Explorer data connection did not understand/support application settings. Azure Stream Analytics could fix this but that is out of scope.

I noticed this line in the documentation regarding application properties:

• In Event Hubs, when creating the event data object, embed the custom properties as part of the data in the body section of the object.

I am not sure what this means... I think it references the EventData class and askes you to reprogram the logic to discard application properties...

Another solution is ingesting the IoT Hub streaming events directly in Azure Data Explorer via the Database data connection for IoT Hub.

Then you have access to the system properties where the 'iothub-connection-device-id' is available.

These can be referenced in the mapping and added to the table.

As an alternative, you could check the system properties of the Database data connection for EventHub. This one could supports the 'iothub-connection-device-id' to.

Please let us know which solution works for you.

---

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

Answer 2

Hi Nathan,

I think you have several possibilities, here are some that you can try:

• Dynamic routing of IoT Hub telemetry
  https://sandervandevelde.wordpress.com/2022/05/13/dynamic-routing-of-iot-hub-telemetry-to-azure-data-explorer/
• Using ADX table update policies
  https://sandervandevelde.wordpress.com/2022/07/29/using-adx-table-update-policies-projecting-raw-messages-to-target-tables/

Kind Regards,

Wilko

---

Please do not forget to "Accept the answer” wherever the information provided helps you, this can be beneficial to other community members. If you have extra questions about this answer, please click "Comment".