How to handle data coming on IOT Hub which has multiple JSON formats?

Question

Hello,
I am working on a use case wherein I am sending data from az3166 iot device to IOT hub. Further I am running a stream analytics query to store data onto blob storage.
The json for the same is
{"messageId":1,"temperature":28.399999618530273,"humidity":45.799999237060547,"EventProcessedUtcTime":"2021-04-27T06:30:10.6759169Z","PartitionId":3,"EventEnqueuedUtcTime":"2021-04-27T06:28:55.0330000Z","IoTHub":{"MessageId":null,"CorrelationId":null,"ConnectionDeviceId":"az3166","ConnectionDeviceGenerationId":"637540017300317851","EnqueuedTime":"2021-04-27T06:28:55.0290000Z"}}

Also , I am Simulating OPC UA server and OPC publisher on IOT Edge and it is sending data to IOT hub in following format.
{"NodeId":"http://www.prosysopc.com/OPCUA/SimulationNodes/#i=1004","ApplicationUri":"urn:host.docker.internal:OPCUA:SimulationServer","DisplayName":"Temperature","Value":{"Value":1.902113,"SourceTimestamp":"2021-04-28T10:47:06Z"},"EventProcessedUtcTime":"2021-04-28T10:48:27.0652993Z","PartitionId":1,"EventEnqueuedUtcTime":"2021-04-28T10:47:15.5030000Z","IoTHub":{"MessageId":null,"CorrelationId":null,"ConnectionDeviceId":"opauavm","ConnectionDeviceGenerationId":"637551916393812820","EnqueuedTime":"2021-04-28T10:47:15.5110000Z"}}

Now if we look at both the json file it is structured differently. I want to write a single Stream Analytics Job that will help me store both the json in blob storage.

PS: I am looking to store Temperature and Pressure parameters from both az3166 and OPC Publisher module.

Hi @Sander van de Velde . Needed help in this.

Answer 1

Hello @Anurag Shelar ,

So multiple types of messages arrive at the IoT Hub and you want to split messages using routing.

There are two main solutions:

1. Routing on the IoT Hub
2. Routing on Azure Stream analytics

Regarding Routing on the IoT Hub, I prefer to do this in a non-functional way. You can add message routes using the content of desired properties but I like to keep it simple and only split on tags, desired properties or the application properties send with the message as context of why the message is sent.

Azure Stream Analytics is great for splitting, and joining, messages based on the content.

In ASA, a split between multiple messages can be as simple as testing for a JSON element only existing in one of the messages.

Check out the ASA query language and IoT Hub routing.