how to maintain multiple streams in single IOT Hub?

kumaraswamy gosukonda 1 Reputation point

i have a situation like, we have single IOT hub which will receive compressed data and uncompressed data. we have single stream analytics service to process IOT hub messages, i have added two inputs in Azure Stream Analytics to process compressed and uncompressed data.

i just wanted to route compressed messages to one input (Stream Analytics) and uncompressed messages to another input to process both stream of data with single instance of IOT Hub and Stream Analytics.

Microsoft Entra ID
Microsoft Entra ID
A Microsoft Entra identity service that provides identity management and access control capabilities. Replaces Azure Active Directory.
20,598 questions
0 comments No comments
{count} votes

2 answers

Sort by: Most helpful
  1. Marilee Turscak-MSFT 36,411 Reputation points Microsoft Employee


    Unfortunately this service is not supported in Q&A yet (slated to be added in May). You will need to ask on MSDN for now.

    I have alerted that team about this thread but they do not appear to have Q&A profiles set up.

    This thread looks somewhat related to your ask, though.

    0 comments No comments

  2. Sander van de Velde | MVP 31,211 Reputation points MVP

    You need to define two separate inputs in your Stream Analytics job.

    Stream Analytics inputs you have to use, are either of type IoT Hub or Event Hub. Both support Event compression, like GZip or Deflate.

    If both the two types of messages were not compressed, you could connect the default, built-in endpoint to multiple inputs, each with their own consumer group and filter within the job.

    Due to the mixed compressed/not compressed telemetry situation, you have to split your IotHub telemetry before handing it over to the stream analytics job.

    This can be done using the IoTHub routing, together with an Eventhub:

    1. Create an eventhub namespace and one eventhub
    2. Create an iot hub custom endpoint and connect it to this eventhub
    3. Disable the fallback route in the IotHub (just to prevent mixing non-filtered messages with the compressed ones)
    4. Create a route with the default built-in endpoint, listening to compressed messages **)
    5. Create a route with the evenhub listening to non-compressed messages **)
    6. In the ASA job create an iothub input for the compressed messages
    7. In the ASA job create a second Eventhub input for the uncompressed messages

    **) You have to use the routing query to split the messages. I recommend using an application property like compressed='gzip' to split the incoming messages over the two routes.

    The Message class has a property bag for that. Splitting using a message body value is hard (not possible?) due to the compressed nature of messages (I expect, only the body is compressed).

    You could also try to accomplish this with two event hubs instead of one iothub endpoint and one eventhub.

    See also this tutorial.

    do not forget to mark the right answer

    (do not forget to mark the right answer)

    0 comments No comments