Edit

Parse Protobuf in Azure Stream Analytics

  • Article

Azure Stream Analytics supports processing events in Protocol Buffer (Protobuf) data formats. You can use the built-in Protobuf deserializer when configuring your inputs. To use the built-in deserializer, specify the Protobuf definition file, message type, and prefix style.

Note

Azure Stream Analytics does not support schema registry for Protocol Buffer (Protobuf) data formats.

Steps to configure a Stream Analytics job

To configure your Stream Analytics job to deserialize events in Protobuf:

  1. After you create your Stream Analytics job, select Inputs.

  2. Select Add input, and then select what input you want to configure to open the pane for input configuration.

  3. Select Event serialization format to show a dropdown list, and then select Protobuf (preview).

    Screenshot that shows selections for configuring Protobuf for an Azure Stream Analytics job.

  4. Complete the configuration by using the following guidance:

    Property name Description
    Protobuf definition file A file that specifies the structure and data types of your Protobuf events
    Message type The message type that you want to deserialize
    Prefix style The setting that determines how long a message is to deserialize Protobuf events correctly

    Screenshot that shows the input boxes on the configuration pane for an Azure Stream Analytics job, after you select Protobuf as the event serialization format.

To learn more about Protobuf data types, see the official Protocol Buffers documentation.

Limitations

  • The Protobuf deserializer takes only one Protobuf definition file at a time. Imports to custom-made Protobuf definition files aren't supported. For example:

    Screenshot that shows an example of a custom-made Protobuf definition file.

    This Protobuf definition file refers to another Protobuf definition file in its imports. Because the Protobuf deserializer would have only the current Protobuf definition file and not know what carseat.proto is, it would be unable to deserialize correctly.

  • When a Protobuf definition file contains a namespace or package, the message type must include it. For example:

    Screenshot that shows an example of a Protobuf definition file with a namespace.

    In the Protobuf deserializer in the portal, the message type must be Namespacetest.Volunteer instead of the usual Volunteer.

  • If the original package or namespace is all uppercase or lowercase, the message type must include it with the first character of the package or namespace in uppercase. For example, if the namespace is nameSpaceTest, use nameSpaceTest.Volunteer. This rule applies to a message type without a namespace or package.

  • You must specify your message in your protobuf definition file without the Optional keyword. In Proto 3, all fields are optional. For example:

    Screenshot that shows an example of a Protobuf definition file with optional keyword in the message.

    This Protobuf definition file shows a message that has Optional keyword. To deserialize correctly, you must remove the keyword:

    Screenshot that shows an example of a Protobuf definition file without optional keyword in the message.

  • When sending messages that were serialized via google.protobuf, the prefix type should be set to base128 because that's the most cross-compatible type.

  • Service messages aren't supported in the Protobuf deserializers. Your job throws an exception if you try to use a service message. For example:

    Screenshot that shows an example of a service message.

  • These data types aren't supported:

    • Any
    • One of (related to enumerations)
    • Durations
    • Struct
    • Field Mask (not supported by protobuf-net)
    • List Value
    • Value
    • Null Value
    • Empty

Note

For direct help with using the Protobuf deserializer, send email to askasa@microsoft.com.

See also

Data types in Azure Stream Analytics