Azure IoT Central Data Export to Event Hub - Configuring Partition Key Through Data Transformation

MesoMat Co 6

I'm using Azure IoT Central's data export to Event Hubs feature with a data transformation, and I would like to set a partition key that the Event Hub will use. I want to make it so that all the events from one device end up on the same Event Hub partition.
What is the correct way to do this? What do I have to do within IoT Central to get my intended partitioning to work properly?

I have tried passing a field called "PartitionKey" in the transformed event body:

{  
    "Measurement": 1.7845966827179327e+308,  
    "PartitionKey": "hrhp2uttrli1",  
    "TimeEnqueued": "1948-11-15T20:07:30.267714401Z",  
    "deviceId": "hrhp2uttrli1",  
    "module": null  
}

This does not make it so that all events from one device go to the same Event Hub partition.

Thanks!

MesoMat Co 6 Reputation points

2022-08-29T16:32:18.863+00:00

I have read through the documentation here:
https://learn.microsoft.com/en-us/azure/iot-central/core/howto-export-to-event-hubs?tabs=connection-string%2Cjavascript

and here:
https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-messages-construct

I believe the only mention of partitioning is with regards to how it's handled from devices to IoT Hub, and nothing seems to explicitly mention how to pass a partition key to the receiving Event Hub.
QuantumCache 20,261 Reputation points

2022-08-30T05:23:00.657+00:00

Hello @MesoMat Co Community SME's on this topic or our team will review your scenario and circle back at the possible earliest time.
Dom 1,631 Reputation points Microsoft Employee

2022-08-31T10:20:05.943+00:00

Hi @MesoMat Co - it may be that IoT Central already uses the DeviceId as the partition key - a quick test here shows my devices consistently sticking to the same partition as they're exported.
MesoMat Co 6 Reputation points

2022-08-31T17:45:23.067+00:00

Thanks for the response!

Is there a data transformation taking place before the export?

Regardless, could you please show what the output message looks like?

When I had my data export set up as shown in the picture in the original post, some devices were still having their events stored in the Event Hub partitions in a round-robin fashion, and other devices would have it so that it wasn't consistent which partition their events would be stored in (e.g. first 4 would go to Partition 0, next 2 would go to Partition 1, next 1 back to Partition 0).

Round-robin behaviour was observed when not specifying the "PartitionKey" field, but there was the "deviceId" field.
Dom 1,631 Reputation points Microsoft Employee

2022-09-01T11:10:29.273+00:00

Having left the test running longer, I'm now seeing the same behaviour that you do, messages from each device are going to different partitions at different times.

I also tried adding an x-opt-partition-key field containing the device id to each message by using a transformation. This didn't help either. I think the x-opt-partition-key needs to be part of the message envelope and not the payload to work. Transformations in IoT Central only work on the message payload.

Is it possible to amend your device code so that the device adds the x-opt-partition-key field to the message header - it may be that IoT Central will pass the header value through to the Event Hub?
MesoMat Co 6 Reputation points

2022-09-01T19:49:44.567+00:00

I appreciate the additional testing, and thank you for confirming that Transformations in IoT Central only affect the message payload.

I have device data coming in to IoT Central from a solution provider's server, and the manipulation of the message's header seems to be something that could take a considerable amount of development time. I also wanted to clarify: is the x-opt-partition-key field supposed to be part of the header of each AMQP message sent to IoT Central or is it another protocol that is used?

Thanks for the suggestion, but I'm unsure if I can pursue it at this time. Are there any other recommendations for setting a partition key?
Dom 1,631 Reputation points Microsoft Employee

2022-09-02T07:15:57.607+00:00

Without testing, I don't know whether it's possible for IoT Central to pass through an x-opt-partition-key fields as part of a message header - it's not a field that IoT Central recognizes or uses itself.

The Event Hubs client SDKs all have ways to set the partition key value - under the hood the SDKs set the x-opt-partition-key value in the header of the message sent to the Event Hub.

What's your reason for wanting to explicitly set the partition key from IoT Central? Do you have some downstream processing that relies on this?
MesoMat Co 6 Reputation points

2022-09-02T20:11:23.893+00:00

Thanks again for the information.

My reason for wanting to explicitly set the partition key from IoT Central is related to downstream processing, more specifically, to be able to guarantee in-order processing of the messages per device when they are retrieved from the Event Hub. Guaranteeing the order usually isn't an issue until the downstream processing needs to be taken offline (like for an update) and then messages accumulate in the Event Hub. Then when the downstream processing starts back up, there's no guarantee a device's messages will be processed in order because one of the partitions might have a later message and that gets processed first.

IoT Central is the first place in my pipeline where I can easily manipulate the data and it's directly before sending the messages to Event Hub, so it seemed like the logical place to set the partition key from.
AshokPeddakotla-MSFT 30,241 Reputation points

2022-09-05T02:52:54.05+00:00

@MesoMat Co Wanted to follow up with you to understand whether the below information helped. Please let us know if there are any other queries.
If below suggestions answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

1 answer

Roman Kiss 2,246 Reputation points

2022-09-02T21:40:03.017+00:00
Hi @MesoMat Co ,

Basically, there are two workarounds for your requirement such as sending the telemetry data to the specific EH partition based on the deviceId, etc.

Using the webhook destination endpoint to send an event to a specified partition in an Event Hub, see here. The Data Transformation will massage the event payload and also a rule for selecting a destination endpoint for each partition. Note, that currently we can not populate a destination endpoint parameters (url, headers, etc.) dynamically, all configuration must be explicitly setup it. Please, vote for this feature here.

Exporting the device telemetry data to the Azure Event Grid as a cloudevents message (via the custom topic) and then delivering to the specific partition in the Event Hub, see here. Using the subscription advanced filtering, the events can be filtered based on the needs. Note, that the AEG model doesn't guarantee delivering events in order.

Thanks
Roman
Please sign in to rate this answer.

2 people found this answer helpful.
MesoMat Co 6 Reputation points

2022-09-09T14:21:17.57+00:00

Thank you for the workarounds, Roman. They were very helpful to consider.

I have a rough setup for option 1 and it seems to be working as expected.

I need to make sure that everything is as reliable as possible, and so I had 2 main points of focus I wanted to ask about the solution that uses the webhook data export and the Event Hub Rest service:

I noticed that there is an optional x-ms-retrypolicy header. I would like to take advantage of the retries to deal with transient errors, so I don't have to do anything with this header because that is the default behavior. Are there any other headers to set or other ways to have more control over how retries are handled? Is the assumption correct that these retries are for the Rest API communicating with the Event Hub?

I think I need to split this up, so please see the next comment too.

MesoMat Co 6 Reputation points

2022-09-09T14:21:39.607+00:00

My second question:
I understand that for both the IoT Central app and the Event Hub there is the Metrics Monitoring in Azure Portal where I can see if there were any errors for either resource. However, I'm unsure of how to see the details of errors when they occur. For example, I can see infrequent errors classified as a "User Error" on the Event Hub side, but I think that's a general title that many errors can be categorized as. I see the option to "Drill into Logs", but I'm unsure about how to configure this to be able to see specific errors or if that's possible to see with the log options available. For example, the Activity Logs for the Event Hub don't tell me anything about these user errors. Is there any documentation that will instruct me to configure a way to see error specifics?

Thanks everyone!
Sign in to comment

Share via

Azure IoT Central Data Export to Event Hub - Configuring Partition Key Through Data Transformation

1 answer