How to set the Event Hub partition key per-event using "Send Multiple Events" action?

Question

How to set the Event Hub partition key per-event using "Send Multiple Events" action?

Veli-Jussi Raitila 446

I am using Logic Apps Standard. Iterating and sending individual events using the Azure Event Hubs "Send Event" action does not give me acceptable performance.

Switching to "Send Multiple Events" (InApp) action in a stateless workflow is better, but for some inexplicable reason I can only specify a single partition key value for the whole batch.

I have prepped the events into an array using the Select action. And I can then provide the entire array to "Send Multiple Events". But how can I specify the partition keys for those events? The objects in the array include the required properties.

EDIT: In fact, the partitionKey property seems to be ignored altogether.

{
  "type": "ServiceProvider",
  "inputs": {
    "parameters": {
      "eventHubName": "logexample",
      "eventDatas": "@body('Select_Events')",
      "partitionKey": "hardcoded"
    },
    "serviceProviderConfiguration": {
      "connectionName": "eventHub",
      "operationId": "sendEvents",
      "serviceProviderId": "/serviceProviders/eventHub"
    }
  }
}

In the example above the Select_Event body holds an array of events mapped to the ContentData property. Also, I've hard-coded the partition key value to make sure that they all end up in the same partition - even on subsequent invocations. However, this does not happen. What am I getting wrong?

MikeUrnun 9,777 Reputation points Moderator

2024-01-29T06:07:33.23+00:00

@Veli-Jussi Raitila Per the ref doc for the "Send Multiple Events" action, the Partition Key param is for a single key. Is it something that you can pre-compute in a prior action, perhaps using rand() functions to pick a random partition key, and then use its value for the Partition Key param?
Veli-Jussi Raitila 446 Reputation points

2024-01-30T07:33:21.3033333+00:00

@MikeUrnun I'm aware of what the documentation says. I specifically need to preserve the processing order (within a partition), and choosing an appropriate partition key is the way to achieve that. I also have a way of constructing one by combining three different fields. This is all pretty standard stuff.

The bad thing is that producing those events one-by-one is terribly inefficient.

I would also expect this to be consistent e.g. by using the same partition key, the events (with the same key) will deterministically end up in the same partition. Picking a random key each time would not give me such guarantees. I will probably have to resort to Azure Functions as prepping the data would become needlessly complex within a workflow.

Now, as you pointed out, the behavior for "Send Multiple Events" does indeed match the documentation in that one can only define a single partition key for a batch. However, in my experimentation this also has a problem: even hard-coding the partition key (for a batch) does not guarantee a predictable partition assignment. And behavior like that is not documented.

I have a "Send multiple events" action inside a Do-Until loop which sends 100 events per iteration. And even if I set a static value for a partition key, each batch is seemingly assigned a partition using round-robin - completely ignoring the partition key setting.
Veli-Jussi Raitila 446 Reputation points

2024-01-30T08:54:28.0833333+00:00

Just ran into this https://github.com/MicrosoftDocs/azure-docs/issues/117375. Even though the original report is about bridging Event Grid with Event Hubs - might be related. The behavior is the same though. It is not clear from the description where the fix was released. At least for me the issue still persists with Logic Apps.
MikeUrnun 9,777 Reputation points Moderator

2024-02-08T03:51:54.0166667+00:00

Hi @Veli-Jussi Raitila - Good find! I can reach out to the owning team internally on the status of the fix that was released. Could you confirm which region you're deploying to?
Veli-Jussi Raitila 446 Reputation points

2024-02-09T08:14:16.5366667+00:00

@MikeUrnun The region is West Europe. I think this is a rather serious issue as it messes up the ordering guarantees and might hence result in data inconsistencies.

Your answer

MikeUrnun 9,777 Reputation points Moderator

2024-01-29T06:07:33.23+00:00

@Veli-Jussi Raitila Per the ref doc for the "Send Multiple Events" action, the Partition Key param is for a single key. Is it something that you can pre-compute in a prior action, perhaps using rand() functions to pick a random partition key, and then use its value for the Partition Key param?
Veli-Jussi Raitila 446 Reputation points

2024-01-30T07:33:21.3033333+00:00

@MikeUrnun I'm aware of what the documentation says. I specifically need to preserve the processing order (within a partition), and choosing an appropriate partition key is the way to achieve that. I also have a way of constructing one by combining three different fields. This is all pretty standard stuff.

The bad thing is that producing those events one-by-one is terribly inefficient.

I would also expect this to be consistent e.g. by using the same partition key, the events (with the same key) will deterministically end up in the same partition. Picking a random key each time would not give me such guarantees. I will probably have to resort to Azure Functions as prepping the data would become needlessly complex within a workflow.

Now, as you pointed out, the behavior for "Send Multiple Events" does indeed match the documentation in that one can only define a single partition key for a batch. However, in my experimentation this also has a problem: even hard-coding the partition key (for a batch) does not guarantee a predictable partition assignment. And behavior like that is not documented.

I have a "Send multiple events" action inside a Do-Until loop which sends 100 events per iteration. And even if I set a static value for a partition key, each batch is seemingly assigned a partition using round-robin - completely ignoring the partition key setting.
Veli-Jussi Raitila 446 Reputation points

2024-01-30T08:54:28.0833333+00:00

Just ran into this https://github.com/MicrosoftDocs/azure-docs/issues/117375. Even though the original report is about bridging Event Grid with Event Hubs - might be related. The behavior is the same though. It is not clear from the description where the fix was released. At least for me the issue still persists with Logic Apps.
MikeUrnun 9,777 Reputation points Moderator

2024-02-08T03:51:54.0166667+00:00

Hi @Veli-Jussi Raitila - Good find! I can reach out to the owning team internally on the status of the fix that was released. Could you confirm which region you're deploying to?
Veli-Jussi Raitila 446 Reputation points

2024-02-09T08:14:16.5366667+00:00

@MikeUrnun The region is West Europe. I think this is a rather serious issue as it messes up the ordering guarantees and might hence result in data inconsistencies.

Share via

How to set the Event Hub partition key per-event using "Send Multiple Events" action?

Your answer