Log compaction + HTTPS in Azure Event Hub

Question

Log compaction + HTTPS in Azure Event Hub

Verschinin Valentin 20

We would like to inquire about using Azure Event Hub's new feature, log compaction, in combination with HTTPS. See

It is not entirely clear from the documentation how these two elements can be combined, if they can be combined at all, as the role of the key remains unclear.

To rephrase it slightly, could you please provide instructions on how to set a message key using the REST API?

Thank you very much. Best, Valentin

Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-09-07T19:34:17.2466667+00:00
Hello Verschinin Valentin,

Welcome to the Microsoft Q&A forum.

I see this detailed document on how to use the log compaction.

Copied from the document:

Publishing events to a compacted event hub is the same as publishing events to a regular event hub. As the client application you only need to determine the compaction key, which you set using partition key.

{ PartitionKey = "Key-1" }

HTTP request:

POST https://your-namespace.servicebus.windows.net/your-event-hub/messages?timeout=60&api-version=2014-01 HTTP/1.1 Authorization: SharedAccessSignature sr=your-namespace.servicebus.windows.net&sig=your-sas-key&se=1403736877&skn=RootManageSharedAccessKey Content-Type: application/atom+xml;type=entry;charset=utf-8 Host: your-namespace.servicebus.windows.net { "PartitionKey" : "Key-1", "Body" : "event payload message" }

Tech community blogpost:
https://techcommunity.microsoft.com/t5/messaging-on-azure-blog/log-compaction-in-azure-event-hubs-public-preview/ba-p/3691722

I hope this helps.
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-09-08T22:36:28.4066667+00:00

Hello Verschinin Valentin,

I am checking to see if you got a chance to look into my above response. Please let me know if you have any further questions.
Verschinin Valentin 20 Reputation points

2023-09-09T12:59:18.9466667+00:00
Hello @Bhargava-MSFT ,

thanks for the quick reply. The AMQP call seems to be fine. However, I am not quite sure that the POST REST call is correct since it doesn't correspond to the documentation here

https://learn.microsoft.com/en-us/rest/api/eventhub/send-event

Are you sure about it? Might be the api-version is not 2014-01? I would expect PartitionKey being part of the header (if REST is supported with compaction at all) and the body being schema-less.

Content-Type: application/atom+xml;type=entry;charset=utf-8 PartitionKey: key-1 ...

Thanks for your help. Best, Valentin
Verschinin Valentin 20 Reputation points

2023-09-12T05:10:57.3833333+00:00

Hello @Bhargava-MSFT , thanks for your help. Best, Valentin
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-09-12T15:08:13.1333333+00:00

Thank you, Verschinin Valentin

If you don't have any further questions, please consider accepting the answer by hitting the Accept answer and up-vote as it helps the community look for answers to similar questions.

Answer accepted by question author

0 additional answers

Your answer

Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-09-07T19:34:17.2466667+00:00

Hello Verschinin Valentin,

Welcome to the Microsoft Q&A forum.

I see this detailed document on how to use the log compaction.

Copied from the document:

Publishing events to a compacted event hub is the same as publishing events to a regular event hub. As the client application you only need to determine the compaction key, which you set using partition key.

{ PartitionKey = "Key-1" }

HTTP request:

POST https://your-namespace.servicebus.windows.net/your-event-hub/messages?timeout=60&api-version=2014-01 HTTP/1.1 Authorization: SharedAccessSignature sr=your-namespace.servicebus.windows.net&sig=your-sas-key&se=1403736877&skn=RootManageSharedAccessKey Content-Type: application/atom+xml;type=entry;charset=utf-8 Host: your-namespace.servicebus.windows.net { "PartitionKey" : "Key-1", "Body" : "event payload message" }

Tech community blogpost:
https://techcommunity.microsoft.com/t5/messaging-on-azure-blog/log-compaction-in-azure-event-hubs-public-preview/ba-p/3691722

I hope this helps.
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-09-08T22:36:28.4066667+00:00

Hello Verschinin Valentin,

I am checking to see if you got a chance to look into my above response. Please let me know if you have any further questions.
Verschinin Valentin 20 Reputation points

2023-09-09T12:59:18.9466667+00:00

Hello @Bhargava-MSFT ,

thanks for the quick reply. The AMQP call seems to be fine. However, I am not quite sure that the POST REST call is correct since it doesn't correspond to the documentation here

https://learn.microsoft.com/en-us/rest/api/eventhub/send-event

Are you sure about it? Might be the api-version is not 2014-01? I would expect PartitionKey being part of the header (if REST is supported with compaction at all) and the body being schema-less.

Content-Type: application/atom+xml;type=entry;charset=utf-8 PartitionKey: key-1 ...

Thanks for your help. Best, Valentin
Verschinin Valentin 20 Reputation points

2023-09-12T05:10:57.3833333+00:00

Hello @Bhargava-MSFT , thanks for your help. Best, Valentin
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-09-12T15:08:13.1333333+00:00

Thank you, Verschinin Valentin

If you don't have any further questions, please consider accepting the answer by hitting the Accept answer and up-vote as it helps the community look for answers to similar questions.

Answer 1

Bhargava-MSFT 31,361 Microsoft Employee Moderator

Hello Verschinin Valentin,

Sorry for the confusion. You are correct.

You need to specify the message key using the REST API to include the PartitionKey in the HTTP header, not in the body.

Also, the API-version is correct as 2014-01

POST https://your-namespace.servicebus.windows.net/your-event-hub/messages?timeout=60&api-version=2014-01 HTTP/1.1  
Authorization: SharedAccessSignature sr=your-namespace.servicebus.windows.net&sig=your-sas-key&se=1403736877&skn=RootManageSharedAccessKey  
Content-Type: application/atom+xml;type=entry;charset=utf-8  
Host: your-namespace.servicebus.windows.net  
BrokerProperties: {"PartitionKey":"key1"}

{  
   "Body" : "event payload message"
}

Please let me know if you are still having issues with this.

Verschinin Valentin 20 Reputation points

2023-09-15T08:12:20.05+00:00
Hello @Bhargava-MSFT ,

unfortunately, setting PartitionKey in the Header doesn't work for us. We still get the following error message from Azure Event Hub.

<Error> <Code>405</Code> <Detail>Compacted event hub does not allow null message key. TrackingId: XYZ, SystemTracker:EH_NAMESPACE:eventhub:TOPIC_NAME~32766, Timestamp:2023-09-15T08:07:20</Detail> </Error>

Are you sure that the feature is supported with REST Post-Requests? Best, Valentin
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-09-15T23:07:24.29+00:00

The error message refers to the partition key of the event data. As the event compaction relies on the partition keys of the event data, sending the event with partition keys is required.

Can you check if the events were sent with partition ID rather than partition key?

The partition key is a set of string or integer values used to determine the partition to which an event should be sent based on a hash of the key value.
Verschinin Valentin 20 Reputation points

2023-09-18T12:03:40.2033333+00:00
Hi @Bhargava-MSFT ,
thanks for the quick reply. We were using for our test the following endpoint

https://learn.microsoft.com/en-us/rest/api/eventhub/send-event

Without partition ID. We were setting partition key as described by you above in the header of the request.

Do you have some idea on what might be wrong here? We assume that setting partition key in the header, as indicated in your answer above, with PartitionKey: key-1, doesn't work. Is there some official documentation from Azure on how REST-Calls are compatible with Azure Event Hub compaction?

Thanks a lot. Best, Valentin
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-09-18T23:10:36.3733333+00:00

Hello Verschinin Valentin,

Sorry, I don't see any documentation on this. I am contacting my internal team to see if they can share any knowledge articles on this.

I will keep you posted.
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-09-21T00:11:54.9033333+00:00
Hello Verschinin Valentin,

As per the discussion with the internal team, they suggested that we can pass the partition key in BrokerProperties request header as below.

BrokerProperties: {"PartitionKey":"key1"}

I have updated the same in the answer section. I hope this helps.

https://learn.microsoft.com/en-us/rest/api/servicebus/message-headers-and-properties

POST https://your-namespace.servicebus.windows.net/your-event-hub/messages?timeout=60&api-version=2014-01 HTTP/1.1 Authorization: SharedAccessSignature sr=your-namespace.servicebus.windows.net&sig=your-sas-key&se=1403736877&skn=RootManageSharedAccessKey Content-Type: application/atom+xml;type=entry;charset=utf-8 Host: your-namespace.servicebus.windows.net BrokerProperties: {"PartitionKey":"key1"} { "Body" : "event payload message" }
Verschinin Valentin 20 Reputation points

2023-09-22T11:16:59.2666667+00:00

Hello @Bhargava-MSFT ,

thanks a lot for your help. I can confirm the solution. When you then consume the message with KafkaAPI, the PartitionKey with the value "key1" appears as the Key of the Kafka message, as expected. Best, Valentin
Bhargava-MSFT 31,361 Reputation points Microsoft Employee Moderator

2023-09-22T13:46:05.73+00:00

Hello Verschinin Valentin,

I'm glad to know the answer was helpful.

If you have any more questions or need further assistance in the future, Please don't hesitate to reach out to us on the Q&A forum. We are happy to help!

Share via

Log compaction + HTTPS in Azure Event Hub

0 additional answers

Your answer