You can create a compacted event hub using the Azure portal by following these steps.
Navigate to your Event Hubs namespace.
On the Event Hubs Namespace page, select Event Hubs in the left menu.
At the top of the window, select + Event Hubs.
Type a name for your event hub, and specify the partition count. Since we're creating a compacted event hub, select compaction policy as compaction and provide the desired value for tombstone retention time.
Select create and create the compacted event hub.
The following example shows how to create a compacted event hub/Kafka topic using an ARM template.
Event Hubs service determines when the compaction job of a given compacted event hub should be executed. Compacted event hub reaches the compaction threshold when there are considerable number of events or the total size of a given event log grows significantly.
Publish event to a compacted topic
Publishing events to a compacted event hub is the same as publishing events to a regular event hub. As the client application you only need to determine the compaction key, which you set using partition key.
Using Event Hubs SDK(AMQP)
With Event Hubs SDK, you can set partition key and publish events as shown below:
C#
var enqueueOptions = new EnqueueEventOptions
{
PartitionKey = "Key-1"
};
await producer.EnqueueEventAsync(eventData, enqueueOptions);
Using Kafka
With Kafka you can set the partition key when you create the ProducerRecord as shown below:
Java
ProducerRecord<String, String> record = new ProducerRecord<String, String>(TOPIC, "Key-1" , "Value-1");
There are no changes required at the consumer side to consume events from a compacted event hub. So, you can use any of the existing consumer applications to consume data from a compacted event hub.
Next steps
For conceptual information on how log compaction work, see Log compaction.
Learn to use Azure Event Hubs to reliably process high-volume data streams to enable you to code applications to send and receive messages through the hub.
This article shows how consumers and producers that use different protocols (AMQP, Apache Kafka, and HTTPS) can exchange events when using Azure Event Hubs.