Should I create EventProcessorClient per partition to increase throughput?

Question

Hi ,

I am doing an integration to sink data into Event Hub using Apache Camel.

I created a small Camel test to read messages from Event Hub. I have another process that runs in parallel to push data into the same event hub. My event hub has 4 partitions.

I only see one EventProcessorClient instance created even after I bumped the number of messages from 20 per minute to 40K per minute. I was expecting that new instances of EventProcessorClient would be auto-created to scale the read. Here is the Camel code https://github.com/apache/camel/blob/main/components/camel-azure/camel-azure-eventhubs/src/main/java/org/apache/camel/component/azure/eventhubs/client/EventHubsClientFactory.java. If only one EventProcessClient is created per route, should I consider to add multiple routes (per partition) to increase the parallelization and thus increasing the throughput?

Thank you,
Jianying

Answer 1

It looks like the camel implementation is built on top of the Azure SDK for Java (doc for EventProcessor). Only one event processor should be created for a consumer group. Within that client, it will create one PartitionProcessors for each partition in your event hub. If your event hub has 4 partitions, there should always be 4 partition processors running regardless of the number of inbound messages.