Premium Event Hubs performance issue

Question

I have a premium Event hub namespace with 16 PUs having three event hubs:-

1. Event hub E1 with 100 partitions
2. Event hub E2 with 60 partitions
3. Event hub E3 with 60 partitions.

Event hub E1 is where data ingestion takes place, I am testing it with 20000 messages/sec with each message being 200 bytes long. The data is also being consumed across all 100 partitions through receivers at about the same rate.

I am noticing the CPU usage to go up to 70% while testing and also noticing intermittent connection issues on the consumer end. My question is, is the above configuration with 16 PUs not enough to handle this throughput?

Answer 1

Hello Varun,

Welcome to the MS Q&A platform.

As per the below document, for the premium namespace, a single PU with one event hub(100 partitions) can offer approximately a core capacity of 5-10 MB/s ingress and 10-20 MB/s egress for both AMQP or Kafka workloads.

https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-scalability

In your case, you are ingesting 20,000 messages/sec, with each message being 200 bytes long. This results in an ingress rate of 4 MB/s (20,000 * 200 bytes = 4,000,000 bytes/s). Since you have 16 PUs, the total ingress capacity should be between 80 MB/s and 160 MB/s (16 PUs * 5-10 MB/s per PU)

Your current configuration with 16 PUs should be able to handle the given throughput. However, other factors, such as network latency and resource contention, may affect the performance.

To troubleshoot the issue, you can monitor the CPU usage and other metrics in the Azure portal and consider adjusting the partition distribution strategy or the retention period to improve the performance.

Additionally, you can set up alerts to get notified when CPU usage reaches certain thresholds.

I hope this helps. Please let us know if you have any further questions.