Ensure Even Distribution of Queue Messages Between Replicas

Question

How is load balancing achieved across replicas in Container Apps?

I'm having issues with performance when trying to scale up a container app to handle a large volume of queue messages. I am trying to scale between 0 and 12 replicas based on the queue size. The application is a python Functions app and I have tried using both Dapr topic binding and direct subscription to the service bus queue.

The container app is scaling up to 12 replicas when there is are a lot of messages in the queue. I would hope that the messages would get evenly distributed across the replicas. Instead, the replicas seem to be greedy and just a few replicas are trying to handle all the requests, while the rest sit idle. The performance here is very bad as the replicas get bogged down and run very slow.

I have tried tweaking Dapr settings such as "maxConcurrentHandlers" but to no effect. I have also tried a variety of values for the service bus queue listener such as "maxConcurrentCalls" but also no difference.

Is there a way to ensure a more even spread of request handling across replicas? Is this issue more likely to be caused by the application side? Any guidance on how to solve this? Happy to share more details on configuration if needed. Thanks.

Answer 1

Thanks for posting your question in the Microsoft Q&A forum.

Here are some suggestions and possible solutions for improving load balancing among replicas:

Ensure that your Service Bus client is configured correctly. For better load balancing, you should use PeekLock mode (default) instead of ReceiveAndDelete and set an appropriate PrefetchCount. A higher prefetch count can improve throughput but might lead to uneven distribution if set too high.

ServiceBusClient(
    connection_string,
    receive_mode=ServiceBusReceiveMode.PEEK_LOCK,
    prefetch_count=20,
    max_concurrent_calls=10
)

Ensure the message lock duration is appropriate for your processing time. If it's too short, messages might be reprocessed by other replicas before completion.

Implement the Competing Consumers pattern correctly. Each replica should independently pull messages from the queue and Consider using multiple queues or topics with subscriptions to distribute load more evenly.

---

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful