Azure Service Bus - Limit rate of messages that can consume from the queue

Question

Scenario -

There are multiple consumer instances that reads and process the messages from the queue. I want to limit the number of messages that can be consumed from the queue per minute, by all the consumers.

Lets say message consume limit per minute is 1000, then all the consumers collectively should only consume maximum of 1000 messages per minute.

However producers should be able to produce and queue messages without any limitation.

Is there any configuration from service bus queue side to limit message consume rate ( I couldn't found any). Otherwise how can I achieve this ?

Thank you.

Answer 1

@Kasun Tharaka Thanks for reaching out. There is no built-in configuration in Azure Service Bus to limit the message consumption rate across multiple consumer instances.

You need to implement your own solution to handle this scenario through all your consumer application. One approach is to use a shared counter to track the number of messages consumed by all consumer instances. Each consumer instance can increment the counter when it receives a message and decrease the counter when it finishes processing the message. If the counter exceeds the message consumption limit per minute, the consumer instances can pause consuming messages until the next minute.

Here are the high-level steps to implement this approach:

1. Create a shared counter: Create a shared counter that can be accessed by all consumer instances. You can use a distributed cache service like Azure Cache for Redis or Azure Cosmos DB to implement the shared counter.
2. Increment the counter when a message is received: When a consumer instance receives a message, it should increment the shared counter.
3. Decrement the counter when a message is processed: When a consumer instance finish processing a message, it should decrease the shared counter.
4. Check the counter before consuming a message: Before a consumer instance consumes a message, it should check the shared counter to ensure that the message consumption limit per minute has not been exceeded. If the limit has been exceeded, the consumer instance should pause consuming messages until the next minute.
5. Monitor the counter: Monitor the shared counter to ensure that the message consumption limit per minute is being enforced correctly. You can use Azure Monitor or other monitoring tools to monitor the counter and alert you if the limit is exceeded.

Note that this approach requires careful coordination between the consumer instances to ensure that the shared counter is updated correctly and that the message consumption limit per minute is enforced consistently. You may also need to adjust the message consumption limit per minute based on the performance characteristics of your consumer instances and the message processing workload.

Please "Accept Answer" if the answer is helpful so that it can help others in the community.