Throttling consumption of Servicebus messages with azure functions

Question

We have built several solutions for clients based on azure functions and servicebus to move data between systems. Each solution hosts many (100+) individual functions triggering on individual topics from many different systems. Although some solutions are older, the majority are .net 8 isolated functions and a solution should be based on that.

Lately we have experienced issues with a system that is unable to process the amount of data and we therefor need to throttle requests. For throttling solutions we are debating a few options as outlined below. I would like input on the below options as well as suggestions to handle it differently.

1. Have trigger functions write to a new queue and build an entirely new function app with the specific purpose of throttling to one specific system. Scale out and max concurrency settings can be used to achieve the throttling level desired.

• This was my initial idea which I still rather like. Biggest concerns are double queue/function processing and single queue for a lot of messages which may become problematic.

2. same as 1, but use auto-forwarding of the servicebus

• Seems to be a better version of 1, but without being able to manipulate messages before 'fanning in'

3. Build custom azure function trigger that consults with a distributed lock or similar before fetching messages from the queue.

• This would be great, but I would really prefer to leverage the existing trigger already built by Microsoft rather than implement my own. If I could add to it, that would be great.

4. Use a distributed lock after receiving messages from the servicebus and throw them back on the queue if the lock is unachievable.

• I really don't like this one. It spends lots of resources (and money due to consumption/premium plans) to do nothing.

As i said. These are the solutions we have been talking about, but I would be very happy to be presented with better (simpler/easier) solutions for throttling.

Also posted on SO: https://stackoverflow.com/questions/78225549/throttling-consumption-of-servicebus-messages-with-azure-functions

Answer 1

To effectively manage the throttling of Service Bus messages with Azure Functions, consider implementing a multi-faceted approach that addresses both Azure Service Bus and Azure Functions characteristics. Service Bus utilizes a credit-based throttling mechanism ensuring fair resource distribution across namespaces by limiting the number of operations per time period. If your workload is particularly sensitive to throttling, moving to a premium namespace could afford more dedicated resources, thereby reducing throttling occurrences due to the predictable performance and the ability to scale resources according to demand.

On the Azure Functions side, controlling scale behavior is crucial. While the platform automatically scales based on the queue size, which can sometimes lead to excessive scaling, you can mitigate this by configuring maxConcurrentCalls in the host.json file. This allows you to control the rate at which your functions process messages. For more predictable behavior, especially under heavy loads, consider switching from a consumption plan to an App Service Plan.

Additionally, writing resilient and efficient function code is essential. Ensure your functions can gracefully handle interruptions and continue where they left off in the event of a failure, thus avoiding redundant operations. Organizing your functions intelligently, considering performance, scaling, and the specific needs of each function, can lead to better overall system efficiency. Reusing connections and avoiding resource-intensive operations can further improve performance and reliability.

Implementing these strategies collectively can lead to a more manageable, efficient, and robust system, effectively reducing the chances of being throttled while maintaining system responsiveness and reliability

In addition as a ref:

https://www.ben-morris.com/managing-serverless-scaling-with-azure-functions-by-throttling-throughput/