Durable Functions Scaling Not Woking

Mark Barge 6

I have have set up a consumption (serverless) based function plan in which I have installed a durable function. The process calls an activity function which is used to calculate a large set of mathematical equations and writes the result to data matrices.

I notice that when called, the activity is allocated 2 CPUs. Given the intensity of the calculations I need a separate VM to be allocated to it for each call so I configured the host file with the following.

"durableTask": {
    "maxConcurrentActivityFunctions": 1,
    "maxConcurrentOrchestratorFunctions": 1
  }

But it doesn't appear to be working. If I make a single call it takes 1 min. If I make 5 concurrent calls they take 4 minutes each.

With this configuration I expected each call to be allocated its own VM and 2 CPUs and should each take just 1 minute, but that clearly is not happening.

I'm likely to end up with 1000s of concurrent calls and clearly at the moment that would be impossible.

Can anyone please explain what is happening here and why it's not working? Also is there a way to monitor how many VM have been allocated to the Durable Function as a whole?

1 answer

Pramod Valavala 20,591 Reputation points Microsoft Employee

2021-09-27T07:30:44.573+00:00

@Mark Barge The setting that you are using controls the number of activity/orchestration functions per instance and in the case of concurrent requests, should scale out to multiple instances based on the events (in this case queue messages from durable functions). This should happen once every 30 seconds and is decided by the scale controller as mentioned in the docs in a way unique to each trigger type.

So, if you have the single activity function taking up all resources, the scale controller should scale out your function app during this cycle, but not directly on calling an activity function. To understand more about the scale controller decisions, you can enable scale controller logs.

To improve scale out, you could switch to using the Premium Plan which along with faster scaling offers higher CPU count per instance.
Please sign in to rate this answer.
Mark Barge 6 Reputation points

2021-09-27T08:46:04.97+00:00

Hi Pramod

Thanks for your response.

Just to be clear, from your answer I suspect that what happened is that I triggered 5 separate calls to the durable function but because they were triggered within 30 seconds of each other, no scaling could be triggered.

If the 5 calls were made every say 45 seconds then the durable function would scale to 5 VMs?

IIs that correct?

Thanks

Mark

Pramod Valavala 20,591 Reputation points Microsoft Employee

2021-09-27T12:43:51.457+00:00

@Mark Barge Not exactly. When the requests are made, they get queued, and the scale controller will scale out the function app every 30 seconds based on the queue length. So, if you were to wait a bit, the function app should get scaled out as requests start getting queued.

Mark Barge 6 Reputation points

2021-10-04T21:40:32.857+00:00

Ok, This is not making a lot of sense to me. My azure durable function take between 2 and 4 minutes to run. I'm not aware of any queue. I call it using a web API call and it runs almost instantly. The start-up process calls the orchestration which in turns calls a single activity function which does all the work. Where the queue?
Are you saying that while its running its still counts as one item in the queue? Otherwise the queue would almost always be empty.
So lets say I call the function twice in a short period of time so that on both calls there has been no scaling out. While both are running the scale controller hits its 30 second point and realizes there are two items running (i.e. in the queue) and so starts spinning up a another VM. Is this correct?

Even if this is the case I cannot accept this. It means that I could have say 100 calls within 30 seconds all sharing the same resources.

I currently use a German company called Contabo (https://contabo.com) to provide VPSs with up to 16 cores each (32 threads). They are much cheaper than VPSs on azure. My alternative to azure is to simply build an array of multiple VPSs (each containing the code to run this process) and simply queue them to ensure I am guaranteed two threads per call. So each machine can handle 16 simultaneous computations.

Will any configuration of Azure Durable Functions replicate this and avoid the need to purchase multiple VPSs? Clearly the Consumption Plan does not. But would the Premium plan solve the problem? If so why?

Many thanks

Mark

Pramod Valavala 20,591 Reputation points Microsoft Employee

2021-10-18T09:46:37.233+00:00

@Mark Barge Queues are internally used by Durable Functions, which is how they are made reliable and workaround individual timeout limits (since they are separate function executions).

The up to 30 seconds time that the scale controller takes should trigger multiple instances to spin up if there are many pending requests (not running requests). Each instance is still bound by the same limits, so if you have 100 calls within 30 seconds, the ones beyond the limit would be queued and picked up by instances once the function app starts scaling out.

Running it in a VM is indeed one option and you could use Azure App Service Plan instead which simplifies management. The Premium Plan is something you could try out before since this gives you the benefit of both (higher compute and elastic scale).

Mark Barge 6 Reputation points

2021-10-19T05:06:34.587+00:00

Your answer does not explain why when I configure the DF to accept only one (maxConcurrentActivityFunctions=1) the function seems to run multiple call at the same time and does not appear to queue them after I call the first one.

If what you say is correct then when I make say 5 calls to the DM within a few seconds of each other I should expect the first one to run and subsequent ones to be queued.

The queuing system should then pick up the remain 4 calls in the queue and scale out 4 more VM to run them.... That does not happen... they all run as soon as I call them on one VM.
Sign in to comment