How to scale out to more workers for a fan-out / fan-in durable function?

Question

How to scale out to more workers for a fan-out / fan-in durable function?

Saxon Druce 25

Hi,

I'm using durable functions and have a single orchestration function which fans out 100 activity functions in parallel, then collates the results.

Each activity does some heavy numeric processing, running for about a minute at 100% CPU single threaded.

Ideally, all 100 tasks would run in parallel and so the final output of the orchestration could be available in one minute (plus some overhead). I expect it to take a while for the workers to be scaled up and so it won't be quite as perfectly parallel as 100 tasks in 1 minute. However I am only seeing about 20 workers being allocated for the tasks, and so overall it takes about 5 minutes to get through the tasks.

Is there something I can do to allow the workers to scale up more quickly to meet the number of tasks?

I am using the Consumption plan, and so since each worker only has a single core, there isn't any benefit to allocating multiple tasks to a single worker at a time - each is using 100% CPU and so they will just cause CPU contention with each other. So I have set maxConcurrentActivityFunctions and maxConcurrentOrchestratorFunctions to 1, which did seem to help a bit.

I've run tests with varying numbers of tasks, and always end up with less workers than tasks, eg:

10 tasks: about 5 workers, about 2-3 minutes total run time
100 tasks: about 20 workers, about 5 minutes total run time
200 tasks: about 30 workers, about 7 minutes total run time

The following shows the number of workers I saw for 5 runs of the orchestration at each of these task sizes:

User's image

Is there anything else I can adjust to improve the throughput of the tasks?

Thanks,
Saxon

0 comments

Answer accepted by question author

0 additional answers

Your answer

Answer 1

Pramod Valavala 20,661 Microsoft Employee Moderator

@Saxon Druce Scaling out of function instances is done by the scale controller which uses heuristics for each trigger type but as such doesn't have a way to configure. The new target-based scaling looks promising but doesn't support Durable Functions now.

As the documentation for the scale controller mentions, the check to scale out is done every 30 seconds and since your activity functions run for at-most a minute, the scale out decisions are not made to expand to more instances.

The settings that you have made are the best possible for the consumption tier.

But if you do require your runs to complete as fast as possible, then your option would be to move to a higher tier like premium where you the scaling decisions are made faster and also you have control to temporarily scale out using the API directly as well.

Saxon Druce 25 Reputation points

2023-10-16T07:56:31.2233333+00:00

Thanks @Pramod Valavala for your answer.

After posting my question I had noticed the documentation you mentioned, about the scale up rate being at most once every 30 seconds on Consumption, but faster on Premium:

New instance rate: For HTTP triggers, new instances are allocated, at most, once per second. For non-HTTP triggers, new instances are allocated, at most, once every 30 seconds. Scaling is faster when running in a Premium plan.

So I also ran some experiments comparing 1000 tasks on both the Consumption and the EP1 Premium plans:

Although Premium did look to be scaling up faster than Consumption, it still took quite some time to get there. The max instances on Premium is 100, so at best this could do 1000 tasks with 100 instances so in 10 minutes which would be okay, but it took significantly longer than that to scale up to that point. I also experimented with the EP3 Premium plan with 4-core workers, which did process roughly 4x as fast, but still took time to scale up.

Can you please point me to some documentation on what you mentioned about using an API to scale out directly? I hadn't come across any info about being able to do that with the Premium plan. Or would that require eg a Dedicated plan with an App Service plan, where I could pre-allocate a number of VMs?

Thanks,
Saxon
Pramod Valavala 20,661 Reputation points Microsoft Employee Moderator

2023-10-18T15:27:35.2466667+00:00

@Saxon Druce Yes. That looks about right. While the premium plans are much faster, scaling up still takes some amount of time

The API approach (REST API to Update an App Service Plan) that I was mentioning is to set the number of instances manually before you need the call(s) and then down scale back once you are done. Not ideal but ensures the right number of instances are up and running by the time you make the call. Useful especially if you have a series of calls that you know you are going to make and need to optimize how fast they run.

Do note that this will incur more cost because premium tier instances are billed by the second they are up.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Share via

How to scale out to more workers for a fan-out / fan-in durable function?

0 additional answers

Your answer