Azure Durable Function Fan-Out Not Appearing to be Parallel (Slow Performance)

Question

I've created a Durable Function (Isolated Worker in .NET 7) to run a backfill of particular data that includes the following:

• An HTTP-triggered function (Starter)
• The orchestrator
• Activity functions for reading data from file and then performing backfill on the records returned

We're on an EP1 plan (3.5 GB) with one other FA on the plan.

I'm using the following in the orchestrator to fan out a specific activity function that does the business logic:

 foreach (var item in itemList)  
{ itemIndex++;  
  activities.Add(context.CallActivityAsync(Constants.FUNCTION_NAME, new backfillRequest
	{   
		data
	})); 
 }   
await Task.WhenAll(activities);

This particular Activity Function does a good amount of business logic/DB queries. We want to fan out as much as possible because the goal is to iterate over 500k+ records (that itemList would have that many items in it for which an activity function is created for each).

However, as I increase the itemList to load test and see how our Azure resources are (to not overload DB etc.) , it doesn't appear like the Activity Functions are running in parallel. I increased the list to 200, so I was expecting 200 instances to be created and running in parallel, but I looked at the logs and found the following for instances of the activity function that is fanning out and the Process CPU % :

As the number of the above activity function increases in fan out (I've done fan outs of 5, 10, 50, up to the 200 above), the average time it takes for it to finish increases. For 1 item, it takes maybe 10 seconds. For 3 items, the average time is 1 minute. For 10 items, the average time is about 2.5 minutes. Is there something I'm missing in the fan out/fan in? I don't have any properties set in terms of maxConcurrentActivityFunctions or maxConcurrentOrchestratorFunctions and I'm not sure what might be causing the instances to seemingly run serially.

Answer 1

Justin Thanks for posting your question in Microsoft Q&A. Based on the description above, you followed the code snippet rightly as described in doc: Fan-out/fan-in scenario in Durable Functions - Cloud backup example.

When you fan out with multiple activity functions, those functions will execute on multiple VM instances concurrently, but scaling will not happen immediately, instead will follow the scaling rules. For example, premium plans follow Event-driven scaling and new instances are allocated (scaling up will take some time) faster than Consumption plan which is every 30 seconds and up to a maximum allowed by the plan. For more info, check out Understanding scaling behaviors. Hence, 200 instances might not be created like you mentioned above.

However, fan-in occurs on a single instance as described in doc: Performance Targets below. Meaning, the output from all the activity functions need to be processed by a single orchestrator on a single instance, which might cause performance issues. In such scenarios, we recommend leveraging sub-orchestrations to split it into smaller batches.

You can review App Insights to find the role instance allocation over the period of your load test timeframe (as chart) and this will help in understanding the behavior. That will also help in diagnosing the issues. For maxConcurrentActivityFunctions, here is the default value (source: host.json Settings) and I don't think this caused the issue.

I hope this helps and let me know if you have any questions.

---

If you found the answer to your question helpful, please take a moment to mark it as Yes for others to benefit from your experience. Or simply add a comment tagging me and would be happy to answer your questions.