Durable Functions guidance on "scaling" external events

Question

HI,

I am exploring using Durable Functions as a way to kick off a huge number of background tasks and then await their completion via external events.

The general pattern would be an orchestrator will call an activity to retrieve a list of items (from a data service, simple activity function that will return quite quickly).

The orchestrator would then at a high level want to loop through each of these items and call an activity that sends an event grid topic to cause another totally separate piece of work to start (say for example a triggered Azure Data Factory pipeline that could itself take 20 minutes to complete). and the orchestrator would then await on an external event for each item.

The activity that sends the event grid topic would smuggle inside the topic data enough information for the separate piece of work to know how to call back to the orchestrator to raise the external event (the orchestrator endpoint, instance id, event etc.).

I have this working well, and in testing a small scenario where I loop through 10 items or so.

My question is on the guidance on how many external events could I raise before the Durable runtime has a problem coping. Are there any whitepapers, articles on how many external events can be created awaited on?

Kind regards,

Colin

Answer 1

Hi @colinrippeyfinarne ,

Please refer to article Azure Storage performance targets for details on the performance, scale and throughput for various scenarios. According to it, the expected maximum throughput for "External Event Processing" is "50 events per second, per instance". This is when using the Azure Storage as the storage provider for durable functions. There are other storage providers also available but in preview, which have shown better results as well. More details available here: Durable Functions storage providers

Please 'Accept as answer' and ‘Upvote’ if it helped so that it can help others in the community looking for help on similar topics.