Azure Durable Function - Queue Instances of Orchestration for Single Instance ID

Question

I have a data layer that is created and updated from multiple input layers. We currently use functions and a Service Bus session to ensure that only a single worker is ever trying to update a data layer at a time, even if there are multiple input layers that need to be aggregated in the queue. We'd like to use a Durable Orchestration though, since the aggregation function is long running with multiple steps. Is there a way to ensure that a single orchestration is only ever running for a Data Layer ID, and the orchestration is still invoked for all pending Input Layers?

I've used an InstanceId, but calling StartNew with an instance ID that is already running errors (I was hoping it would queue).

I've thought of creating eternal orchestrators that loop and wait for external events, since external events queue. My understanding though is this would create an eternal orchestrator per Data Layer ID, which are significant.

I've also thought of a still using Service Bus sessions, and have the function that invokes the orchestrator monitor its progress - but this would then keep a long running function active that is, technically, doing nothing - not sure how that would impact the runtime?

I've also looked at durable entities to trigger the orchestrator, but I don't think there is a reliable way to block within the durable entity method and wait for the orchestration to complete - and it may be beyond the intended use case of the durable entity anyway since we wouldn't use it to store state, just for coordination.

Answer 1

Hi @David Faivre - PCS ,

Thanks for reaching out to Q&A forum.

By setting maxConcurrentCalls and maxConcurrentSessions to 1 , you would achieve singleton processing and I think this would solve your problems

https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus?tabs=in-process%2Cextensionv5%2Cextensionv3&pivots=programming-language-csharp#hostjson-settings

Feel free to reach out to me if you have any queries or concerns.