Durable Function orchestrator race condition?

Question

Durable Function orchestrator race condition?

Namrata Singhal 0

have an Azure Durable Function that is triggered when users send SMS messages. I'm using the function to collect multiple messages if they arrive within a short duration, and create one consolidated message.

The first message triggers the function and calls the start_new method of the durable function orchestrator, subsequent messages raise a new_message event

The issue I'm facing is that if two messages arrive very close, both of them start the orchestrator because the second message arrives before the orchestration for the first message was initialized. So the durable function runtime starts a new orchestration for each message, because of which one of the messages overwrites the other since they both belong to the same orchestration with the same predetermined orchestration id.

Is this a known issue and how can I get around it?

Here's the code I have:

Queue Trigger (run when a new message arrives on a message queue)

Sai Prabhu Naveen Parimi 2,265 Reputation points Microsoft External Staff Moderator

2025-06-23T22:18:34.9233333+00:00

@Namrata Singhal

This can indeed be tricky since the orchestrator can kick off multiple instances if the messages arrive too closely together, all utilizing the same orchestration ID.

Here's what you can try to address this issue:

Use a Unique Orchestration ID: Instead of a static orchestration ID, generate a unique ID for each orchestration instance. This way, resulting orchestrators won’t overwrite each other's state. If the messages are expected to be collected over a short duration, consider using a composite key that includes a timestamp or other unique identifiers.

Implement a Delay: Before you trigger a new orchestration, you could introduce a short delay or check if another orchestration instance is currently in progress for the same "session". If there is one, you can use WaitForExternalEvent to pause until it completes or adjust how you consolidate your messages.

Leverage Durable Timers: You can set up a durable timer to wait for a certain period to consolidate incoming messages before starting the orchestration. If a new message arrives within this duration, you simply reset your timer.

Critical Sections: As part of the Azure Durable Functions, you could consider defining a critical section when you need to interact with external systems or handle message input. Use LockAsync if your operations allow it to ensure that only one orchestration accesses shared resources at a time.

External Event Handling: If the design allows, implement external event handling (via WaitForExternalEvent), where the orchestrator will await the arrival of messages rather than immediately starting a new orchestration.
Namrata Singhal 0 Reputation points

2025-06-24T14:42:05.2066667+00:00

Thank you so much for your kind help. My answers to your comments are in bold below, and unfortunately, we are still grappling with this issue. Are you aware of somebody who might be able to consult for a few hours on this topic for compensation? Please see below for my replies - many thanks if there is any other help you can provide

Use a Unique Orchestration ID: Instead of a static orchestration ID, generate a unique ID for each orchestration instance. This way, resulting orchestrators won’t overwrite each other's state. If the messages are expected to be collected over a short duration, consider using a composite key that includes a timestamp or other unique identifiers.

We're already using a unique orchestration id -- it's based on the user id. The issue is that if a message arrives before the runtime has had time to start the orchestration instance, then two instances with the same orchestration id are started (start_new does not raise an exception), so one of the messages overwrites the other and we end up with only one message

Implement a Delay: Before you trigger a new orchestration, you could introduce a short delay or check if another orchestration instance is currently in progress for the same "session". If there is one, you can use WaitForExternalEvent to pause until it completes or adjust how you consolidate your messages.

I'm already checking whether another instance in currently in progress. If that returns False, I call start_new to start the new instance, but then the second message comes in before the instance is even started, before it enters the "Pending" state, so checking to see if another instance is running still returns False

Leverage Durable Timers: You can set up a durable timer to wait for a certain period to consolidate incoming messages before starting the orchestration. If a new message arrives within this duration, you simply reset your timer.

That is the entire idea -- I have both a timer and a wait_for_external_event as separate tasks, and I'm waiting for either of them to complete. If the timer completes, I know I'm done, but if the wait is triggered with a new message then I append it to the existing message. But since the orchestrator starts anew each time, it never gets to the raise_event() call because it's always calling start_new() as I explained above

Critical Sections: As part of the Azure Durable Functions, you could consider defining a critical section when you need to interact with external systems or handle message input. Use LockAsync if your operations allow it to ensure that only one orchestration accesses shared resources at a time.There are no external systems or shared resources so no need for critical sections

External Event Handling: If the design allows, implement external event handling (via WaitForExternalEvent), where the orchestrator will await the arrival of messages rather than immediately starting a new orchestrationSee above
Namrata Singhal 0 Reputation points

2025-06-24T14:49:15.01+00:00

Many thanks for your help. PLease find my answers below in BOLD. Is there any other help you recommend? For instance - are you aware of anybody who could consult with us (for compensation) for a few hours to get us solved on this or based on these comments below is there any other help you may provide? Thank you so much

This can indeed be tricky since the orchestrator can kick off multiple instances if the messages arrive too closely together, all utilizing the same orchestration ID. >>> Here's what you can try to address this issue: >>> Use a Unique Orchestration ID: Instead of a static orchestration ID, generate a unique ID for each orchestration instance. This way, resulting orchestrators won’t overwrite each other's state. If the messages are expected to be collected over a short duration, consider using a composite key that includes a timestamp or other unique identifiers.

We're already using a unique orchestration id -- it's based on the user id. The issue is that if a message arrives before the runtime has had time to start the orchestration instance, then two instances with the same orchestration id are started (start_new does not raise an exception), so one of the messages overwrites the other and we end up with only one message>>> Implement a Delay: Before you trigger a new orchestration, you could introduce a short delay or check if another orchestration instance is currently in progress for the same "session". If there is one, you can use WaitForExternalEvent to pause until it completes or adjust how you consolidate your messages.I'm already checking whether another instance in currently in progress. If that returns False, I call start_new to start the new instance, but then the second message comes in before the instance is even started, before it enters the "Pending" state, so checking to see if another instance is running still returns False

Leverage Durable Timers: You can set up a durable timer to wait for a certain period to consolidate incoming messages before starting the orchestration. If a new message arrives within this duration, you simply reset your timer.

That is the entire idea -- I have both a timer and a wait_for_external_event as separate tasks, and I'm waiting for either of them to complete. If the timer completes, I know I'm done, but if the wait is triggered with a new message then I append it to the existing message. But since the orchestrator starts anew each time, it never gets to the raise_event() call because it's always calling start_new() as I explained above>

Critical Sections: As part of the Azure Durable Functions, you could consider defining a critical section when you need to interact with external systems or handle message input. Use LockAsync if your operations allow it to ensure that only one orchestration accesses shared resources at a time.There are no external systems or shared resources so no need for critical sections>>> External Event Handling: If the design allows, implement external event handling (via WaitForExternalEvent), where the orchestrator will await the arrival of messages rather than immediately starting a new orchestrationSee above
Sai Prabhu Naveen Parimi 2,265 Reputation points Microsoft External Staff Moderator

2025-06-25T23:58:36.9933333+00:00

@Namrata Singhal

It sounds like you're running into a classic race condition issue with Azure Durable Functions due to timing in message processing. Since you're using the same orchestration ID for multiple messages that arrive closely together, it's causing both instances to start up and overwrite each other.

Here are a few strategies you can consider to tackle this:

Ensure Unique Orchestration IDs: I see you mentioned using unique orchestration IDs based on user ID, which is great! If your use case allows, you could try using a combination of user ID and a timestamp or a random GUID appended to prevent the overlap during those rapid incoming requests.

Add a Consolidation Delay: Implement a timer that activates just before you start the orchestration. During this period, if another message comes in, you can either queue it or reset the timer to capture additional messages. This can help avoid multiple triggers within a very short time frame.

Use an Intermediate Status Check: Before calling start_new, check if an orchestration with the same ID is still in "running" or "pending" status. If it is, the second message can log its content for when the first orchestration completes without needing to start a new one, preserving the original message.

Utilize Durable Timers Effectively: You’ve already mentioned using timers and external events, which is a good approach! You could extend this by leveraging a WaitForExternalEvent that allows the orchestrator to pause until new messages arrive or the timer completes, helping it to consolidate tasks more effectively.

Implement an Aggregator Function: Consider creating an external aggregator function that manages the incoming messages and consolidates them before calling the orchestrator. This way, the orchestrator is involved only after batch processing, minimizing the chance of hit overlap.

Review the Event Handling Logic: Check your orchestration's handling logic to ensure that if an orchestrator with a specific ID is already in progress, any new messages are adapted to fit into that workflow instead of starting a new one.
Sai Prabhu Naveen Parimi 2,265 Reputation points Microsoft External Staff Moderator

2025-06-28T00:32:44.7433333+00:00

@Namrata Singhal

Just checking in to see if you've had a chance to review my previous response. Let me know if you have any additional questions.

Your answer

Sai Prabhu Naveen Parimi 2,265 Reputation points Microsoft External Staff Moderator

2025-06-23T22:18:34.9233333+00:00

@Namrata Singhal

This can indeed be tricky since the orchestrator can kick off multiple instances if the messages arrive too closely together, all utilizing the same orchestration ID.

Here's what you can try to address this issue:

Use a Unique Orchestration ID: Instead of a static orchestration ID, generate a unique ID for each orchestration instance. This way, resulting orchestrators won’t overwrite each other's state. If the messages are expected to be collected over a short duration, consider using a composite key that includes a timestamp or other unique identifiers.

Implement a Delay: Before you trigger a new orchestration, you could introduce a short delay or check if another orchestration instance is currently in progress for the same "session". If there is one, you can use WaitForExternalEvent to pause until it completes or adjust how you consolidate your messages.

Leverage Durable Timers: You can set up a durable timer to wait for a certain period to consolidate incoming messages before starting the orchestration. If a new message arrives within this duration, you simply reset your timer.

Critical Sections: As part of the Azure Durable Functions, you could consider defining a critical section when you need to interact with external systems or handle message input. Use LockAsync if your operations allow it to ensure that only one orchestration accesses shared resources at a time.

External Event Handling: If the design allows, implement external event handling (via WaitForExternalEvent), where the orchestrator will await the arrival of messages rather than immediately starting a new orchestration.
Sai Prabhu Naveen Parimi 2,265 Reputation points Microsoft External Staff Moderator

2025-06-28T00:32:44.7433333+00:00

@Namrata Singhal

Just checking in to see if you've had a chance to review my previous response. Let me know if you have any additional questions.

Share via

Durable Function orchestrator race condition?

Your answer