Durable orchestrators executing in DR environment accidentally

Question

Hi,

I have a function app with durable orchestrators deployed to app service in production and the DR. these are 2 separate app services , one in the prod East US 2 and another in DR in central US region.

The task hub name and the storage account used is same in both prod and DR region. there is a technical limitation due to which we have to keep 1 storage account and task hub name.

We usually keep the app service in DR region stopped so that orchestrations don't kick in automatically in DR region as we don't want them to run

Occasionally we are seeing the orchestrations run randomly in the DR region even after keeping the app service in DR stopped. Can you please help me troubleshooting why the orchestrators are starting in DR region and how I can prevent this ?

Answer 1

Hi @Parag Kale ,

This behavior can occur when Production and DR Durable Functions share the same Durable Task Hub (hubName) and storage account. Any active Function App connected to the same task hub can potentially pick up and process available orchestration work.

Recommended checks:

• Confirm the DR Function App is fully stopped and cannot be temporarily started by deployments, scaling, auto-heal, or platform activity.
• Verify that no other Function App is using the same hubName and storage account.
• Compare the Durable Functions configuration between Production and DR.
• Use Durable Functions HTTP APIs to verify the actual orchestration status instead of relying solely on the portal.
• Review Production and DR application/platform logs to identify which host processed the orchestration.

Prevention:

If the architecture requires one shared storage account and task hub, the key requirement is to ensure that only the intended active Function App can process work from that task hub.

For an active/passive DR model:

Keep the DR Function App fully stopped while Production is active.

Prevent deployment, autoscaling, auto-heal, or other processes from unintentionally starting the DR app.

• During a failover, activate DR only after Production processing has been stopped or isolated.

Monitor both applications to confirm which host is processing Durable work.

Ideally, separate task hubs should be used for independent applications/environments, because having multiple applications actively listening to the same Durable Task Hub can result in work being processed by an unintended application. However, if the architecture requires the same storage account and hubName, strict active/passive compute control becomes especially important.

The key point is that the shared task hub is the source of the potential cross-region processing. Stopping the DR app should prevent it from processing new work, but we should first verify that the DR host was completely inactive and that no other Function App was connected to the same task hub at the time the orchestration was processed.

Hope this helps!

---

If the resolution was helpful, kindly take a moment to click on and click on Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 2

As far as I can tell, the issue results from the shared Durable Task Hub. Both Function Apps use the same storage account and task hub name, so they are effectively workers on the same Durable Functions workload. There is no built-in primary/secondary designation. If the DR Function host starts for any reason, it could acquire leases from the shared task hub and execute production orchestrations.

A stopped App Service doesn't run orchestrators, so an execution in DR indicates the DR Functions host became active. Investigate the exact timestamp using App Service Activity Logs, Application Insights, deployment logs, and restart/start events to determine why the DR host started. Possible causes include deployments, configuration changes, platform operations, scaling events, or automation. Do not assume health checks or warm-up features are waking a stopped app without evidence.

To avoid it, prevent the passive DR app from accessing the shared Durable storage while in standby (for example, by blocking storage access or using a standby configuration). Then restore access as part of the DR activation process. Disabling the orchestrator with AzureWebJobs.<FunctionName>.Disabled=true may not be ideal because it treats the symptom rather than preventing the DR app from becoming a worker.

---

If the above response helps answer your question, remember to "Accept Answer" so that others in the community facing similar issues can easily find the solution. Your contribution is highly appreciated.

hth

Marcin