Function Optimized Container Apps shutdowns during activity function running.

Question

Function Optimized Container Apps shutdowns during activity function running.

Shigeru Nakagaki 205

Hi.

recently I faced a weird issue.

I triggered Container Apps by three HTTP requests.

When there is no retry by exception, everything are ok.

But when some exceptions were happened in activity function which invoked with context.call_activity_with_retry() method, Container Apps shut down without waiting for the retrying activity function to finish.

( shut down means zero replica.)

The last log before shut down is

MS_DURABLE_FUNCTION_EVENTS_LOGS
{
    "ProviderName":"DurableTask-AzureStorage",
    "TaskName":"OrchestrationServiceStats",
    "EventId":112,
    "EventTimestamp":"2025-08-29T07:03:09.1348287Z",
    "Pid":9,
    "Tid":39,
    "Level":4,
    "ExtensionGUID":"5eb34c26-4482-456b-a885-80e6c4d318b9",
    "Account":"xxx",
    "TaskHub":"xxx",
    "StorageRequests":107,
    "MessagesSent":4,
    "MessagesRead":4,
    "MessagesUpdated":10,
    "TableEntitiesWritten":42,
    "TableEntitiesRead":43,
    "PendingOrchestrators":0,
    "PendingOrchestratorMessages":0,
    "ActiveOrchestrators":0,
    "ActiveActivities":2,
    "AppName":"xxx",
    "ExtensionVersion":"2.2.0.221989"
}

ActiveActivities is two, but Container Apps shut down.
( ActiveOrchestrators is zero, why?)

What do you think could be causing this issue?

thanks

Praveen Kumar Gudipudi 2,290 Reputation points Microsoft External Staff Moderator

2025-08-29T09:37:04.0966667+00:00
Shigeru Nakagaki,

Container Apps are designed to scale to zero when idle. To keep orchestrators active.

ActiveOrchestrators = 0 Means No orchestrator functions are currently running.

So please

scale:

minReplicas: 1

-->Use Durable Functions with Premium Plan or Dedicated Hosting

If possible, consider hosting Durable Functions in:

Premium App Service Plan (supports Always On)

Dedicated VM or Kubernetes (for full control over lifecycle)

Please check above suggested steps and let me know with an update.
Krishna Chowdary Paricharla 2,915 Reputation points Microsoft External Staff Moderator

2025-08-30T00:51:50.9966667+00:00
Hello Shigeru Nakagaki,

This behavior usually happens because Azure Functions with Consumption/Elastic Premium/Container Apps (optimized for Functions) will scale down to zero when the orchestrator is idle — i.e., no active orchestrator loops are pending. Durable Functions separate orchestrator and activity roles:

ActiveOrchestrators = 0 means no orchestrator executions are currently running.

ActiveActivities = 2 shows retry are in progress, but scaling logic for Functions/Container Apps only tracks orchestrator load, not background activities.

So, the host considers the workload “idle” and tears down the container before the retries complete.

Recommendations:

Ensure you have Always Ready / Minimum Replica configured in Container Apps hosting, so instances don’t drop to zero during long activity retries.

Alternatively, consider using Durable Functions on Premium Plan (or set scaling rules to keep at least 1 replica alive).

Review retries patterns — call_activity_with_retry is orchestrator-driven, but if the orchestrator finishes prematurely or is not checkpointed, retries won’t be preserved correctly.

For your reference, please refer the below documentation:

Set scaling rules in Azure Container Apps

Performance and scale in Durable Functions (Azure Functions)

Hope this helps!
Shigeru Nakagaki 205 Reputation points

2025-09-01T00:24:09.13+00:00

Hi, Krishna

I want to know why retries of call_activity_with_retry won’t be preserved correctly.

When retries happen, some logic has to be done in orchestrator function?

thanks
Krishna Chowdary Paricharla 2,915 Reputation points Microsoft External Staff Moderator

2025-09-01T08:52:18.9233333+00:00
Hello Shigeru Nakagaki,

In Durable Functions, retries with call_activity_with_retry are managed by the orchestrator, not the activity itself.

When an activity fails, the orchestrator records a failure event in its state table and schedules the next retry according to your policy.

If the orchestrator instance is unloaded or the app scales down to 0 replicas, it must later be re-hydrated to continue the retry process.

If the orchestrator has already finished or isn’t checkpointed properly before scale-in, then there’s no orchestration context alive to schedule the retry — so the retry won’t be preserved.

That’s why in your logs you see ActiveOrchestrators = 0 while activities are still running. The platform thinks the orchestration work is done, so scaling logic tears down the container, and retries can’t continue.

What you can do:

Make sure your orchestrator function does not complete until retries are scheduled (the Durable runtime handles this automatically if the instance stays alive).

Configure minimum replicas / Always Ready in Container Apps so the app doesn’t scale to 0 while retries are pending.

Alternatively, Durable Functions on Premium plan guarantees retries are preserved because the orchestrator never scales to 0.

For your reference, please refer the below documentation:
Handling errors in Durable Functions (Azure Functions)
Shigeru Nakagaki 205 Reputation points

2025-09-02T01:18:44.2466667+00:00
Make sure your orchestrator function does not complete until retries are scheduled (the Durable runtime handles this automatically if the instance stays alive).

I think the orchestrator function does not finish during activity retries with following code. Am I write?

retry_options = df.RetryOptions( first_retry_interval_in_milliseconds=1000, max_number_of_attempts=3, ) result = yield context.call_activity_with_retry("HogeActivity", retry_options, some_input)
Krishna Chowdary Paricharla 2,915 Reputation points Microsoft External Staff Moderator

2025-09-02T09:48:21.6533333+00:00
Hello Shigeru Nakagaki,
Yes, you are correct! When you use

retry_options = df.RetryOptions( first_retry_interval_in_milliseconds=1000, max_number_of_attempts=3, ) result = yield context.call_activity_with_retry("HogeActivity", retry_options, some_input)

the orchestrator will not finish until either:

the activity succeeds and returns a result, or

all retries are exhausted and the final failure is thrown.

So, your code is fine — the Durable runtime automatically checkpoints the orchestrator state between retries and resumes when the retry delay has passed.

The shutdown issue you are seeing is not because the orchestrator “finished early,” but because the hosting environment (Container Apps scaling to 0) tears down the instance while no active orchestrators are detected. That’s why configuring minimum replicas / Always Ready or running on a plan that doesn’t scale to 0 is needed.
Shigeru Nakagaki 205 Reputation points

2025-09-03T00:30:58.2133333+00:00

I believe always running one replica at least is not the solution.

Why are not there active orchestrator even there are retrying active activity functions?
Krishna Chowdary Paricharla 2,915 Reputation points Microsoft External Staff Moderator

2025-09-03T21:32:45.92+00:00

Hello Shigeru Nakagaki.

In Durable Functions, retries are orchestrator-driven but the orchestrator goes idle while waiting for activity results or retry timers. That’s why you see ActiveActivities > 0 but ActiveOrchestrators = 0. This is expected behavior — the orchestrator isn’t counted as “active” during waits. To prevent scale-to-zero from interrupting retries, you need either a minimum replica in Container Apps or to run on a Premium plan where orchestrations are preserved.
Krishna Chowdary Paricharla 2,915 Reputation points Microsoft External Staff Moderator

2025-09-04T20:02:30.08+00:00

Hello Shigeru Nakagaki,

Just checking in to see if you've had a chance to review my previous response and let me know if you have any additional questions.
Krishna Chowdary Paricharla 2,915 Reputation points Microsoft External Staff Moderator

2025-09-06T01:12:32.4+00:00

Hello Shigeru Nakagaki,

I wanted to follow up to see if you’ve had an opportunity to look over my earlier response. Please let me know if you have any further questions.
Shigeru Nakagaki 205 Reputation points

2025-09-08T01:14:09.7166667+00:00

A. Orchestrator is not active during activity function is running. B. Container apps will start timing themselves to tear down if there is not active orchestration function.

These specification are not understandable.
If there are any running activity functions, the container app should not be torn down.

But if you say "It's specification", we have to follow that.
A workaround is to increase the "cooldownPeriod" of the container app.
Krishna Chowdary Paricharla 2,915 Reputation points Microsoft External Staff Moderator

2025-09-08T18:05:09.3633333+00:00

Hello Shigeru Nakagaki,

You are correct—this behavior can seem counter-intuitive. In Durable Functions, the scale controller monitors only the orchestrator load and does not consider activity execution. Therefore, even if activities are still running, the application may be treated as idle and scaled down if no orchestrator is active. This is the intended platform behavior, not a bug.

Your workaround is effective: increasing the cooldownPeriod or setting a minimum replica will prevent the container from shutting down while activities are being retried. Alternatively, using the Premium plan fully resolves this, as orchestrations are always maintained.
Krishna Chowdary Paricharla 2,915 Reputation points Microsoft External Staff Moderator

2025-09-10T23:44:46.0933333+00:00

Hello Shigeru Nakagaki,

Just checking in to see if you've had a chance to review my previous response and let me know if you have any additional questions
Shigeru Nakagaki 205 Reputation points

2025-09-11T00:22:45.1533333+00:00

Ahhh, well, it's okay...(?)

Answer accepted by question author

0 additional answers

Your answer

Praveen Kumar Gudipudi 2,290 Reputation points Microsoft External Staff Moderator

2025-08-29T09:37:04.0966667+00:00

Shigeru Nakagaki,

Container Apps are designed to scale to zero when idle. To keep orchestrators active.

ActiveOrchestrators = 0 Means No orchestrator functions are currently running.

So please

scale:

minReplicas: 1

-->Use Durable Functions with Premium Plan or Dedicated Hosting

If possible, consider hosting Durable Functions in:

Premium App Service Plan (supports Always On)

Dedicated VM or Kubernetes (for full control over lifecycle)

Please check above suggested steps and let me know with an update.
Krishna Chowdary Paricharla 2,915 Reputation points Microsoft External Staff Moderator

2025-08-30T00:51:50.9966667+00:00

Hello Shigeru Nakagaki,

This behavior usually happens because Azure Functions with Consumption/Elastic Premium/Container Apps (optimized for Functions) will scale down to zero when the orchestrator is idle — i.e., no active orchestrator loops are pending. Durable Functions separate orchestrator and activity roles:

ActiveOrchestrators = 0 means no orchestrator executions are currently running.

ActiveActivities = 2 shows retry are in progress, but scaling logic for Functions/Container Apps only tracks orchestrator load, not background activities.

So, the host considers the workload “idle” and tears down the container before the retries complete.

Recommendations:

Ensure you have Always Ready / Minimum Replica configured in Container Apps hosting, so instances don’t drop to zero during long activity retries.

Alternatively, consider using Durable Functions on Premium Plan (or set scaling rules to keep at least 1 replica alive).

Review retries patterns — call_activity_with_retry is orchestrator-driven, but if the orchestrator finishes prematurely or is not checkpointed, retries won’t be preserved correctly.

For your reference, please refer the below documentation:

Set scaling rules in Azure Container Apps

Performance and scale in Durable Functions (Azure Functions)

Hope this helps!
Shigeru Nakagaki 205 Reputation points

2025-09-01T00:24:09.13+00:00

Hi, Krishna

I want to know why retries of call_activity_with_retry won’t be preserved correctly.

When retries happen, some logic has to be done in orchestrator function?

thanks
Krishna Chowdary Paricharla 2,915 Reputation points Microsoft External Staff Moderator

2025-09-01T08:52:18.9233333+00:00

Hello Shigeru Nakagaki,

In Durable Functions, retries with call_activity_with_retry are managed by the orchestrator, not the activity itself.

When an activity fails, the orchestrator records a failure event in its state table and schedules the next retry according to your policy.

If the orchestrator instance is unloaded or the app scales down to 0 replicas, it must later be re-hydrated to continue the retry process.

If the orchestrator has already finished or isn’t checkpointed properly before scale-in, then there’s no orchestration context alive to schedule the retry — so the retry won’t be preserved.

That’s why in your logs you see ActiveOrchestrators = 0 while activities are still running. The platform thinks the orchestration work is done, so scaling logic tears down the container, and retries can’t continue.

What you can do:

Make sure your orchestrator function does not complete until retries are scheduled (the Durable runtime handles this automatically if the instance stays alive).

Configure minimum replicas / Always Ready in Container Apps so the app doesn’t scale to 0 while retries are pending.

Alternatively, Durable Functions on Premium plan guarantees retries are preserved because the orchestrator never scales to 0.

For your reference, please refer the below documentation:
Handling errors in Durable Functions (Azure Functions)
Shigeru Nakagaki 205 Reputation points

2025-09-02T01:18:44.2466667+00:00

Make sure your orchestrator function does not complete until retries are scheduled (the Durable runtime handles this automatically if the instance stays alive).

I think the orchestrator function does not finish during activity retries with following code. Am I write?

retry_options = df.RetryOptions( first_retry_interval_in_milliseconds=1000, max_number_of_attempts=3, ) result = yield context.call_activity_with_retry("HogeActivity", retry_options, some_input)
Krishna Chowdary Paricharla 2,915 Reputation points Microsoft External Staff Moderator

2025-09-02T09:48:21.6533333+00:00

Hello Shigeru Nakagaki,
Yes, you are correct! When you use

retry_options = df.RetryOptions( first_retry_interval_in_milliseconds=1000, max_number_of_attempts=3, ) result = yield context.call_activity_with_retry("HogeActivity", retry_options, some_input)

the orchestrator will not finish until either:

the activity succeeds and returns a result, or

all retries are exhausted and the final failure is thrown.

So, your code is fine — the Durable runtime automatically checkpoints the orchestrator state between retries and resumes when the retry delay has passed.

The shutdown issue you are seeing is not because the orchestrator “finished early,” but because the hosting environment (Container Apps scaling to 0) tears down the instance while no active orchestrators are detected. That’s why configuring minimum replicas / Always Ready or running on a plan that doesn’t scale to 0 is needed.
Shigeru Nakagaki 205 Reputation points

2025-09-03T00:30:58.2133333+00:00

I believe always running one replica at least is not the solution.

Why are not there active orchestrator even there are retrying active activity functions?
Krishna Chowdary Paricharla 2,915 Reputation points Microsoft External Staff Moderator

2025-09-03T21:32:45.92+00:00

Hello Shigeru Nakagaki.

In Durable Functions, retries are orchestrator-driven but the orchestrator goes idle while waiting for activity results or retry timers. That’s why you see ActiveActivities > 0 but ActiveOrchestrators = 0. This is expected behavior — the orchestrator isn’t counted as “active” during waits. To prevent scale-to-zero from interrupting retries, you need either a minimum replica in Container Apps or to run on a Premium plan where orchestrations are preserved.
Krishna Chowdary Paricharla 2,915 Reputation points Microsoft External Staff Moderator

2025-09-04T20:02:30.08+00:00

Hello Shigeru Nakagaki,

Just checking in to see if you've had a chance to review my previous response and let me know if you have any additional questions.
Krishna Chowdary Paricharla 2,915 Reputation points Microsoft External Staff Moderator

2025-09-06T01:12:32.4+00:00

Hello Shigeru Nakagaki,

I wanted to follow up to see if you’ve had an opportunity to look over my earlier response. Please let me know if you have any further questions.
Shigeru Nakagaki 205 Reputation points

2025-09-08T01:14:09.7166667+00:00

A. Orchestrator is not active during activity function is running. B. Container apps will start timing themselves to tear down if there is not active orchestration function.

These specification are not understandable.
If there are any running activity functions, the container app should not be torn down.

But if you say "It's specification", we have to follow that.
A workaround is to increase the "cooldownPeriod" of the container app.
Krishna Chowdary Paricharla 2,915 Reputation points Microsoft External Staff Moderator

2025-09-08T18:05:09.3633333+00:00

Hello Shigeru Nakagaki,

You are correct—this behavior can seem counter-intuitive. In Durable Functions, the scale controller monitors only the orchestrator load and does not consider activity execution. Therefore, even if activities are still running, the application may be treated as idle and scaled down if no orchestrator is active. This is the intended platform behavior, not a bug.

Your workaround is effective: increasing the cooldownPeriod or setting a minimum replica will prevent the container from shutting down while activities are being retried. Alternatively, using the Premium plan fully resolves this, as orchestrations are always maintained.
Krishna Chowdary Paricharla 2,915 Reputation points Microsoft External Staff Moderator

2025-09-10T23:44:46.0933333+00:00

Hello Shigeru Nakagaki,

Just checking in to see if you've had a chance to review my previous response and let me know if you have any additional questions
Shigeru Nakagaki 205 Reputation points

2025-09-11T00:22:45.1533333+00:00

Ahhh, well, it's okay...(?)

Answer 1

Hello Shigeru Nakagaki,

You are correct—this behavior can seem counter-intuitive. In Durable Functions, the scale controller monitors only the orchestrator load and does not consider activity execution. Therefore, even if activities are still running, the application may be treated as idle and scaled down if no orchestrator is active. This is the intended platform behavior, not a bug.

Your workaround is effective: increasing the cooldownPeriod or setting a minimum replica will prevent the container from shutting down while activities are being retried. Alternatively, using the Premium plan fully resolves this, as orchestrations are always maintained.

Please accept as answer and do a Thumbs-up to upvote this response if you are satisfied with the community help. Your upvote will be beneficial for the community users facing similar issues.

User's image

Share via

Function Optimized Container Apps shutdowns during activity function running.

0 additional answers

Your answer