Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
A long-running hosted agent can be interrupted at any time by a crash, an out-of-memory kill, a redeploy, or a scale-in. This article shows how to make your agent's background responses recoverable and how to resume a recovered run from its last checkpoint.
Turn on crash recovery
Crash recovery is off by default. Enable it explicitly for the surface you use.
Responses protocol
Set resilient_background=True on ResponsesServerOptions:
from azure.ai.agentserver.responses import ResponsesAgentServerHost, ResponsesServerOptions
app = ResponsesAgentServerHost(
options=ResponsesServerOptions(resilient_background=True),
)
Recovery applies only to responses that are stored and run in the background - that is, requests with store=true and background=true. When you enable the opt-in and the container crashes mid-response, the framework reinvokes your handler on restart, replays persisted stream events to reconnecting clients, and preserves conversation state.
Important
Without resilient_background=True, a background response that crashes is marked failed with error.code="server_error" - the framework does not reinvoke the handler. Foreground (background=false) responses are always marked failed on crash, because their client connection is already gone.
Invocations / task primitives
When you build directly on the task primitives, you opt in to the resilient task subsystem. Call set_resilient_tasks_enabled(True) before host startup so the framework constructs the TaskManager and runs the startup recovery scan. Without it, get_task_manager() raises TaskManagerNotInitialized and .run() and .start() can't run a task. Declaring a @task doesn't enable the subsystem on its own:
from azure.ai.agentserver.core.tasks import set_resilient_tasks_enabled
set_resilient_tasks_enabled(True) # call at import time, before host lifespan startup
What you get for free
When you turn on recovery, you get the framework half with no handler changes:
| Behavior | Detail |
|---|---|
| Handler reinvocation | The restarted container reenters your handler with the same request, input, and metadata. |
| Stream replay | Persisted SSE events replay to reconnecting clients. |
| Conversation lock | Prevents concurrent conflicting writes to the same conversation. |
| No-op cleanup | Marks nonrecoverable responses failed instead of silently rerunning them. |
A naive recovered handler still produces a correct response - it just reruns the whole turn. Making the recovered attempt resume where it left off is the handler half you take on when you need it.
Detect a recovered entry
On reinvocation, branch on the recovery marker rather than reconstructing the original request.
@app.response_handler
async def handler(request, context, cancellation_signal):
if context.is_recovery:
# Seed from the last checkpoint instead of starting over.
stream = ResponseEventStream.from_snapshot(context.persisted_response)
start_phase = len(stream.response.output) # completed, checkpointed phases
else:
stream = ResponseEventStream(response_id=context.response_id, request=request)
start_phase = 0
...
Choose a resume strategy
Pick a strategy based on where your progress state lives.
| Strategy | Where progress lives | Recovery behavior |
|---|---|---|
| Naive rerun | Nowhere | Rerun the whole turn. Correct, but unsafe for non-idempotent side effects unless they're fenced. |
| Framework checkpoints | Persisted response snapshots | Seed from context.persisted_response, resume after the checkpointed output items. |
| Upstream-owned resume | Your framework or app store | Rebuild from an agent-framework checkpoint or your database. See Manage state for long-running agents. |
| Watermark overlay | Small metadata watermarks | Combine with any strategy to avoid repeating side effects the upstream can't deduplicate. |
Prefer phase boundaries that checkpoint cleanly: complete one output item per phase, then checkpoint. If a phase crashes before its checkpoint it reruns; after the checkpoint the recovered attempt skips it.
Fence non-idempotent side effects
Before an action an upstream system can't deduplicate (for example, sending an email or charging a card), stamp and flush a watermark, then clear it after the side effect commits:
context.conversation_chain_metadata["email_sent"] = True
await context.conversation_chain_metadata.flush() # fence before the side effect
await email_service.send(...)
Handle graceful shutdown
Graceful shutdown is different from terminal failure. A handler that can't finish during shutdown should defer for recovery so the record stays in progress and a later lifetime reclaims it:
if context.is_shutting_down: # or ctx.shutdown.is_set() for tasks
await context.exit_for_recovery() # leaves the response in_progress for re-invocation
Crash recovery reenters the same attempt state; it doesn't consume retry budget, and a wall-clock timeout doesn't reset because the process restarted.