Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
A long-running hosted agent can be interrupted at any time by a crash, an out-of-memory kill, a redeploy, or a scale-in. This article shows how to make your agent's background responses recoverable and how to resume a recovered run from its last checkpoint.
Note
Long-running agents are in preview. APIs and package versions are subject to change.
Turn on crash recovery
Crash recovery is off by default. Enable it explicitly for the surface you use.
Responses protocol
Set resilient_background=True on ResponsesServerOptions:
from azure.ai.agentserver.responses import ResponsesAgentServerHost, ResponsesServerOptions
app = ResponsesAgentServerHost(
options=ResponsesServerOptions(resilient_background=True),
)
Recovery applies only to responses that are stored and run in the background - that is, requests with store=true and background=true. When you enable the opt-in and the container crashes mid-response, the framework reinvokes your handler on restart, replays persisted stream events to reconnecting clients, and preserves conversation state.
Important
Without resilient_background=True, a background response that crashes is marked failed with error.code="server_error" - the framework does not reinvoke the handler. Foreground (background=false) responses are always marked failed on crash, because their client connection is already gone.
Invocations / task primitives
When you build directly on the task primitives, declaring a @task or @multi_turn_task handler automatically enables the startup recovery scan. If you register tasks lazily after host startup, force-enable the scan before startup:
from azure.ai.agentserver.core.tasks import set_resilient_tasks_enabled
set_resilient_tasks_enabled(True) # call at import time, before host lifespan startup
What you get for free
When you turn on recovery, you get the framework half with no handler changes:
| Behavior | Detail |
|---|---|
| Handler reinvocation | The restarted container reenters your handler with the same request, input, and metadata. |
| Stream replay | Persisted SSE events replay to reconnecting clients. |
| Conversation lock | Prevents concurrent conflicting writes to the same conversation. |
| No-op cleanup | Marks nonrecoverable responses failed instead of silently rerunning them. |
A naive recovered handler still produces a correct response - it just reruns the whole turn. Making the recovered attempt resume where it left off is the handler half you take on when you need it.
Detect a recovered entry
On reinvocation, branch on the recovery marker rather than reconstructing the original request.
@app.response_handler
async def handler(request, context, cancellation_signal):
if context.is_recovery:
# Seed from the last checkpoint instead of starting over.
stream = ResponseEventStream.from_snapshot(context.persisted_response)
start_phase = len(stream.response.output) # completed, checkpointed phases
else:
stream = ResponseEventStream(response_id=context.response_id, request=request)
start_phase = 0
...
Choose a resume strategy
Pick a strategy based on where your progress state lives.
| Strategy | Where progress lives | Recovery behavior |
|---|---|---|
| Naive rerun | Nowhere | Rerun the whole turn. Correct, but unsafe for non-idempotent side effects unless they're fenced. |
| Framework checkpoints | Persisted response snapshots | Seed from context.persisted_response, resume after the checkpointed output items. |
| Upstream-owned resume | Your framework or app store | Rebuild from an agent-framework checkpoint or your database. See Manage state for long-running agents. |
| Watermark overlay | Small metadata watermarks | Combine with any strategy to avoid repeating side effects the upstream can't deduplicate. |
Prefer phase boundaries that checkpoint cleanly: complete one output item per phase, then checkpoint. If a phase crashes before its checkpoint it reruns; after the checkpoint the recovered attempt skips it.
Fence non-idempotent side effects
Before an action an upstream system can't deduplicate (for example, sending an email or charging a card), stamp and flush a watermark, then clear it after the side effect commits:
context.conversation_chain_metadata["email_sent"] = True
await context.conversation_chain_metadata.flush() # fence before the side effect
await email_service.send(...)
Handle graceful shutdown
Graceful shutdown is different from terminal failure. A handler that can't finish during shutdown should defer for recovery so the record stays in progress and a later lifetime reclaims it:
if context.is_shutting_down: # or ctx.shutdown.is_set() for tasks
await context.exit_for_recovery() # leaves the response in_progress for re-invocation
Crash recovery reenters the same attempt state; it doesn't consume retry budget, and a wall-clock timeout doesn't reset because the process restarted.