Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Managed compute in Microsoft Foundry hosts models on dedicated GPU capacity, while Foundry manages the serving runtime and infrastructure. Production workloads need predictable latency, high concurrency, and sustained throughput, measured in requests served per second and input and output tokens processed and generated per second. A single model instance often isn't enough to meet these needs. Multi-instance deployments add capacity by running multiple instances of the same model. Each model instance uses 1, 2, 4, or 8 GPUs, as defined in the deployment template. The deployment's capacity counts model instances, not GPUs, so Total GPU count = GPUs per instance × capacity.
Prompt caching behavior
Many managed compute runtimes reuse work across requests that share an identical processed prefix when the underlying engine and model support it. Where supported, this behavior is enabled by default. A cache is instance-local and isn't shared across the instances of a deployment. A request routed to another instance can reuse a prefix only if it's already cached on that instance.
When an engine reports cached-token counts, it surfaces them in the standard OpenAI-compatible usage.prompt_tokens_details.cached_tokens field on the response. An explicit cached_tokens: 0 is a real, reported value meaning zero cached tokens were counted for that request. An omitted prompt_tokens_details field or cached_tokens key means cached-token reporting isn't available for that response. This condition might occur because the engine and model architecture combination doesn't support caching, or because usage metrics aren't enabled on the engine.
{
"usage": {
"prompt_tokens": 512,
"completion_tokens": 128,
"prompt_tokens_details": {
"cached_tokens": 384
}
}
}
For the full usage response schema, see the Chat completions API reference.
For a streaming response, include the final usage object only when the request sets both stream and stream_options.include_usage:
{
"prompt_cache_key": "my-app-support-tenant-42",
"stream": true,
"stream_options": { "include_usage": true }
}
For the full streaming request and response schema, see the Chat completions API reference.
Engine saturation
A managed compute runtime that can't take on more work rejects new requests with HTTP 429. Requests that the runtime already accepted continue to process normally; only new requests are rejected.
For example, the vLLM runtime that many managed compute deployments use enforces two admission gates:
--max-num-queued-reqslimits unfinished requests, including those waiting or in prefill or decode. Many small requests arriving together can use up all the request slots and trigger HTTP 429 even when the prefill-token budget has room.--max-num-queued-tokenslimits the current prefill-token backlog. A few long-context requests can fill this token budget and trigger HTTP 429 while the request count stays low. Each request's full prompt length is charged until its prefill finishes.
Together, these limits help prevent excessive backlogs and overload-related engine crashes while admitted requests continue processing.
When a single instance repeatedly reaches these runtime admission limits and returns 429, consider adding more model instances to the same deployment for extra serving capacity. Routing traffic across multiple instances in turn affects how prefix-cache reuse works, which the next section covers.
Prompt-aware routing
For a deployment with multiple model instances, prompt-aware routing is enabled by default where the engine and model support it. It favors sending requests with a shared prefix to the same instance to improve cache reuse. An optional prompt_cache_key provides an affinity hint for related requests, such as those with a shared system message:
{
"messages": [
{ "role": "system", "content": "You are a support assistant for Contoso. Answer concisely and cite the relevant policy section when applicable." },
{ "role": "user", "content": "What is the refund window for a defective item?" }
],
"prompt_cache_key": "my-app-support-tenant-42"
}
For the full request and response schema, see the Chat completions API reference.
prompt_cache_key is a routing-affinity hint for requests that share a common prefix. A request with a fixed prompt_cache_key can still be routed to a different instance than previous requests with the same key, for example under load (spillover), and that instance can have a cold cache for that prefix. This behavior is normal, not an exception.
Sizing a deployment
Active concurrency is the number of unfinished, in-flight inference requests at a point in time. It's not requests per second, and it's not the number of idle or open connections. The concurrency a single instance can sustain depends on input and output length, caching behavior, your workload mix, the model and runtime, and the GPU footprint of the deployment template.
A useful sizing approach is to measure the concurrency a single model instance sustains for your workload, then add headroom before scaling out to more instances. Provisioning and warmup after a capacity change takes time, so plan for that transition rather than expecting it to be instantaneous.