How model router works

Model router in Microsoft Foundry is a trained routing model that decides, for every request, which model in its pool should answer. It works the same way whether you call your own model router deployment or Auto (model router without a deployment). This article follows a request from arrival to response, so you can predict how model router behaves, configure it with confidence, and set realistic expectations for cost, quality, and latency.

  • Model router ranks eligible models on the text of each request, including its complexity, reasoning needs, and task type, and then applies your routing mode.
  • Only eligible models compete: models in your pool or model subset that are available in your region and can serve what the request needs. A single request can narrow this set further with per-request controls (preview), by listing models or by requiring support for every parameter it sends.
  • Your own deployments of the selected model in the same Foundry resource serve the request first (preview): your provisioned throughput (PTU) deployment for OpenAI models, and then your pay-as-you-go deployment. Router-managed capacity serves it otherwise.
  • If a model returns a retryable error, model router fails over to the next-ranked model automatically.
  • Every response's model field names the model that answered.

Important

Features or capabilities marked (preview) in this article are in public preview and are subject to the Supplemental Terms of Use for Microsoft Azure Previews. Previews are provided without a service-level agreement, and Microsoft doesn't recommend them for production workloads. Certain features might not be supported or might have constrained capabilities.

How a request is routed

Every request goes through the same steps, from the eligibility check to the response. The following diagram shows that path, including how model router chooses capacity and fails over.

Diagram of how model router routes a request. It keeps the eligible models, ranks them in your routing mode, and serves the top model from your PTU deployment, then your pay-as-you-go deployment, then router-managed capacity, before it fails over.

  1. Find eligible models. Model router starts from your model pool, or from your model subset if you chose one. It keeps only the models that can serve this request:
    • The model is available in your region.
    • If it's a Claude model, it's deployed in the same Foundry resource, with a deployment type that matches the model router deployment.
    • If it's a Fireworks model, Fireworks is enabled for your Azure subscription.
    • The model supports what the request needs, such as function tools, JSON schema output, or image input.
    • If the request sets routing_config.models (preview), the model is in that list. This list can only narrow your pool or model subset.
    • If the request sets routing_config.require_parameters to true (preview), the model supports every parameter in the request as sent.
  2. Rank the models. Model router analyzes the full request, including the system message, user messages, tool definitions, and conversation history, and estimates how well each eligible model will answer it. Then it applies your routing mode to pick one. With session affinity (preview), model router can keep a conversation on the model that served the previous turn:
    • Adaptive (the default when the request has the preview header): when a different model ranks first for the new turn, model router keeps the previous model only if two things are true. The earlier part of the conversation is unchanged, so that model's prompt cache is likely still warm. And the previous model's expected quality for the new turn is within your routing mode's quality range of the top-ranked model. Otherwise, it switches to the top-ranked model.
    • Sticky: model router tries the previous model first on every turn while that model is eligible.
  3. Choose capacity. If the same Foundry resource has your own deployment of the selected model, that deployment serves the request: a provisioned throughput (PTU) deployment first for OpenAI models, and then a pay-as-you-go deployment. Otherwise, router-managed capacity serves it. For the full order, see How model router chooses capacity.
  4. Call the model. Model router sends the request to the selected model. By default, it drops optional parameters that the model doesn't support, and it passes reasoning effort when the model supports it.
  5. Fail over if needed. If the call fails with a retryable error, model router tries again, as described in Failover.
  6. Return the response. The model field names the model that served the request, for example gpt-5.6-luna-2026-07-09.

By default, model router repeats these steps for every request, so the turns of one conversation can go to different models. In the Responses API, when a request continues a conversation with previous_response_id or conversation, model router considers the earlier turns when it ranks models.

The routing decision adds a small overhead to each request. To see it for your own traffic, check Router overhead in the Request latency chart on your deployment's Monitor tab. Model router analyzes your prompts to route them, but it doesn't store them.

How model router ranks models

Model router isn't a large language model. It's a purpose-built machine learning model, optimized for fast inference, that predicts which model performs best for a given prompt. It's trained on billions of interactions across many domains, including question answering, code generation, mathematical reasoning, summarization, conversation, and agentic workflows with tool calls.

Its core skill is judging difficulty. Not every coding question is equally hard, and not every summary needs the same depth of reasoning. Model router separates prompts that any capable model handles well, which suit fast, lower-cost models, from prompts that need deeper reasoning, nuanced judgment, or complex tool orchestration, where frontier models justify their higher cost. That's how it lowers cost while keeping quality comparable: it uses frontier models only when a prompt needs them.

Production requests are rarely tidy, single-sentence questions, so model router is built for:

  • Long contexts, where the routing signal can be spread across thousands of tokens.
  • Multi-turn conversations, where earlier turns provide context and the latest user message carries the most routing signal.
  • Agentic and tool-calling requests, where the model must produce structured tool invocations.

The routing decision uses the text of the request. Image input works, but images don't influence the decision. Audio input isn't supported.

Routing modes in depth

After model router ranks the eligible models, the routing mode decides how it trades quality against cost. Quality is measured against the best eligible model for each prompt.

Mode Models considered Model selected Typical traffic pattern
Balanced (default) Models within a small quality range of the best model for the prompt (for example, 1–2%) The most cost-effective model in that range Spread across the pool, based on prompt complexity
Cost Models within a wider quality band (for example, 5–6%) The most cost-effective model in that band Mostly smaller, lower-cost models, with larger models only when a prompt needs them
Quality All eligible models The highest-quality model for the prompt, regardless of cost Mostly frontier and high-capability models

Start most workloads with Balanced. Use Cost for high-volume, budget-sensitive workloads, such as content classification or simple Q&A. Use Quality for critical tasks, such as legal review, medical summaries, or complex reasoning.

Where the routing mode comes from

Scope How the mode is set Learn more
Your deployment You choose the mode when you deploy, or change it later. Changes take up to five minutes to take effect. Choose a routing mode
Auto Balanced, unless an admin sets properties.instant.routing.mode on the Foundry resource. Configure Auto for your resource
One request (preview) routing_config.mode overrides the mode for that request. Choose a routing mode for one request

With the Foundry-Features: ModelRouterControls=V1Preview header, the response reports the mode that model router applied in model_selection_details.model_router_details.mode. To see how a mode distributes your traffic, check Requests by model on your deployment's Monitor tab.

Model pool and model subsets

The model pool is the set of models that a model router can route to. As of November 11, 2026, the pool for version 2025-11-18 has 43 models from OpenAI, Anthropic, xAI, DeepSeek, Meta, and Fireworks. For the list, see Supported models.

  • Regional availability. In each region, model router routes only across the pool models that are available in that region.
  • Claude models. Model router routes to a Claude model only when that model is deployed in the same Foundry resource, with a deployment type that matches the model router deployment. For steps, see Add Claude models.
  • Fireworks models. Model router routes to a Fireworks model only when Fireworks is enabled for your Azure subscription. For steps, see Enable Fireworks on Foundry.
  • Pool changes. A deployment that uses the full pool gets new models automatically, and retired models leave the pool. A model subset doesn't get new models; you add them yourself.
  • Pool models only. Model router routes only to models in its pool. Your own deployment of a pool model can serve requests for that model, as described in How model router chooses capacity, but other deployments don't join the pool.
  • The model field selects model router. Set model to your deployment name or to auto. A request with model set to another model's name doesn't go through model router.

A model subset is the set of models you choose to route to. Select it with the Route to a subset of models option in the Foundry portal or with properties.routing.models in a deployment request. When you choose a subset, model router recalibrates its routing decisions to optimize across those models.

  • Your subset is your failover set. Failover stays inside the subset, so prompts aren't processed by models you didn't approve. Choose at least two models to keep failover.
  • Context window. The effective context window is the smallest context window among the models that model router can route to. For long prompts, choose a subset of models with large context windows.
  • Auto. An admin can set a subset for Auto on the whole Foundry resource with properties.instant.routing.models.
  • One request (preview). routing_config.models limits routing to the models you list, and a single model pins the request to that model. It can only narrow routing: it never adds a model that the deployment's model subset or Auto's model list excludes.

For steps, see Route to a subset of models and Limit a request to specific models.

How model router chooses capacity

After model router selects a model, it decides where that model runs. Your own deployments of that model in the same Foundry resource come first (preview). Model router moves down this order only when it has no deployment at a step or the deployment returns a retryable error, such as HTTP 429:

  1. Your provisioned throughput deployment, for OpenAI models. If the selected model is an OpenAI model, with format OpenAI in the model pool table, and the resource has a Global Provisioned, Data Zone Provisioned, or Regional Provisioned deployment of its exact name and version, that deployment serves the request on the provisioned throughput units (PTUs) that you already pay for.
  2. Your pay-as-you-go deployment. If the resource has a Standard, Global Standard, or Data Zone Standard deployment of the selected model's exact name and version, that deployment serves the request. This step applies to every model provider in the pool.
  3. Router-managed capacity. Model router uses capacity that it provides for the selected model.
  4. The next-ranked model. If router-managed capacity also returns a retryable error, model router fails over to the next-ranked model, which follows the same order.

These rules also apply:

  • Data Zone Standard stays in the data zone. When your own deployments serve requests for a Data Zone Standard model router deployment, model router uses only your Data Zone Provisioned and Data Zone Standard deployments, so data stays in the data zone. Auto processes data as Global Standard.
  • There's nothing to configure. To make a deployment eligible, deploy the exact model name and version from the model pool table in the same resource. For details, see Use your own deployments for routed requests.
  • Your deployment's capacity and pricing apply. A request that your deployment serves uses that deployment's capacity: its PTUs, or its quota and rate limits. On a provisioned deployment, you pay only the model router routing fee, because you already pay for the PTUs by the hour. On a pay-as-you-go deployment, you pay that deployment's usage plus the routing fee. Your model router deployment's own limits still apply to every request it routes. For details, see Quota when your deployments serve requests.

How parameters and reasoning effort are handled

Models support different parameters, so model router checks the request at two points: before ranking, to decide which models are eligible, and after it selects a model, to fit the request to that model.

  • Before ranking, always. Model router excludes models that can't serve what the request needs, such as function tools, JSON schema output, or image input.
  • After selection, by default. Model router drops optional parameters that the selected model doesn't support, such as temperature, top_p, presence_penalty, frequency_penalty, logit_bias, logprobs, and stop on reasoning models, or reasoning_effort on non-reasoning models. It can also adapt the request, for example by sending a developer message as a system message to a model that doesn't support the developer role. The request doesn't fail.

To make every parameter mandatory, set routing_config.require_parameters to true with per-request controls (preview). The parameter check then moves before ranking:

routing_config.require_parameters When model router checks parameters A parameter the model doesn't support
false (default) After it selects a model Dropped or adapted, and the request continues.
true (preview) Before ranking, as part of eligibility The model is excluded. If no model is left, the request fails with HTTP 400, and no model is called.

For example, a request that sets temperature might go to a reasoning model that doesn't accept it, with temperature dropped. With require_parameters set to true, the same request goes only to models that accept temperature. For details, see How model router handles parameters and Require support for every parameter.

Reasoning effort

Send reasoning effort in each API's native field. The routing_config object doesn't have a reasoning-effort field.

API Reasoning effort field
Chat Completions reasoning_effort
Responses reasoning.effort
Messages output_config.effort
  • Model router passes the value to the selected model when that model supports it. It drops the value for a model that doesn't, such as a non-reasoning model. When you set routing_config.require_parameters to true, only models that support reasoning effort are eligible.
  • Some values, such as minimal or xhigh, are supported by only some reasoning models. Model router doesn't route a request to a reasoning model that rejects the value you sent.
  • model_selection_details doesn't report the effort. For OpenAI reasoning models, the Responses API reports the effort used in the response's reasoning.effort. Claude models don't report it.

Session affinity and prompt caching

Prompt caching works through the underlying models. A cache hit happens when the same model serves consecutive requests with overlapping prompt prefixes. Because model router can choose a different model for each turn, session affinity (preview) raises the odds of a cache hit by keeping related requests on one model.

Session affinity requires the Foundry-Features: ModelRouterControls=V1Preview header. Without the header, model router routes each request independently. With the header, the default strategy is adaptive, even if you don't send routing_config.

Strategy Behavior
adaptive (default) Weighs the cache against quality on every turn. When a different model ranks first, model router keeps the previous model only if the earlier part of the conversation is unchanged and that model's prompt cache is likely still warm, and if the previous model's expected quality is within your routing mode's quality range of the top-ranked model. Otherwise, it switches to the top-ranked model. In Quality mode, it keeps the previous model only when that model's expected quality matches the top-ranked model.
sticky Tries the previous model first on every turn while it's eligible, without the quality check. The association expires after 30 minutes without a successful request.
none No affinity for the request. Model router uses normal routing.

Model router recognizes a conversation by a session ID that you send, by the conversation or previous_response_id in a Responses API request, or by a fingerprint of the earlier messages in the request. It stores only the association between the conversation and the model, not the conversation content.

Policy, safety, capability, quota, and availability rules, and failover, take precedence over affinity. Affinity is best-effort and never fails a request. To set it up, see Keep a conversation on one model. To learn how caching works, see Prompt caching. To track cache reuse, check Cache hit rate on your deployment's Monitor tab.

Failover

Failover is on by default, and there's nothing to configure. If the selected model fails with a retryable error, model router retries the request on the next-best eligible model, within your routing mode and model subset.

  • Capacity first, then the next model. If your own deployment of the selected model returns a retryable error, such as HTTP 429, model router first tries the rest of the capacity order for the same model: your pay-as-you-go deployment after your provisioned throughput deployment, and then router-managed capacity. Only then does it move to the next-ranked model.
  • Failover stays in your subset. Prompts aren't processed by models outside your model subset. A single-model subset, or a request pinned to one model, has no other model to fail over to.
  • No switching within a response. In the Responses API, after a response starts, model router doesn't switch models within that response.
  • Affinity doesn't block failover. Failover takes precedence over session affinity.

To check whether failover happened, send the Foundry-Features: ModelRouterControls=V1Preview header and inspect model_selection_details.model_router_details.routing_trace. More than one attempt means that failover happened. For a deployment, the Monitor tab also shows Fallback rate over time and Fallback by source model. For details, see Inspect routing attempts and failover.

When to use model router

Model router fits most workloads that send a mix of prompts. A direct model deployment fits when every request must go to one specific model. Many apps use both.

Choose model router when

  • Your prompts vary. A mix of simple and complex prompts benefits most. Each prompt goes to a model suited to it, with quality comparable to a single general-purpose model.
  • Cost matters. Smaller, lower-cost models handle simple prompts, and frontier models are reserved for complex ones.
  • Responsiveness matters. In high-traffic, user-facing apps, such as chat and customer support, simpler prompts go to faster models. That can lower average latency compared with always calling a frontier model.
  • You want one endpoint. You manage one model name, one rate limit, and one content filter.
  • You're building agents. Model router supports tools, and it can pick a fast model for a classification step and a reasoning model for an analysis step.
  • You want automatic failover with no extra configuration.
  • You already pay for capacity. Model router serves the selected model from your own deployments first (preview): provisioned throughput (PTU) for OpenAI models, and then pay-as-you-go. Routed traffic fills capacity that you already have before it uses router-managed capacity.
  • You want less model lifecycle work. With the full model pool, new models join and retired models leave without changes to your code.

Choose a direct model deployment when

  • Every request must use the same model. Model router names the model in each response, but it can choose different models for different prompts.
  • You need audio input. Model router doesn't accept audio.

Combine both approaches

  • Use model router as the default path for general traffic to capture cost savings across most requests.
  • Use direct deployments for specialized, compliance-mandated, or parameter-sensitive workloads.

Within model router, start with Auto for prototypes and light traffic, and move to your own deployment for production. For the tradeoffs, see Choose Auto or your own deployment.

Best practices

  • Evaluate before you move production traffic. Compare model router with your current model on your own prompts for quality, cost, and latency. For guidance, see Evaluate model router for your workload.
  • Start with Balanced, and then tune. Watch the routing distribution on your deployment's Monitor tab. Then move high-volume, budget-sensitive workloads to Cost, and critical reasoning tasks to Quality.
  • Change one routing lever at a time. Keep the dataset, baseline, and application configuration fixed when you change the routing mode or model subset, and then rerun the evaluation. That way, you can attribute changes in quality, cost, and latency to that one change.
  • Use a model subset as your compliance gate. Put the models that your security team approved in the subset. New models don't join a subset without your opt-in. Azure Policy is checked when a deployment is created or updated, not for each request, so the subset is what restricts routing. For details, see Govern model router with Azure Policy.
  • Keep at least two models in a subset so failover still works.
  • Design for the smallest context window, or raise it. If your prompts often exceed the smallest context window in the pool, choose a subset of models that support your context length.
  • Use session affinity for multi-turn conversations (preview) to improve prompt-cache reuse.
  • Use a deployment for production traffic and coding agents. Auto suits prototypes and light traffic. Coding agents send many large requests and need higher rate limits.
  • Log the model field. It's your primary observability signal. Build dashboards on it to track which models handle your traffic.

Practices to avoid

  • Single-model subsets. You lose routing optimization, cost savings, and failover, and you still pay the routing fee. To pin one request to a model, use routing_config.models (preview) instead.
  • Forcing a model with the model field. A request with model set to another model's name bypasses model router. Set model to your deployment name or to auto.

Example

Follow one request

Suppose a support app uses a Global Standard model router deployment with the Balanced routing mode and a model subset of gpt-5.6-sol, gpt-5.6-luna, gpt-6-sol, and claude-sonnet-5. The same Foundry resource has a deployment of claude-sonnet-5 and two deployments of gpt-5.6-luna version 2026-07-09: a Global Provisioned deployment and a Global Standard deployment. The app sends a Chat Completions request that includes a function tool and sets temperature.

  1. Eligibility. All four models are in the subset. claude-sonnet-5 is eligible because it's deployed in the resource. Model router excludes any model that can't call function tools.
  2. Ranking. Suppose gpt-6-sol ranks highest for this prompt, and gpt-5.6-luna falls within the Balanced quality range at a lower cost. Model router selects gpt-5.6-luna. In Quality mode, it would select gpt-6-sol.
  3. Capacity. gpt-5.6-luna is an OpenAI model, and the resource has a Global Provisioned deployment of version 2026-07-09, so that deployment serves the request first. The request uses PTUs that you already pay for, so it adds only the routing fee.
  4. Parameters. If the selected model doesn't support temperature, model router drops it and sends the rest of the request. If the request had set routing_config.require_parameters to true, model router would have excluded models that don't support temperature in step 1 instead, before ranking.
  5. Failover. Suppose your provisioned deployment is fully used and returns HTTP 429. Model router retries gpt-5.6-luna on your Global Standard deployment, where the request counts against that deployment's quota and you pay its usage plus the routing fee. If that attempt also fails with a retryable error, model router tries router-managed capacity for gpt-5.6-luna. If that fails too, it moves to the next-ranked model in the subset, such as gpt-6-sol.
  6. Response. The model field names the model that answered, for example gpt-5.6-luna-2026-07-09. When you send the Foundry-Features: ModelRouterControls=V1Preview header, routing_trace lists the models that model router attempted.

To see how each routing mode spreads your own traffic across models, compare the Requests by model chart on your deployment's Monitor tab before and after you change the mode. For a structured comparison on your own prompts, see Evaluate model router for your workload.