Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Model router picks a model for every request, so monitoring answers three questions: which models serve your traffic, what that traffic costs compared with a single model, and what happened on any one request. In this article, you use the Monitor tab in the Foundry portal, Azure Monitor metrics, Microsoft Cost Management, and per-request routing metadata.
- The Monitor tab on your model router deployment shows requests, tokens, estimated cost, cache hit rate, latency, and fallbacks for the last 24 hours, 7 days, or 30 days.
- Estimated cost: model router vs. single-model baselines shows what the same traffic would have cost if every request had gone to one model.
- Every response's
modelfield names the model that served the request, for examplegpt-5.6-luna-2026-07-09. - With the
Foundry-Features: ModelRouterControls=V1Previewheader, each response also includesmodel_selection_details: the routing mode, the ordered model attempts, and the session affinity decision (preview). - Per-request routing details work with Auto (model router without a deployment) and with model router deployments, on the Responses, Chat Completions, and Messages APIs.
Important
Features or capabilities marked (preview) in this article are in public preview and are subject to the Supplemental Terms of Use for Microsoft Azure Previews. Previews are provided without a service-level agreement, and Microsoft doesn't recommend them for production workloads. Certain features might not be supported or might have constrained capabilities.
Prerequisites
- A model router deployment for the Monitor tab, Azure Monitor metrics, and Cost Management. To create one, see Deploy model router.
- For per-request routing details, your Foundry resource endpoint and API key in the
FOUNDRY_RESOURCE_ENDPOINTandFOUNDRY_API_KEYenvironment variables. To find them, see Get your endpoint and key. These details work with Auto too. - For the Python examples, Python 3.10 or later and the OpenAI Python library. Install it with
pip install openai. For the REST API examples, curl. - For billed costs in the Azure portal, access to Cost Management data. See Assign access to Cost Management data.
Use the Monitor tab
The Monitor tab is a dashboard for one model router deployment. It brings traffic, latency, cost, prompt caching, and reliability together in one place, with no queries to write.
- In the Foundry portal, go to Build > Models, and select your model router deployment.
- Select the Monitor tab.
- To focus the dashboard, select a category filter: Overview, Performance, Cost, Reliability, or Usage.
- Select a time range: Last 24 hours, Last 7 days, or Last 30 days.
| View or chart | What it shows | Use it to |
|---|---|---|
| Requests, Total tokens, Estimated cost, and Cache hit rate summary cards | Totals for the selected time range. Estimated cost also shows the average cost per request. | Check volume and spend at a glance. |
| Requests by model | The share of requests routed to each model. | Confirm that the model mix fits your routing mode and model subset. |
| Request latency | P50 and P95 latency, time to first token (TTFT), and time to last token (TTLT). The Overall view splits Model processing, Router overhead, and End-to-end latency. The By model view compares the routed models. | Separate routing time from model time, and find slow models. |
| Estimated cost: model router vs. single-model baselines | What the same traffic would have cost if every request had gone to one model that you select. | Quantify savings. See Compare cost with single-model baselines. |
| Cache hit rate over time and Cache hit rate by model | The share of input tokens served from prompt cache. Only models that report cache usage appear. | Check whether prompt caching pays off. To raise cache reuse in conversations, see Keep a conversation on one model. |
| Request success rate over time | The share of requests that succeed. | Spot error spikes. |
| Fallback rate over time and Fallback by source model | How often model router retried a request on another model after a service error, and which models the retries fell back from. | Find models that trigger failover. See Inspect routing attempts and failover. |
| Requests over time | Request volume across the time range. | Spot traffic peaks and compare them with your rate limits. See Quotas and limits for model router. |
Compare cost with single-model baselines
Estimated cost: model router vs. single-model baselines answers the question every team asks after adopting model router: what would this traffic have cost on one model? It compares model router's estimated cost with what the same traffic would have cost if every request had gone to a baseline model that you select.
- On your model router deployment's Monitor tab, select a time range.
- In Estimated cost: model router vs. single-model baselines, select one or more baseline models.
- Select Chart to compare the costs visually, or Table to see the numbers.
- In the table, read Baseline cost and Router savings for each baseline model.
How to read the results:
- Baseline cost is what the same traffic would have cost if every request had gone to that baseline model.
- Router savings compares that baseline cost with model router's estimated cost for the same traffic.
- Both values are estimates, based on token usage and current pricing. For billed amounts, use Cost Management.
- Choose baselines that match real alternatives: the model that served this workload before model router, and the model you'd deploy if you didn't use model router.
- A cheaper baseline doesn't guarantee the same quality. Before you move traffic to a single model, compare quality with an evaluation. See Evaluate model router for your workload.
View metrics in Azure Monitor
Azure Monitor collects platform metrics for your Foundry resource, including your model router deployment. Use it when you want model router metrics next to your other Azure resources.
- In the Azure portal, go to your Foundry resource.
- Select Monitoring > Metrics.
- Select a metric, such as Model Requests or Input Tokens, and add a filter on
ModelDeploymentNamefor your model router deployment. - To see the models that model router routed to, select Apply splitting, and split the metric by
ModelName.
For the metrics and logs that Foundry deployments emit, see Monitor model deployments in Microsoft Foundry Models.
Track costs in Cost Management
You pay a model router routing fee for each 1 million input tokens, plus the selected model's usage at that model's rates. Microsoft Cost Management shows what you're billed. For the routing fee, see Model router pricing.
- In the Azure portal, go to Cost Management > Cost analysis.
- Filter by your Foundry resource.
- Filter by tag: select the Deployment tag, and set it to your model router deployment name.
When your own deployment of the selected model serves a routed request (preview), you pay the model router routing fee, plus that deployment's usage if it's a pay-as-you-go deployment. A provisioned throughput (PTU) deployment adds no usage charge, because you already pay for its PTUs. For how model router chooses that capacity, see How model router chooses capacity. For more about cost views, see Start using Cost analysis.
Get per-request routing details
Every response names the model that served it in the model field. To see how model router got there, send the Foundry-Features: ModelRouterControls=V1Preview header (preview). Each response then includes model_selection_details.model_router_details.
- The header works on all five endpoints: Responses and Chat Completions on the resource and project endpoints, and Messages on the resource endpoint. It works with Auto and with model router deployments.
- The same header turns on per-request controls and session affinity. When you send the header, session affinity uses the
adaptivestrategy by default, even when you don't sendrouting_config. For details, see Control model router routing per request. - The response doesn't show whether your own deployment or router-managed capacity served the request.
| Field | What it tells you |
|---|---|
model |
The model that served the response, including its version. |
model_selection_details.model_router_details.mode |
The routing mode applied to the request: balanced, quality, or cost. |
model_selection_details.model_router_details.routing_trace |
The ordered model attempts, with the HTTP status, any error, and the reported latency. More than one attempt means failover happened. |
model_selection_details.model_router_details.session_affinity |
The session affinity strategy, how model router identified the conversation, and whether it kept or switched the model. |
During the preview, model_selection_details and each field in it are optional. Parse them defensively, tolerate new fields, and don't infer routing or affinity details when a field is absent.
Request routing metadata
The OpenAI Python library returns model_selection_details as an extension field, so read it from response.model_extra.
import os
from openai import OpenAI
client = OpenAI(
base_url=f"{os.environ['FOUNDRY_RESOURCE_ENDPOINT']}/openai/v1/",
api_key=os.environ["FOUNDRY_API_KEY"],
# The preview header adds model_selection_details to each response.
default_headers={"Foundry-Features": "ModelRouterControls=V1Preview"},
)
response = client.responses.create(
model="auto", # Or your model router deployment name
input="In one sentence, name the most popular tourist destination in Seattle.",
)
details = (response.model_extra or {}).get("model_selection_details") or {} # Optional during preview
router_details = details.get("model_router_details") or {}
print(f"Model: {response.model}")
print(f"Routing mode: {router_details.get('mode', 'not reported')}")
print(f"Session affinity: {router_details.get('session_affinity', 'not reported')}")
for entry in router_details.get("routing_trace", []):
print(f"Attempts: {[attempt.get('model') for attempt in entry.get('attempts', [])]}")
The following example shows the metadata for a later turn in a conversation:
{
"model": "gpt-5.6-luna-2026-07-09",
"model_selection_details": {
"model_router_details": {
"mode": "balanced",
"session_affinity": {
"strategy": "adaptive",
"source": "derived_fingerprint",
"decision": "retain"
},
"routing_trace": [
{
"latency_ms": 840,
"attempts": [
{ "model": "gpt-5.6-luna", "result": { "status": 200 } }
]
}
]
}
}
}
Read the metadata from each API
- Responses: read
model_selection_detailsfrom the response object. When you stream, read it from theresponseobject in theresponse.completedevent. - Chat Completions: read
model_selection_detailsfrom the chat completion object. When you stream, read it from the final chunk, which has nochoices. - Messages: read
model_selection_detailsfrom the message object. When you stream, read it from themessage_deltaevent.
The following Python example streams a Responses API request and reads the metadata from the response.completed event:
import os
from openai import OpenAI
client = OpenAI(
base_url=f"{os.environ['FOUNDRY_RESOURCE_ENDPOINT']}/openai/v1/",
api_key=os.environ["FOUNDRY_API_KEY"],
default_headers={"Foundry-Features": "ModelRouterControls=V1Preview"},
)
stream = client.responses.create(model="auto", input="Write a haiku about routing.", stream=True)
for event in stream:
if event.type == "response.output_text.delta":
print(event.delta, end="", flush=True)
elif event.type == "response.completed":
# The routing metadata arrives with the completed response.
details = (event.response.model_extra or {}).get("model_selection_details") or {}
print(f"\nModel: {event.response.model}")
print(f"Routing details: {details.get('model_router_details', 'not reported')}")
Inspect routing attempts and failover
When the selected model fails with a retryable error, model router retries the request on the next-best eligible model, within your routing mode and model subset. The routing_trace array records each attempt in order, so it shows exactly when failover happened.
- Chat Completions returns one
routing_traceentry. Responses can return one entry per output item, and each entry has anoutput_id. - Each entry has
latency_ms, the latency reported for that entry, and an orderedattemptsarray. - Each attempt has the
modelname andresult.status. A failed attempt can also includeresult.error.codeandresult.error.message. - More than one attempt in an entry means failover happened.
The following illustrative trace shows a failed attempt followed by a successful one:
{
"model": "gpt-5.6-sol-2026-07-09",
"model_selection_details": {
"model_router_details": {
"mode": "balanced",
"routing_trace": [
{
"latency_ms": 19,
"attempts": [
{ "model": "gpt-6-sol", "result": { "status": 429 } },
{ "model": "gpt-5.6-sol", "result": { "status": 200 } }
]
}
]
}
}
}
The following Python example sends a Chat Completions request and prints each attempt:
import os
from openai import OpenAI
client = OpenAI(
base_url=f"{os.environ['FOUNDRY_RESOURCE_ENDPOINT']}/openai/v1/",
api_key=os.environ["FOUNDRY_API_KEY"],
default_headers={"Foundry-Features": "ModelRouterControls=V1Preview"},
)
response = client.chat.completions.create(
model="auto", # Or your model router deployment name
messages=[{"role": "user", "content": "In one sentence, name the most popular tourist destination in Seattle."}],
)
details = (response.model_extra or {}).get("model_selection_details") or {}
router_details = details.get("model_router_details") or {}
print(f"Model: {response.model}")
print(f"Routing mode: {router_details.get('mode', 'not reported')}")
for number, entry in enumerate(router_details.get("routing_trace", []), start=1):
print(f"Routing entry {number} (latency: {entry.get('latency_ms', 'not reported')} ms)")
for attempt_number, attempt in enumerate(entry.get("attempts", []), start=1):
result = attempt.get("result", {})
status = result.get("status", "unknown")
outcome = "served" if isinstance(status, int) and 200 <= status < 300 else "failed"
print(f" Attempt {attempt_number}: {attempt.get('model', 'unknown')} - HTTP {status} ({outcome})")
if result.get("error"):
print(f" Error: {result['error'].get('code')} - {result['error'].get('message')}")
For the illustrative trace, the code prints:
Model: gpt-5.6-sol-2026-07-09
Routing mode: balanced
Routing entry 1 (latency: 19 ms)
Attempt 1: gpt-6-sol - HTTP 429 (failed)
Attempt 2: gpt-5.6-sol - HTTP 200 (served)
Requests don't always include multiple attempts, and a failed attempt can omit error. Model names, statuses, attempt counts, and latency vary by request.
To see failover across all traffic, use Fallback rate over time and Fallback by source model on the Monitor tab. If you route to a model subset, include at least two models so that failover has somewhere to go. See Route to a subset of models.
Interpret session affinity metadata
Session affinity (preview) keeps related requests on one model, which raises the odds of prompt-cache hits. With the Foundry-Features: ModelRouterControls=V1Preview header, it's on by default with the adaptive strategy. Without the header, each request is routed independently, and the response has no affinity metadata. To choose a strategy or send a session ID, see Keep a conversation on one model.
The session_affinity object reports what happened:
| Field | Value | Meaning |
|---|---|---|
strategy |
adaptive |
The default. Model router keeps the conversation on the previous model while that model's prompt cache is likely still warm and its expected quality for the new turn is close to the best-ranked model for your routing mode. Otherwise, it routes to the better model. |
strategy |
sticky |
Model router tries the previous model first on every turn while that model is eligible. The association expires after 30 minutes without a successful request. |
strategy |
none |
No affinity for this request. The response omits source and decision. |
source |
session_id_payload |
The session ID came from routing_config.session_affinity.session_id in the request body. |
source |
session_id_header |
The session ID came from the x-ms-session-id request header. |
source |
derived_fingerprint |
Model router derived the conversation from the request itself, because the request had no valid session ID. |
source |
not_available |
No conversation identifier was available. |
decision |
initialize |
The first turn, or a new association. |
decision |
retain |
The same model as the previous turn served this turn. |
decision |
switch |
A different model served this turn. |
- The response never returns the session ID itself.
- Earlier previews reported the strategy in a field named
mode. Readstrategyin new code. - Session affinity is best-effort and never fails a request. If affinity lookup or storage isn't available, model router routes the request normally and omits the
session_affinityobject. Don't infer a decision when the object or itsdecisionfield is absent. - Model router stores only the model association, not your conversation content.
Diagnose an affinity switch
Policy, safety, capability, quota, and availability rules, and failover, take precedence over affinity. When decision is switch, read routing_trace with it to find out why:
{
"model": "gpt-5.6-sol-2026-07-09",
"model_selection_details": {
"model_router_details": {
"mode": "balanced",
"session_affinity": {
"strategy": "sticky",
"source": "session_id_payload",
"decision": "switch"
},
"routing_trace": [
{
"latency_ms": 51,
"attempts": [
{ "model": "gpt-6-sol", "result": { "status": 429 } },
{ "model": "gpt-5.6-sol", "result": { "status": 200 } }
]
}
]
}
}
}
- A failed attempt on the previous model comes first: failover caused the switch. In this illustrative example, the associated model,
gpt-6-sol, returned a retryable status, so model router failed over togpt-5.6-sol. - No failed attempt, with
sticky: the previous model wasn't eligible for this request, for example because of a policy, capability, quota, or availability rule. - No failed attempt, with
adaptive: the previous model's prompt cache was likely cold, or its expected quality for the new turn wasn't close enough to the best-ranked model.