Monitor model router in Microsoft Foundry

Model router picks a model for every request, so monitoring answers three questions: which models serve your traffic, what that traffic costs compared with a single model, and what happened on any one request. In this article, you use the Monitor tab in the Foundry portal, Azure Monitor metrics, Microsoft Cost Management, and per-request routing metadata.

  • The Monitor tab on your model router deployment shows requests, tokens, estimated cost, cache hit rate, latency, and fallbacks for the last 24 hours, 7 days, or 30 days.
  • Estimated cost: model router vs. single-model baselines shows what the same traffic would have cost if every request had gone to one model.
  • Every response's model field names the model that served the request, for example gpt-5.6-luna-2026-07-09.
  • With the Foundry-Features: ModelRouterControls=V1Preview header, each response also includes model_selection_details: the routing mode, the ordered model attempts, and the session affinity decision (preview).
  • Per-request routing details work with Auto (model router without a deployment) and with model router deployments, on the Responses, Chat Completions, and Messages APIs.

Important

Features or capabilities marked (preview) in this article are in public preview and are subject to the Supplemental Terms of Use for Microsoft Azure Previews. Previews are provided without a service-level agreement, and Microsoft doesn't recommend them for production workloads. Certain features might not be supported or might have constrained capabilities.

Prerequisites

  • A model router deployment for the Monitor tab, Azure Monitor metrics, and Cost Management. To create one, see Deploy model router.
  • For per-request routing details, your Foundry resource endpoint and API key in the FOUNDRY_RESOURCE_ENDPOINT and FOUNDRY_API_KEY environment variables. To find them, see Get your endpoint and key. These details work with Auto too.
  • For the Python examples, Python 3.10 or later and the OpenAI Python library. Install it with pip install openai. For the REST API examples, curl.
  • For billed costs in the Azure portal, access to Cost Management data. See Assign access to Cost Management data.

Use the Monitor tab

The Monitor tab is a dashboard for one model router deployment. It brings traffic, latency, cost, prompt caching, and reliability together in one place, with no queries to write.

  1. In the Foundry portal, go to Build > Models, and select your model router deployment.
  2. Select the Monitor tab.
  3. To focus the dashboard, select a category filter: Overview, Performance, Cost, Reliability, or Usage.
  4. Select a time range: Last 24 hours, Last 7 days, or Last 30 days.
View or chart What it shows Use it to
Requests, Total tokens, Estimated cost, and Cache hit rate summary cards Totals for the selected time range. Estimated cost also shows the average cost per request. Check volume and spend at a glance.
Requests by model The share of requests routed to each model. Confirm that the model mix fits your routing mode and model subset.
Request latency P50 and P95 latency, time to first token (TTFT), and time to last token (TTLT). The Overall view splits Model processing, Router overhead, and End-to-end latency. The By model view compares the routed models. Separate routing time from model time, and find slow models.
Estimated cost: model router vs. single-model baselines What the same traffic would have cost if every request had gone to one model that you select. Quantify savings. See Compare cost with single-model baselines.
Cache hit rate over time and Cache hit rate by model The share of input tokens served from prompt cache. Only models that report cache usage appear. Check whether prompt caching pays off. To raise cache reuse in conversations, see Keep a conversation on one model.
Request success rate over time The share of requests that succeed. Spot error spikes.
Fallback rate over time and Fallback by source model How often model router retried a request on another model after a service error, and which models the retries fell back from. Find models that trigger failover. See Inspect routing attempts and failover.
Requests over time Request volume across the time range. Spot traffic peaks and compare them with your rate limits. See Quotas and limits for model router.

Compare cost with single-model baselines

Estimated cost: model router vs. single-model baselines answers the question every team asks after adopting model router: what would this traffic have cost on one model? It compares model router's estimated cost with what the same traffic would have cost if every request had gone to a baseline model that you select.

  1. On your model router deployment's Monitor tab, select a time range.
  2. In Estimated cost: model router vs. single-model baselines, select one or more baseline models.
  3. Select Chart to compare the costs visually, or Table to see the numbers.
  4. In the table, read Baseline cost and Router savings for each baseline model.

How to read the results:

  • Baseline cost is what the same traffic would have cost if every request had gone to that baseline model.
  • Router savings compares that baseline cost with model router's estimated cost for the same traffic.
  • Both values are estimates, based on token usage and current pricing. For billed amounts, use Cost Management.
  • Choose baselines that match real alternatives: the model that served this workload before model router, and the model you'd deploy if you didn't use model router.
  • A cheaper baseline doesn't guarantee the same quality. Before you move traffic to a single model, compare quality with an evaluation. See Evaluate model router for your workload.

View metrics in Azure Monitor

Azure Monitor collects platform metrics for your Foundry resource, including your model router deployment. Use it when you want model router metrics next to your other Azure resources.

  1. In the Azure portal, go to your Foundry resource.
  2. Select Monitoring > Metrics.
  3. Select a metric, such as Model Requests or Input Tokens, and add a filter on ModelDeploymentName for your model router deployment.
  4. To see the models that model router routed to, select Apply splitting, and split the metric by ModelName.

For the metrics and logs that Foundry deployments emit, see Monitor model deployments in Microsoft Foundry Models.

Track costs in Cost Management

You pay a model router routing fee for each 1 million input tokens, plus the selected model's usage at that model's rates. Microsoft Cost Management shows what you're billed. For the routing fee, see Model router pricing.

  1. In the Azure portal, go to Cost Management > Cost analysis.
  2. Filter by your Foundry resource.
  3. Filter by tag: select the Deployment tag, and set it to your model router deployment name.

When your own deployment of the selected model serves a routed request (preview), you pay the model router routing fee, plus that deployment's usage if it's a pay-as-you-go deployment. A provisioned throughput (PTU) deployment adds no usage charge, because you already pay for its PTUs. For how model router chooses that capacity, see How model router chooses capacity. For more about cost views, see Start using Cost analysis.

Get per-request routing details

Every response names the model that served it in the model field. To see how model router got there, send the Foundry-Features: ModelRouterControls=V1Preview header (preview). Each response then includes model_selection_details.model_router_details.

  • The header works on all five endpoints: Responses and Chat Completions on the resource and project endpoints, and Messages on the resource endpoint. It works with Auto and with model router deployments.
  • The same header turns on per-request controls and session affinity. When you send the header, session affinity uses the adaptive strategy by default, even when you don't send routing_config. For details, see Control model router routing per request.
  • The response doesn't show whether your own deployment or router-managed capacity served the request.
Field What it tells you
model The model that served the response, including its version.
model_selection_details.model_router_details.mode The routing mode applied to the request: balanced, quality, or cost.
model_selection_details.model_router_details.routing_trace The ordered model attempts, with the HTTP status, any error, and the reported latency. More than one attempt means failover happened.
model_selection_details.model_router_details.session_affinity The session affinity strategy, how model router identified the conversation, and whether it kept or switched the model.

During the preview, model_selection_details and each field in it are optional. Parse them defensively, tolerate new fields, and don't infer routing or affinity details when a field is absent.

Request routing metadata

The OpenAI Python library returns model_selection_details as an extension field, so read it from response.model_extra.

import os
from openai import OpenAI

client = OpenAI(
    base_url=f"{os.environ['FOUNDRY_RESOURCE_ENDPOINT']}/openai/v1/",
    api_key=os.environ["FOUNDRY_API_KEY"],
    # The preview header adds model_selection_details to each response.
    default_headers={"Foundry-Features": "ModelRouterControls=V1Preview"},
)

response = client.responses.create(
    model="auto",  # Or your model router deployment name
    input="In one sentence, name the most popular tourist destination in Seattle.",
)

details = (response.model_extra or {}).get("model_selection_details") or {}  # Optional during preview
router_details = details.get("model_router_details") or {}

print(f"Model: {response.model}")
print(f"Routing mode: {router_details.get('mode', 'not reported')}")
print(f"Session affinity: {router_details.get('session_affinity', 'not reported')}")
for entry in router_details.get("routing_trace", []):
    print(f"Attempts: {[attempt.get('model') for attempt in entry.get('attempts', [])]}")

The following example shows the metadata for a later turn in a conversation:

{
  "model": "gpt-5.6-luna-2026-07-09",
  "model_selection_details": {
    "model_router_details": {
      "mode": "balanced",
      "session_affinity": {
        "strategy": "adaptive",
        "source": "derived_fingerprint",
        "decision": "retain"
      },
      "routing_trace": [
        {
          "latency_ms": 840,
          "attempts": [
            { "model": "gpt-5.6-luna", "result": { "status": 200 } }
          ]
        }
      ]
    }
  }
}

Read the metadata from each API

  • Responses: read model_selection_details from the response object. When you stream, read it from the response object in the response.completed event.
  • Chat Completions: read model_selection_details from the chat completion object. When you stream, read it from the final chunk, which has no choices.
  • Messages: read model_selection_details from the message object. When you stream, read it from the message_delta event.

The following Python example streams a Responses API request and reads the metadata from the response.completed event:

import os
from openai import OpenAI

client = OpenAI(
    base_url=f"{os.environ['FOUNDRY_RESOURCE_ENDPOINT']}/openai/v1/",
    api_key=os.environ["FOUNDRY_API_KEY"],
    default_headers={"Foundry-Features": "ModelRouterControls=V1Preview"},
)

stream = client.responses.create(model="auto", input="Write a haiku about routing.", stream=True)
for event in stream:
    if event.type == "response.output_text.delta":
        print(event.delta, end="", flush=True)
    elif event.type == "response.completed":
        # The routing metadata arrives with the completed response.
        details = (event.response.model_extra or {}).get("model_selection_details") or {}
        print(f"\nModel: {event.response.model}")
        print(f"Routing details: {details.get('model_router_details', 'not reported')}")

Inspect routing attempts and failover

When the selected model fails with a retryable error, model router retries the request on the next-best eligible model, within your routing mode and model subset. The routing_trace array records each attempt in order, so it shows exactly when failover happened.

  • Chat Completions returns one routing_trace entry. Responses can return one entry per output item, and each entry has an output_id.
  • Each entry has latency_ms, the latency reported for that entry, and an ordered attempts array.
  • Each attempt has the model name and result.status. A failed attempt can also include result.error.code and result.error.message.
  • More than one attempt in an entry means failover happened.

The following illustrative trace shows a failed attempt followed by a successful one:

{
  "model": "gpt-5.6-sol-2026-07-09",
  "model_selection_details": {
    "model_router_details": {
      "mode": "balanced",
      "routing_trace": [
        {
          "latency_ms": 19,
          "attempts": [
            { "model": "gpt-6-sol", "result": { "status": 429 } },
            { "model": "gpt-5.6-sol", "result": { "status": 200 } }
          ]
        }
      ]
    }
  }
}

The following Python example sends a Chat Completions request and prints each attempt:

import os
from openai import OpenAI

client = OpenAI(
    base_url=f"{os.environ['FOUNDRY_RESOURCE_ENDPOINT']}/openai/v1/",
    api_key=os.environ["FOUNDRY_API_KEY"],
    default_headers={"Foundry-Features": "ModelRouterControls=V1Preview"},
)
response = client.chat.completions.create(
    model="auto",  # Or your model router deployment name
    messages=[{"role": "user", "content": "In one sentence, name the most popular tourist destination in Seattle."}],
)

details = (response.model_extra or {}).get("model_selection_details") or {}
router_details = details.get("model_router_details") or {}
print(f"Model: {response.model}")
print(f"Routing mode: {router_details.get('mode', 'not reported')}")

for number, entry in enumerate(router_details.get("routing_trace", []), start=1):
    print(f"Routing entry {number} (latency: {entry.get('latency_ms', 'not reported')} ms)")
    for attempt_number, attempt in enumerate(entry.get("attempts", []), start=1):
        result = attempt.get("result", {})
        status = result.get("status", "unknown")
        outcome = "served" if isinstance(status, int) and 200 <= status < 300 else "failed"
        print(f"  Attempt {attempt_number}: {attempt.get('model', 'unknown')} - HTTP {status} ({outcome})")
        if result.get("error"):
            print(f"    Error: {result['error'].get('code')} - {result['error'].get('message')}")

For the illustrative trace, the code prints:

Model: gpt-5.6-sol-2026-07-09
Routing mode: balanced
Routing entry 1 (latency: 19 ms)
  Attempt 1: gpt-6-sol - HTTP 429 (failed)
  Attempt 2: gpt-5.6-sol - HTTP 200 (served)

Requests don't always include multiple attempts, and a failed attempt can omit error. Model names, statuses, attempt counts, and latency vary by request.

To see failover across all traffic, use Fallback rate over time and Fallback by source model on the Monitor tab. If you route to a model subset, include at least two models so that failover has somewhere to go. See Route to a subset of models.

Interpret session affinity metadata

Session affinity (preview) keeps related requests on one model, which raises the odds of prompt-cache hits. With the Foundry-Features: ModelRouterControls=V1Preview header, it's on by default with the adaptive strategy. Without the header, each request is routed independently, and the response has no affinity metadata. To choose a strategy or send a session ID, see Keep a conversation on one model.

The session_affinity object reports what happened:

Field Value Meaning
strategy adaptive The default. Model router keeps the conversation on the previous model while that model's prompt cache is likely still warm and its expected quality for the new turn is close to the best-ranked model for your routing mode. Otherwise, it routes to the better model.
strategy sticky Model router tries the previous model first on every turn while that model is eligible. The association expires after 30 minutes without a successful request.
strategy none No affinity for this request. The response omits source and decision.
source session_id_payload The session ID came from routing_config.session_affinity.session_id in the request body.
source session_id_header The session ID came from the x-ms-session-id request header.
source derived_fingerprint Model router derived the conversation from the request itself, because the request had no valid session ID.
source not_available No conversation identifier was available.
decision initialize The first turn, or a new association.
decision retain The same model as the previous turn served this turn.
decision switch A different model served this turn.
  • The response never returns the session ID itself.
  • Earlier previews reported the strategy in a field named mode. Read strategy in new code.
  • Session affinity is best-effort and never fails a request. If affinity lookup or storage isn't available, model router routes the request normally and omits the session_affinity object. Don't infer a decision when the object or its decision field is absent.
  • Model router stores only the model association, not your conversation content.

Diagnose an affinity switch

Policy, safety, capability, quota, and availability rules, and failover, take precedence over affinity. When decision is switch, read routing_trace with it to find out why:

{
  "model": "gpt-5.6-sol-2026-07-09",
  "model_selection_details": {
    "model_router_details": {
      "mode": "balanced",
      "session_affinity": {
        "strategy": "sticky",
        "source": "session_id_payload",
        "decision": "switch"
      },
      "routing_trace": [
        {
          "latency_ms": 51,
          "attempts": [
            { "model": "gpt-6-sol", "result": { "status": 429 } },
            { "model": "gpt-5.6-sol", "result": { "status": 200 } }
          ]
        }
      ]
    }
  }
}
  • A failed attempt on the previous model comes first: failover caused the switch. In this illustrative example, the associated model, gpt-6-sol, returned a retryable status, so model router failed over to gpt-5.6-sol.
  • No failed attempt, with sticky: the previous model wasn't eligible for this request, for example because of a policy, capability, quota, or availability rule.
  • No failed attempt, with adaptive: the previous model's prompt cache was likely cold, or its expected quality for the new turn wasn't close enough to the best-ranked model.