Priority pay-per-token for Foundation Model APIs

This page describes priority pay-per-token for Foundation Model APIs pay-per-token, including how it behaves and how to send priority requests.

What is priority pay-per-token?

Priority pay-per-token, also referred to as priority mode, is a pay-per-token capability for latency-sensitive, real-time applications. When you send a request with the service_tier parameter set to "priority", Azure Databricks admits the request ahead of standard best-effort pay-per-token traffic on the same model. This keeps availability more consistent during periods of high traffic.

Priority pay-per-token is opt-in per request, so you can send priority and standard requests to the same model. It requires no capacity commitment and is billed at a higher per-token rate than standard pay-per-token requests.

Databricks recommends priority mode when:

  • You need more consistent performance and availability than standard pay-per-token, but aren't ready to commit to dedicated capacity.
  • Your production applications require higher availability.

Supported models

Priority pay-per-token is available for OpenAI and Google Gemini models through ADI Services, provided by Databricks. To access these models in your Azure Databricks environment, see ADI Services.

How priority pay-per-token behaves

Consider the following behavior before you use priority pay-per-token:

  • Consistent performance and availability Priority pay-per-token is designed to keep availability consistent under load. It does not guarantee a specific time-to-first-token or end-to-end latency target. Priority requests target a higher availability than standard pay-per-token requests. Azure Databricks defines availability at a tier as the number of successful requests divided by the total number of admitted requests at that tier.
  • Best-effort capacity. Priority mode does not reserve capacity and there is no capacity commitment. For guaranteed capacity, use provisioned throughput.
  • Fallback to standard pay-per-token. If priority capacity is fully subscribed, requests are served at standard pay-per-token availability and billed at standard pay-per-token rates.
  • Per-token premium. Priority requests are billed at a higher per-token rate than standard pay-per-token requests.

Priority pay-per-token compared to provisioned throughput

Priority pay-per-token and provisioned throughput both target production workloads, but they make different trade-offs:

Consideration Priority pay-per-token Provisioned throughput
Capacity Best-effort, shared. Dedicated, reserved capacity.
Commitment None. Opt in per request. Requires a provisioned endpoint.
Availability More consistent than standard pay-per-token under load. Predictable, based on reserved capacity.
Billing Per-token, at a premium over standard pay-per-token. Based on provisioned model units.

Send a priority request

To use priority mode, set the service_tier parameter to "priority" on a per-request basis. The following example uses the OpenAI client:

from databricks_openai import DatabricksOpenAI

client = DatabricksOpenAI()

response = client.chat.completions.create(
    model="databricks-model-name",
    messages=[
      {
        "role": "user",
        "content": "What is a mixture of experts model?",
      }
    ],
    max_tokens=256,
    service_tier="priority",
)

See Foundation model REST API reference for parameter syntax and Use foundation models for more query options.

Capacity limits

Each cluster supports a maximum total number of tokens per minute across all tenants. Azure Databricks sets a per-tenant limit during onboarding so that a single tenant cannot consume all cluster capacity. If your workload requires more capacity than the per-tenant limit allows, request provisioned throughput.

For more Foundation Model APIs limits, see Foundation Model APIs limits and quotas.

Additional resources

  • Last updated on