Scale model inference throughput

Increase model traffic through Unity Gateway gradually when launching a workload or switching models. This guidance applies to proprietary and open-weight models provided by Azure Databricks and models accessed through connected external providers.

Increase traffic gradually

Available capacity depends on the model, provider, region, and serving mode. Requests can encounter temporary capacity errors even when traffic stays within the configured rate limits. A quota increase alone does not reserve serving capacity.

To increase throughput:

1. Check the configured service rate limits and the limits for the model or external provider. Estimate input and output token rates and concurrent requests, including representative prompt and response sizes.
2. Start at a known stable traffic level. For a new workload, start with a small representative load.
3. Increase traffic in small steps. Hold each level steady while monitoring request success rate, latency, token usage, queue depth, and throttling or capacity errors. Increase again only when the workload is stable and meets your application's operating targets.
4. If throttling, capacity errors, or latency increase beyond your application's tolerance, pause the ramp and return to the last stable traffic level. Reduce concurrency and queue or defer lower-priority work.

Choose step sizes and observation periods based on your workload and the guidance for the model or provider. Repeat this validation when changing models, regions, or serving capacity.

Handle throttling and capacity errors

Inspect both the HTTP status and the error body from the API you call. Error codes and messages can differ between unified and provider-native APIs.

Response	Meaning and action
429 Too Many Requests	A configured rate limit or downstream throttling prevents the request. Some providers also throttle sharp increases in traffic. Identify the limit from the error body, reduce the request rate, and increase traffic gradually. If the error requires a quota or billing change, resolve that condition before retrying.
503 Service Unavailable	The service or model is temporarily unable to serve the request. This can happen during a traffic increase, including below the configured rate limits. Retry within a bounded budget and pause the ramp if errors persist.
Other provider overload errors	Some provider APIs use other responses, such as 529, for temporary overload. Follow the error guidance for the provider and API you use.

A capacity error alone does not establish that autoscaling is in progress or indicate when capacity becomes available.

Configure retries

• Honor the Retry-After header when the response includes it. Otherwise, retry transient throttling and capacity errors with exponential backoff and random jitter.
• Limit total attempts and elapsed time to fit your application's latency budget. If errors persist, reduce traffic instead of adding more retries.
• Check the SDK's retry behavior before adding application retries. Account for retries across layers so they do not multiply the load.
• Handle errors after a streaming response starts in the stream consumer. Replaying a request after receiving output can duplicate output or work.

For persistent errors, contact your Databricks account team with request IDs, timestamps, model and provider names, observed token rates, concurrency, and the traffic ramp timeline. For connected external providers, also check the provider's service status and account limits.

Batch inference

For large offline workloads with supported models, use AI Functions, which manage parallelization, retries, and scaling. Follow their batch inference guidance instead of applying a separate retry loop to each row.