Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Important
This feature is in Public Preview.
These examples run large-scale batch inference on AI Runtime serverless GPUs. They cover a range of modalities, such as LLM text generation and speech-to-text, using frameworks like Ray Data, vLLM, and Transformers.
| Example | Interface | Description |
|---|---|---|
| Qwen2.5-32B batch inference with Ray Data and vLLM | Notebook | Run multilingual batch inference with eight persistent vLLM replicas of Qwen2.5-32B-Instruct on 8 H100 GPUs and save the results to Unity Catalog. |
| Batch speech-to-text with Whisper | Notebook | Transcribe audio in batches with OpenAI Whisper large-v3-turbo on serverless GPUs, using libraries pre-installed in AI Runtime. |
| Batch inference with Qwen2.5-7B-Instruct and vLLM | CLI | Use Ray Data and vLLM to generate responses on an 8x H100 node and write the results to a Unity Catalog volume as Parquet. |