Batch inference

These examples run large-scale batch inference on AI Runtime serverless GPUs. They cover a range of modalities, such as LLM text generation and speech-to-text, using frameworks like Ray Data, vLLM, and Transformers.

Example	Interface	Description
Qwen2.5-32B batch inference with Ray Data and vLLM	Notebook	Run multilingual batch inference with eight persistent vLLM replicas of Qwen2.5-32B-Instruct on 8 H100 GPUs and save the results to Unity Catalog.
Batch speech-to-text with Whisper	Notebook	Transcribe audio in batches with OpenAI Whisper large-v3-turbo on serverless GPUs, using libraries pre-installed in AI Runtime.
Batch inference with Qwen2.5-7B-Instruct and vLLM	CLI	Use Ray Data and vLLM to generate responses on an 8x H100 node and write the results to a Unity Catalog volume as Parquet.
Audio transcription and diarization with Whisper and pyannote	CLI	Use Ray Data to distribute speech transcription and speaker diarization across a GPU cluster and write the results to a Unity Catalog volume as Parquet.
Protein embeddings with BioNeMo ESM-2	CLI	Use Ray Data and BioNeMo ESM-2 to embed proteins across an 8x H100 node, then train and evaluate a linear membrane-protein classifier.
Batch image understanding with Qwen3-VL-8B and vLLM	CLI	Use Ray Data and vLLM to caption and classify images on an 8x H100 node and write structured results to Parquet.