Batch inference

Important

This feature is in Public Preview.

These examples run large-scale batch inference on AI Runtime serverless GPUs. They cover a range of modalities, such as LLM text generation and speech-to-text, using frameworks like Ray Data, vLLM, and Transformers.

Example	Interface	Description
Qwen2.5-32B batch inference with Ray Data and vLLM	Notebook	Run multilingual batch inference with eight persistent vLLM replicas of Qwen2.5-32B-Instruct on 8 H100 GPUs and save the results to Unity Catalog.
Batch speech-to-text with Whisper	Notebook	Transcribe audio in batches with OpenAI Whisper large-v3-turbo on serverless GPUs, using libraries pre-installed in AI Runtime.
Batch inference with Qwen2.5-7B-Instruct and vLLM	CLI	Use Ray Data and vLLM to generate responses on an 8x H100 node and write the results to a Unity Catalog volume as Parquet.