Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
In this article, you learn how to run distributed AI workloads on Azure Kubernetes Service (AKS) using Ray for the compute runtime and Kueue for admission control. This solution covers the full lifecycle from infrastructure provisioning through training, batch inference, and online serving.
Important
Open-source software is mentioned throughout AKS documentation and samples. Software that you deploy is excluded from AKS service-level agreements, limited warranty, and Azure support. As you use open-source technology alongside AKS, consult the support options available from the respective communities and project maintainers to develop a plan.
Microsoft takes responsibility for building the open-source packages that we deploy on AKS. That responsibility includes having complete ownership of the build, scan, sign, validate, and hotfix process, along with control over the binaries in container images. For more information, see Vulnerability management for AKS and AKS support coverage.
What is Ray?
Ray is an open-source framework for scaling AI and Python applications. It provides a unified runtime for distributed training, hyperparameter tuning, batch inference, and model serving, so you can scale workloads across multiple nodes without rewriting application logic.
Ray simplifies distributed computing by handling scheduling, fault tolerance, and resource management. The framework supports machine learning libraries like PyTorch, TensorFlow, and Hugging Face through integrations such as Ray Train, Ray Data, and Ray Serve. For more information, see the Ray GitHub repository.
What is KubeRay?
KubeRay is a Kubernetes operator that manages the lifecycle of Ray clusters. It provides custom resources - RayJob for batch workloads and RayService for persistent serving endpoints - that automate cluster creation, scaling, and teardown. For more information, see the KubeRay GitHub repository.
What is Kueue?
Kueue is a Kubernetes-native job queueing controller that manages workload admission based on resource quotas. Instead of letting every submitted job consume resources immediately, Kueue gates admission against defined quotas - jobs that fit run, jobs that don't wait in line. For a detailed overview of Kueue concepts and configuration, see Kueue overview on AKS.
Solution architecture
This solution combines Kueue for admission control with KubeRay for Ray cluster lifecycle management on AKS. Terraform provisions the infrastructure, Helm installs the operators from Microsoft Container Registry (MCR), and workload identity provides secure access to Azure Blob Storage without stored credentials.
The deployment process consists of three modules:
- Infrastructure — Terraform provisions the AKS cluster with GPU node pools, installs KubeRay and Kueue operators via Helm, creates Azure Blob Storage, and configures workload identity.
- Kueue queues — Kubernetes manifests define ResourceFlavors (CPU and GPU node types), ClusterQueues (quotas and admission policies), and LocalQueues (namespace-scoped submission points).
- Workloads — RayJob and RayService manifests submit AI workloads that Kueue admits based on available quota.
Ray workloads start with suspend: true. Kueue evaluates quota availability and unsuspends admitted workloads, at which point KubeRay creates the Ray cluster and runs the job.
Workload examples
| Example | Type | GPUs | Description |
|---|---|---|---|
| Fine-tune Aurora weather model | RayJob | 1×A100 | LoRA fine-tune of the Microsoft Aurora weather foundation model |
| Train an LLM | RayJob | 4×A100 | Distributed Qwen2.5-7B LoRA fine-tune with LLaMA-Factory |
| Run batch inference | RayJob | 1×A100 | vLLM offline inference with a trained LoRA adapter |
| Serve a model online | RayService | 1×GPU | Persistent HTTP endpoint with Ray Serve |