Edit

Use GPUs for compute-intensive workloads in AKS on Azure Local for multi-rack deployments

This article describes how to deploy and configure GPU-enabled node pools in AKS enabled by Azure Arc on Azure Local for multi-rack deployments. Graphical processing units (GPUs) are useful for compute-intensive workloads such as machine learning, deep learning, and AI inferencing.

The Azure Local platform exposes physical GPUs to AKS Arc worker virtual machines (VMs), using discrete device assignment (DDA) with vfio-pci PCI passthrough. Workloads inside the AKS Arc cluster can run directly against the device.

Supported GPU models and VM sizes

AKS on Azure Local for multi-rack deployments supports the following GPU hardware and SKUs:

GPU model Supported VM sizes vCPU Memory (GiB) GPUs per VM
NVIDIA RTX Pro 6000 Blackwell (96 GiB) Standard_NC16_RTX6000Pro_1 16 64 1
NVIDIA RTX Pro 6000 Blackwell (96 GiB) Standard_NC32_RTX6000Pro_1 32 128 1

Note

Multi-GPU SKUs (such as Standard_NC16_RTX6000Pro_2 and Standard_NC32_RTX6000Pro_2) are present in the catalog but aren't yet generally available.

Before you begin

To create a GPU-enabled node pool, make sure the following requirements are met:

  • An Azure Local cluster for multi-rack deployments is deployed and registered with Azure Arc. For more information, see the multi-rack deployment overview.

  • Ensure the custom location of your Azure Local cluster has physical compute nodes with available NVIDIA RTX Pro 6000 GPUs. Your Azure Local administrator can confirm this from a management node by querying the platform cluster for the nvidia.com/nvidia-rtx-pro-6000 resource on compute-role nodes:

    kubectl get nodes -l platform.afo-nc.microsoft.com/role=compute -o json \
  | jq -r '.items[] | .metadata.name as $n
           | .status.allocatable | to_entries[]
           | select(.key | test("nvidia|gpu|vfio"; "i"))
           | "\($n)\t\(.key) = \(.value)"'

    The output lists each compute node and its allocatable GPU-related resources, for example:

    nvidia.com/gpu_vfio = 0
nvidia.com/nvidia-rtx-pro-6000 = 2
nvidia.com/gpu_vfio = 0
nvidia.com/nvidia-rtx-pro-6000 = 2

    At least one compute node must report a nonzero nvidia.com/nvidia-rtx-pro-6000 value, and the sum across compute nodes must be at least the --node-count you plan to request in Step 2. If no nodes appear or all values are 0, contact your administrator or Microsoft Support before proceeding.

  • Install the latest version of Azure CLI and the aksarc extension.

  • Configure either the tenant proxy or the logical network connectivity before proceeding to enable outbound access to NVIDIA software. For BYO proxy instructions, see Add BYO proxy how-to for AKS on Azure Local multi-rack.

  • Ensure the AKS Arc cluster is created with SSH keys at cluster creation time and that you have access to these SSH keys. Steps 3 and 4 in this article require SSH access into each GPU worker VM to install the NVIDIA driver and Container Toolkit, and SSH keys can only be configured during cluster creation.

Note

You must manually install the NVIDIA driver, NVIDIA Container Toolkit, and NVIDIA Kubernetes device plugin so that the GPU is schedulable as nvidia.com/gpu from pod specs. Steps 3–5 in this article walk through each piece; Step 6 verifies end-to-end with a pod workload.

Step 1: List available GPU-enabled VM sizes

After you deploy the Azure Local cluster, confirm that GPU-enabled SKUs are visible on your custom location:

az aksarc vmsize list --custom-location <custom location ID> --resource-group <resource group name> --output table

The output includes the Standard_NC*_RTX6000Pro_* SKUs if you're on this release.

Step 2: Create a cluster with a GPU-enabled node pool

Add GPU node pools to an AKS Arc cluster. The control plane and system node pool use a standard non-GPU VM size. The GPU SKU is only for the user node pool that runs your workloads.

Important

You can't create an AKS Arc cluster with a GPU-enabled SKU as the initial (system) node pool. You must first create the cluster with a standard non-GPU VM size, then add a GPU-enabled user node pool.

  1. Create the AKS Arc cluster (without a GPU node pool). For detailed steps and parameter guidance, see Create Kubernetes clusters using Azure CLI.

  2. Add a GPU-enabled Linux node pool. The following example adds a two-node pool of Standard_NC16_RTX6000Pro_1 VMs (one NVIDIA RTX Pro 6000 GPU per node):

    az aksarc nodepool add \
  --cluster-name <aks cluster name> \
  --resource-group <resource group name> \
  --name <nodepool name> \
  --node-count <node count> \
  --node-vm-size Standard_NC16_RTX6000Pro_1 \
  --os-type Linux

    The platform validates GPU capacity upon command execution. If the cluster doesn't have enough physical GPUs to back the requested node count, the command fails.

  3. (Optional) Connect to the cluster via Azure Arc proxy:

    az connectedk8s proxy --name <aks cluster name> --resource-group <resource group name>

Step 3: Install the NVIDIA driver inside the worker VM

The Azure Local platform delivers the physical GPU to the worker VM through PCI passthrough, but the NVIDIA driver inside the guest operating system isn't preinstalled. The worker VM runs Azure Linux 3, and the supported install path is the Microsoft-signed cuda-open RPM from packages.microsoft.com.

Important

The NVIDIA RTX Pro 6000 (Blackwell architecture) requires the open kernel modules. Use the cuda-open package. The NVIDIA cuda package can't be used because Azure Linux 3 prevents the .run installer from executing due to security enforcement that prevents unsigned module execution.

Repeat the steps in this section on each worker VM in the GPU node pool. To open an in-guest shell, use SSH (see Connect to Windows or Linux worker nodes with SSH).

  1. Confirm the GPU device is visible to the guest:

    lspci -nn | grep -i nvidia

    Expect something like:

    0e:00.0 3D controller [0302]: NVIDIA Corporation Device [10de:2bb5] (rev a1)

  2. Make sure the worker VM has outbound HTTPS access to packages.microsoft.com.

    If your environment requires an HTTPS proxy for outbound traffic, configure tdnf to use it. Replace <proxy-url> with the proxy that's permitted to reach packages.microsoft.com.

    echo 'proxy=<proxy-url>' | sudo tee -a /etc/tdnf/tdnf.conf

    Note

    tdnf reads its proxy from /etc/tdnf/tdnf.conf. The same proxy is required for Step 4 and for any future package updates on the worker VM.

    Verify connectivity to the repository. If you configured a proxy above, pass it explicitly with -x; otherwise omit the -x flag:

    curl -sI -x <proxy-url> https://packages.microsoft.com/azurelinux/3.0/prod/nvidia/x86_64/ | head -1

    Expected response: HTTP/2 200.

  3. Enable the Azure Linux NVIDIA repository. By default, only the base, ms-oss, and ms-non-oss repos are configured - the cuda-open package lives in a separate NVIDIA repo that you must add once:

    sudo tee /etc/yum.repos.d/azurelinux-nvidia.repo > /dev/null <<'EOF'
[azurelinux-official-nvidia]
name=Azure Linux Official NVIDIA 3.0 x86_64
baseurl=https://packages.microsoft.com/azurelinux/3.0/prod/nvidia/x86_64
gpgkey=https://packages.microsoft.com/keys/microsoft.asc
gpgcheck=1
enabled=1
sslverify=1
EOF

sudo tdnf makecache

  4. Install the build prerequisites and the NVIDIA driver in a single tdnf transaction so the kernel package and cuda-open resolve to matching versions. Despite the package name, cuda-open is the driver package with open kernel modules - it doesn't include the CUDA toolkit (nvcc), which is installed separately in Step 4.

    sudo tdnf install -y \
  kernel kernel-devel kernel-headers kernel-drivers-gpu \
  libdrm-devel gcc make glibc-devel \
  cuda-open

    cuda-open is signed by Mariner Trusted Base, so the kernel modules load even with lockdown=integrity enabled.

    Important

    cuda-open is built against a specific kernel ABI (see the version suffix, for example cuda-open-580.105.08-4_6.6.139.1.1.azl3). If the install pulls in a newer kernel than the one currently running, you must reboot before loading the modules, or modprobe nvidia fails with Module nvidia not found in directory /lib/modules/<running-kernel>. Check with uname -r against rpm -q kernel, and run sudo reboot if they differ.

  5. Load the NVIDIA kernel modules and verify with nvidia-smi. The first invocation must run as root so the driver can create the /dev/nvidia* device nodes; subsequent unprivileged calls work because the nodes are created with mode 0666.

    sudo modprobe nvidia
sudo modprobe nvidia_uvm
sudo modprobe nvidia_drm

lsmod | grep '^nvidia'
sudo nvidia-smi

    Successful output from sudo nvidia-smi lists the NVIDIA RTX Pro 6000 with the installed driver version. At this point the driver is fully validated inside the worker VM.

Step 4: Install the NVIDIA Container Toolkit on each GPU worker VM

The driver makes the GPU usable from the worker VM's host OS, but containers still need a runtime hook to see /dev/nvidia* and the matching user-space libraries (libcuda.so, libnvidia-ml.so). The nvidia-container-toolkit package provides that hook and registers it with containerd.

Run on each GPU worker VM:

sudo tdnf install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=containerd --set-as-default
sudo systemctl restart containerd

Verify that the package is installed and that the nvidia runtime is registered with containerd:

rpm -q nvidia-container-toolkit
grep -A3 'runtimes.nvidia' /etc/containerd/config.toml

The grep command should print a [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia] block whose runtime_type is io.containerd.runc.v2 and whose BinaryName points at /usr/bin/nvidia-container-runtime. If the block is missing, re-run sudo nvidia-ctk runtime configure --runtime=containerd --set-as-default and restart containerd.

Step 5: Install the NVIDIA Kubernetes device plugin in the AKS Arc cluster

The device plugin advertises GPUs to the AKS Arc cluster's scheduler as the nvidia.com/gpu resource. Without it, pods that request nvidia.com/gpu: 1 stay Pending.

The following manifest uses the device-plugin image mirrored on Microsoft Container Registry (MCR). You can also find it on GitHub or Nvidia websites.

  1. Connect to your AKS Arc cluster by using az connectedk8s proxy. For step-by-step guidance, see Connect to the cluster. Use the --file option to write a dedicated kubeconfig (for example, ./<aks cluster name>.kubeconfig), leave the proxy running, and point kubectl at that kubeconfig in a second terminal for the remaining steps.

  2. Save the following manifest as nvidia-device-plugin.yaml:

    apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin-ds
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: nvidia-device-plugin-ds
    spec:
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      priorityClassName: "system-node-critical"
      containers:
        - image: mcr.microsoft.com/oss/v2/nvidia/k8s-device-plugin:v0.19.1
          name: nvidia-device-plugin-ctr
          env:
            - name: FAIL_ON_INIT_ERROR
              value: "false"
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: device-plugin
              mountPath: /var/lib/kubelet/device-plugins
      volumes:
        - name: device-plugin
          hostPath:
            path: /var/lib/kubelet/device-plugins

  3. Apply the manifest:

    kubectl apply -f nvidia-device-plugin.yaml

Verify the DaemonSet is running and the GPU is advertised:

kubectl get pods -n kube-system -l name=nvidia-device-plugin-ds -o wide

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.allocatable.nvidia\.com/gpu}{"\n"}{end}'

GPU-enabled nodes should report 1 under nvidia.com/gpu. When they do, continue to Step 6. Check your GPU availability or contact Microsoft Support if it doesn't show expected value.

Step 6: (Optional) Run a GPU-enabled pod workload

To confirm the GPU is usable end-to-end from a Kubernetes pod, deploy a small CUDA workload and check that it completes against the GPU. This step assumes Steps 4 and 5 are complete (container toolkit installed and nvidia.com/gpu advertised on the node).

Note

Run the following kubectl commands in the same shell to which you pointed the kubeconfig file in Step 5.

  1. First, list the nodes in your cluster using kubectl get nodes. The output looks like the following example:

    NAME             STATUS  ROLES                 AGE   VERSION
moc-l9qz36vtxzj  Ready   control-plane,master  6m14s  v1.29.0
moc-lhbkqoncefu  Ready   <none>                3m19s  v1.29.0
moc-li87udi8l9s  Ready   <none>                3m5s   v1.29.0

  2. Use kubectl describe node to confirm that the GPU is schedulable on your worker node. Under the Capacity section, the GPU appears as nvidia.com/gpu: 1:

    kubectl describe <node> | grep -i gpu

    The output displays the GPUs from the worker node and looks similar to the following example:

    Capacity:
  cpu:                16
  ephemeral-storage:  103110508Ki
  memory:             65536000Ki
  nvidia.com/gpu:     1
  pods:               110

  3. Create a new file named gpupod.yaml with the following content:

    apiVersion: v1
kind: Pod
metadata:
  name: cuda-vector-add
spec:
  restartPolicy: OnFailure
  containers:
    - name: cuda-vector-add
      image: "k8s.gcr.io/cuda-vector-add:v0.1"
      resources:
        limits:
          nvidia.com/gpu: 1

  4. Apply the manifest to deploy the sample application:

    kubectl apply -f gpupod.yaml

  5. Verify that the pod started, completed running, and the GPU is assigned:

    kubectl describe pod cuda-vector-add | grep -i gpu

    The previous command should show one GPU assigned:

    nvidia.com/gpu: 1
nvidia.com/gpu: 1

  6. Check the log file of the pod to see if the test passed:

    kubectl logs cuda-vector-add

    The following is example output from the previous command:

    [Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

Next steps

  • Last updated on