CUDA-capable device(s) is/are busy or unavailable.

Fadi Magdy Henain 15 Reputation points
2025-03-17T11:39:25.7766667+00:00

Even though the NVIDIA GPU is detected (nvidia-smi is operational), the CUDA toolkit is installed, and there is free GPU memory, running the vllm serve model results in a CUDA error.

Error Details:

  • Error Message: RuntimeError: CUDA error: CUDA-capable device(s) is/are busy or unavailable. CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
Azure Virtual Machines
Azure Virtual Machines
An Azure service that is used to provision Windows and Linux virtual machines.
9,054 questions
{count} votes

2 answers

Sort by: Most helpful
  1. Fadi Magdy Henain 15 Reputation points
    2025-03-19T11:32:25.0533333+00:00

    Hi,

    The VM were created with NvidiaGpuDriverLinux. and it has the NVidia Driver and Cuda toolkit running. we have tried setting CUDA_LAUNCH_BLOCKING=1 and rebooting the machine with no success.

    nvidia-smi  Wed Mar 12 10:18:20 2025       

    +-----------------------------------------------------------------------------------------+

    | NVIDIA-SMI 550.144.03             Driver Version: 550.144.03     CUDA Version: 12.4     |

    |-----------------------------------------+------------------------+----------------------+

    How to check if it's a hardware problem? Should we recreate the VM?

    Regards,

    Fadi


  2. Anonymous
    2025-03-24T15:08:37.29+00:00

    Hi Fadi Magdy,

    To check for hardware issues, use nvidia-smi for GPU health and run stress tests like cuda-memcheck or third-party tools. Recreating the VM may help resolve configuration problems; ensure GPU drivers and the CUDA toolkit are properly set up as per the Azure AKS GPU Cluster Guide

    https://learn.microsoft.com/en-us/azure/aks/gpu-cluster?tabs=add-ubuntu-gpu-node-pool

    Also, confirm the GPU isn't in a low-power state or restricted by VM settings.

    If you have any further queries, please let us know we are glad to help you.

    If it was helpful, please click "Upvote" on this post to let us know.

    Thank You.

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.