CUDA-capable device(s) is/are busy or unavailable.

Question

CUDA-capable device(s) is/are busy or unavailable.

Fadi Magdy Henain 15

Even though the NVIDIA GPU is detected (nvidia-smi is operational), the CUDA toolkit is installed, and there is free GPU memory, running the vllm serve model results in a CUDA error.

Error Details:

Error Message: RuntimeError: CUDA error: CUDA-capable device(s) is/are busy or unavailable. CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.

Anonymous

2025-03-18T14:13:26.3633333+00:00

Hi Fadi Magdy,

The error "CUDA-capable device(s) is/are busy or unavailable" can occur when the GPU is occupied or there is a configuration issue. First, ensure the GPU is not in use by another process by checking with nvidia-smi. Verify that the installed CUDA toolkit version is compatible with your GPU and the software being used. Setting the environment variable CUDA_LAUNCH_BLOCKING=1 may help by providing immediate error reporting. Restarting the system can also reset the GPU state. Make sure the NVIDIA driver version is up-to-date and compatible with the CUDA version. If the issue persists, hardware-related problems might be a cause. Refer to Microsoft's Azure Machine Learning GPU troubleshooting for further information.

If it was helpful, please click "Upvote" on this post to let us know.

Thank You.
Anonymous

2025-03-19T09:27:53.55+00:00

Hi Fadi Magdy, ,

Just checking in to see if you have got a chance to see my response to your question in resolving the issue.

If it was helpful, please click "Upvote" on this post to let us know.

Thank You.
Anonymous

2025-03-20T08:38:07.2166667+00:00

Hi Fadi Magdy,

Thank you for the update. Since the NVIDIA driver and CUDA toolkit are operational, and setting CUDA_LAUNCH_BLOCKING=1 and rebooting did not resolve the issue, you can run a GPU diagnostic test using tools like gpu-burn or monitor GPU utilization with nvidia-smi. Ensure the driver (550.144.03) and CUDA version (12.4) are compatible with the GPU model. Recreating the VM with the NVIDIA GPU Driver Extension for

Linuxhttps://learn.microsoft.com/en-us/azure/virtual-machines/extensions/hpccompute-gpu-linux may help resolve possible misconfigurations.

For a clean setup, follow the Azure N-series GPU driver setup guide https://learn.microsoft.com/en-us/azure/virtual-machines/linux/n-series-driver-setup

If it was helpful, please click "Upvote" on this post to let us know.

Thank You.
Fadi Magdy Henain 15 Reputation points

2025-03-20T09:22:16.9566667+00:00

Hi,

The VM were created with NvidiaGpuDriverLinux. and it has the NVidia Driver and Cuda toolkit running. we have tried setting CUDA_LAUNCH_BLOCKING=1 and rebooting the machine with no success.

nvidia-smi Wed Mar 12 10:18:20 2025

+-----------------------------------------------------------------------------------------+

| NVIDIA-SMI 550.144.03 Driver Version: 550.144.03 CUDA Version: 12.4 |

|-----------------------------------------+------------------------+----------------------+

How to check if it's a hardware problem? Should we recreate the VM?

Regards,

Fadi
Anonymous

2025-03-25T05:40:03+00:00

Hi Fadi Magdy,

Just checking in to see if you have got a chance to see my response to your question in resolving the issue.

If it was helpful, please click "Upvote" on this post to let us know

Thank You.
Anonymous

2025-03-26T08:41:42.7866667+00:00

Hi Fadi Magdy,

I wanted to check if you had the opportunity to review the information which was provided in my previous posted comment.

If it was helpful, please click "Upvote" on this post to let us know

Thank You.

2 answers

Your answer

Anonymous

2025-03-18T14:13:26.3633333+00:00

Hi Fadi Magdy,

The error "CUDA-capable device(s) is/are busy or unavailable" can occur when the GPU is occupied or there is a configuration issue. First, ensure the GPU is not in use by another process by checking with nvidia-smi. Verify that the installed CUDA toolkit version is compatible with your GPU and the software being used. Setting the environment variable CUDA_LAUNCH_BLOCKING=1 may help by providing immediate error reporting. Restarting the system can also reset the GPU state. Make sure the NVIDIA driver version is up-to-date and compatible with the CUDA version. If the issue persists, hardware-related problems might be a cause. Refer to Microsoft's Azure Machine Learning GPU troubleshooting for further information.

If it was helpful, please click "Upvote" on this post to let us know.

Thank You.
Anonymous

2025-03-19T09:27:53.55+00:00

Hi Fadi Magdy, ,

Just checking in to see if you have got a chance to see my response to your question in resolving the issue.

If it was helpful, please click "Upvote" on this post to let us know.

Thank You.
Anonymous

2025-03-20T08:38:07.2166667+00:00

Hi Fadi Magdy,

Thank you for the update. Since the NVIDIA driver and CUDA toolkit are operational, and setting CUDA_LAUNCH_BLOCKING=1 and rebooting did not resolve the issue, you can run a GPU diagnostic test using tools like gpu-burn or monitor GPU utilization with nvidia-smi. Ensure the driver (550.144.03) and CUDA version (12.4) are compatible with the GPU model. Recreating the VM with the NVIDIA GPU Driver Extension for

Linuxhttps://learn.microsoft.com/en-us/azure/virtual-machines/extensions/hpccompute-gpu-linux may help resolve possible misconfigurations.

For a clean setup, follow the Azure N-series GPU driver setup guide https://learn.microsoft.com/en-us/azure/virtual-machines/linux/n-series-driver-setup

If it was helpful, please click "Upvote" on this post to let us know.

Thank You.
Fadi Magdy Henain 15 Reputation points

2025-03-20T09:22:16.9566667+00:00

Hi,

The VM were created with NvidiaGpuDriverLinux. and it has the NVidia Driver and Cuda toolkit running. we have tried setting CUDA_LAUNCH_BLOCKING=1 and rebooting the machine with no success.

nvidia-smi Wed Mar 12 10:18:20 2025

+-----------------------------------------------------------------------------------------+

| NVIDIA-SMI 550.144.03 Driver Version: 550.144.03 CUDA Version: 12.4 |

|-----------------------------------------+------------------------+----------------------+

How to check if it's a hardware problem? Should we recreate the VM?

Regards,

Fadi
Anonymous

2025-03-25T05:40:03+00:00

Hi Fadi Magdy,

Just checking in to see if you have got a chance to see my response to your question in resolving the issue.

If it was helpful, please click "Upvote" on this post to let us know

Thank You.
Anonymous

2025-03-26T08:41:42.7866667+00:00

Hi Fadi Magdy,

I wanted to check if you had the opportunity to review the information which was provided in my previous posted comment.

If it was helpful, please click "Upvote" on this post to let us know

Thank You.

Answer 1

Fadi Magdy Henain 15

Hi,

The VM were created with NvidiaGpuDriverLinux. and it has the NVidia Driver and Cuda toolkit running. we have tried setting CUDA_LAUNCH_BLOCKING=1 and rebooting the machine with no success.

nvidia-smi Wed Mar 12 10:18:20 2025

+-----------------------------------------------------------------------------------------+

| NVIDIA-SMI 550.144.03 Driver Version: 550.144.03 CUDA Version: 12.4 |

|-----------------------------------------+------------------------+----------------------+

How to check if it's a hardware problem? Should we recreate the VM?

Regards,

Fadi

Anonymous

2025-03-24T09:07:27.4466667+00:00

Hi Fadi Magdy,

To check for hardware issues, use nvidia-smi for GPU health and run stress tests like cuda-memcheck or third-party tools. Recreating the VM may help resolve configuration problems; ensure GPU drivers and the CUDA toolkit are properly set up as per the Azure AKS GPU Cluster Guide https://learn.microsoft.com/en-us/azure/aks/gpu-cluster?tabs=add-azure-linux-gpu-node-pool

Also, confirm the GPU isn't in a low-power state or restricted by VM settings.

If it was helpful, please click "Upvote" on this post to let us know.

Thank You.

Answer 2

Hi Fadi Magdy,

To check for hardware issues, use nvidia-smi for GPU health and run stress tests like cuda-memcheck or third-party tools. Recreating the VM may help resolve configuration problems; ensure GPU drivers and the CUDA toolkit are properly set up as per the Azure AKS GPU Cluster Guide

https://learn.microsoft.com/en-us/azure/aks/gpu-cluster?tabs=add-ubuntu-gpu-node-pool

Also, confirm the GPU isn't in a low-power state or restricted by VM settings.

If you have any further queries, please let us know we are glad to help you.

If it was helpful, please click "Upvote" on this post to let us know.

Thank You.

Share via

CUDA-capable device(s) is/are busy or unavailable.

2 answers

nvidia-smi Wed Mar 12 10:18:20 2025

Your answer