After installing nvidia driver on NV12ads A10 and rebooting, VM is inaccessible

Question

After installing nvidia driver on NV12ads A10 and rebooting, VM is inaccessible

塚田真輝 10

First, check if the GPU is attached.

azureuser@i-vm-a10:~$ lspci | grep -i NVIDIA 0002:00:00.0 3D controller: NVIDIA Corporation GA102GL [A10] (rev a1)

Install ubuntu-drivers utility

sudo apt update && sudo apt install -y ubuntu-drivers-common

Install nvidia-driver-550

azureuser@i-vm-a10:~$ sudo apt install -y nvidia-driver-550

reboot

sudo reboot

Accessing VMs via ssh

channel 0: open failed: connect failed: Connection refused
stdio forwarding failed
Connection closed by UNKNOWN port 65535

The following two base images were tried.

NVIDIA GPU-Optimized VMI with vGPU driver
Ubuntu Server 22.04 LTS

It is possible to reboot and re-access the system before putting in the driver.

塚田真輝 10

I gave it an hour or so and then I could access it via ssh

but cannot nvidia-smi

azureuser@i-vm-a10:~$ nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

Nikhil Duserla 8,100 Reputation points Microsoft External Staff Moderator

2024-10-15T11:38:31.8133333+00:00

Hi @塚田真輝,

Did you get a chance to look into the above comment? Let us know if the issue has been resolved.
Nikhil Duserla 8,100 Reputation points Microsoft External Staff Moderator

2024-10-16T12:21:05+00:00

Hi @塚田真輝,

Could you please update me on whether the issue has been resolved or if any help is still required?

1 answer

Your answer

塚田真輝 10 Reputation points

2024-10-12T06:37:53.1966667+00:00

I gave it an hour or so and then I could access it via ssh

but cannot nvidia-smi

azureuser@i-vm-a10:~$ nvidia-smi NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
Nikhil Duserla 8,100 Reputation points Microsoft External Staff Moderator

2024-10-15T11:38:31.8133333+00:00

Hi @塚田真輝,

Did you get a chance to look into the above comment? Let us know if the issue has been resolved.
Nikhil Duserla 8,100 Reputation points Microsoft External Staff Moderator

2024-10-16T12:21:05+00:00

Hi @塚田真輝,

Could you please update me on whether the issue has been resolved or if any help is still required?

Answer 1

Nikhil Duserla 8,100 Microsoft External Staff Moderator

Hi @塚田真輝,

Welcome to the Microsoft Q&A Platform! Thank you for your question and for providing the information.

We understand from your query that you are experiencing an issue while trying to access with nvidia-smi. $ nvidia-smi NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

At this time, you need to check whether the nvidia driver is correctly installed, or update to the latest version of nvidia-drivers.

The Azure NVads A10 v5 VMs only support GRID 14.1(510.73) or higher driver versions. The vGPU driver for the A10 SKU is a unified driver that supports both graphics and compute workloads. Please refer to this: https://learn.microsoft.com/en-us/azure/virtual-machines/linux/n-series-driver-setup#supported-distributions-and-drivers

Uninstall the vGPU driver to start from a clean slate. Reinstall the vGPU driver using the correct installation method for your system. Make sure to follow the official NVIDIA documentation for installing the vGPU driver on Ubuntu Server 22.04 LTS. https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/index.html

If you have any further queries, do let us know.

塚田真輝 10 Reputation points

2024-11-10T06:15:48.99+00:00

thank you for answering @Nikhil Duserla

This problem has not yet been solved. Currently using Standard NC16as T4 v3 instead。

Anyone have any experience with NV12ads A10?
Nikhil Duserla 8,100 Reputation points Microsoft External Staff Moderator

2024-11-11T20:31:58.77+00:00

Hi @塚田真輝,

Thank you for reaching out to us again. I was just checking with you to see if you've gone through the answer provided above.

The Azure NVads A10 v5 VMs only support GRID 14.1(510.73) or higher driver versions. The vGPU driver for the A10 SKU is a unified driver that supports both graphics and compute workloads. Please refer to this: https://learn.microsoft.com/en-us/azure/virtual-machines/linux/n-series-driver-setup#supported-distributions-and-drivers

Uninstall the vGPU driver to start from a clean slate. Reinstall the vGPU driver using the correct installation method for your system. Make sure to follow the official NVIDIA documentation for installing the vGPU driver on Ubuntu Server 22.04 LTS. https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/index.html

Still, if you are facing the same issue, please do let us know.

Share via

After installing nvidia driver on NV12ads A10 and rebooting, VM is inaccessible

1 answer

Your answer