NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

Question

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

Tianchen Guo 5

Hi Team,

I suddenly encountered this issue last night and I have tried several solutions online but no one worked, including these 2 threads:

https://learn.microsoft.com/en-us/answers/questions/1328794/nvidia-smi-has-failed-because-it-couldnt-communica

https://learn.microsoft.com/en-us/answers/questions/1669319/nvidia-smi-has-failed-because-it-couldnt-communica

Issue: ~$ nvidia-smi

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

uninstall all previous drivers -> follow https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/deployment-history?tabs=azure-portal#deployment-operations-and-error-message -> does not work.
uninstall all previous drivers -> follow https://learn.microsoft.com/en-us/azure/virtual-machines/linux/n-series-driver-setup?wt.mc_id=searchAPI_azureportal_inproduct_rmskilling&sessionId=7ec697615f424d5fb668b3e866efa4e5#nvidia-grid-drivers and https://docs.nvidia.com/vgpu/4.9/grid-vgpu-user-guide/index.html#installing-vgpu-drivers-linux -> download https://download.microsoft.com/download/c/3/4/c3484f19-fe76-4495-a65d-a5222ead9517/NVIDIA-Linux-x86_64-550.144.03-grid-azure.run (I also tried https://download.microsoft.com/download/7/e/c/7ec792c9-3654-4f78-b1a0-41a48e10ca6d/NVIDIA-Linux-x86_64-550.127.05-grid-azure.run)-> install but it needs a secure boot (I am not sure how to use it in cloud VM since we can not type F2) -> finish installation but it still does not work.
uninstall all previous drivers -> using ubuntu-drivers devices to search -> sudo apt install the recommended version -> still does not work.

My VM Environment:

~$ lspci | grep -i nvidia
0001:00:00.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1)

~$ uname -m && cat /etc/*release
x86_64
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.5 LTS"
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
~$ gcc --version
gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

~$ ubuntu-drivers devices
ERROR:root:aplay command not found
== /sys/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0004:00/VMBUS:00/47505500-0001-0000-3130-444531454238/pci0001:00/0001:00:00.0 ==
modalias : pci:v000010DEd00001EB8sv000010DEsd000012A2bc03sc02i00
vendor   : NVIDIA Corporation
model    : TU104GL [Tesla T4]
driver   : nvidia-driver-470-server - distro non-free
driver   : nvidia-driver-418-server - distro non-free
driver   : nvidia-driver-555 - third-party non-free
driver   : nvidia-driver-560 - third-party non-free
driver   : nvidia-driver-565 - third-party non-free recommended
driver   : nvidia-driver-570-server - distro non-free
driver   : nvidia-driver-450-server - distro non-free
driver   : nvidia-driver-470 - distro non-free
driver   : nvidia-driver-550 - third-party non-free
driver   : nvidia-driver-535-server - distro non-free
driver   : xserver-xorg-video-nouveau - distro free builtin

# remove previous drivers
sudo apt-get remove --purge '^nvidia-.*'
sudo apt-get remove --purge '^libnvidia-.*'
sudo apt-get remove --purge '^linux-objects-nvidia-.*'
sudo apt-get remove --purge '^linux-signatures-nvidia-.*'
sudo apt-get autoremove -y
sudo apt-get clean

Any help would be appreciated!

Mounika Reddy Anumandla 6,940 Reputation points Microsoft External Staff Moderator

2025-03-20T07:12:06.0233333+00:00

Hi Tianchen Guo,

Some users found this workaround helpful to uninstall previous drivers: https://askubuntu.com/questions/206283/how-can-i-uninstall-a-nvidia-driver-completelyAs per my understanding, first check if secure boot is enabled, Since Ubuntu signs its NVIDIA drivers, they work with Secure Boot enabled. When a VM has Trusted Launch and Secure Boot enabled, installing third-party drivers requires MOK for the system to boot.

Ubuntu packages NVIDIA proprietary drivers. Those drivers come directly from NVIDIA and are simply packaged by Ubuntu so that they can be automatically managed by the system. Downloading and installing drivers from another source can lead to a broken system. Moreover, installing third-party drivers requires extra-steps on VMs with TrustedLaunch and Secure Boot enabled. They require the user to add a new Machine Owner Key for the system to boot. Drivers from Ubuntu are signed by Canonical and will work with Secure Boot.

The document should work for you: https://learn.microsoft.com/en-us/azure/virtual-machines/linux/n-series-driver-setup?wt.mc_id=searchAPI_azureportal_inproduct_rmskilling&sessionId=7ec697615f424d5fb668b3e866efa4e5#ubuntu

Hope this helps!

Let me know if you have any further queries!

Tianchen Guo 5

Hi Mounika,

Thanks for your reply. However it is still does not work.

My entire working flow is:

sudo apt-get remove --purge '^nvidia-.*'
sudo apt-get install ubuntu-desktop
sudo rm /etc/X11/xorg.conf
echo 'nouveau' | sudo tee -a /etc/modules
sudo apt autoremove

Stop VM
Go to the Azura configuration to check the Security type is trusted launch VM.
Start VM

~$ mokutil --sb-state
SecureBoot enabled

~$ sudo dmesg | grep -i "trusted launch"
~$ sudo apt update && sudo apt install -y ubuntu-drivers-common
~$ sudo reboot # it is not necessary but I did that

# follow 
~$ sudo ubuntu-drivers install

Configuring Secure Boot ├────────────────────────┐
 │                                                                          │ 
 │ Your system has UEFI Secure Boot enabled.                                  
 │                                                                            
 │ UEFI Secure Boot requires additional configuration to work with            
 │ third-party drivers.                                                       
 │                                                                            
 │ The system will assist you in configuring UEFI Secure Boot. To permit      
 │ the use of third-party drivers, a new Machine-Owner Key (MOK) has been     
 │ generated. This key now needs to be enrolled in your system's firmware.    
 │                                                                            
 │ To ensure that this change is being made by you as an authorized user,     
 │ and not by an attacker, you must choose a password now and then confirm    
 │ the change after reboot using the same password, in both the "Enroll       
 │ MOK" and "Change Secure Boot state" menus that will be presented to you    
 │ when this system reboots.         
 Enter a password for Secure Boot. It will be asked again after a   │ 
    │ reboot.  
***
 Enter a password for Secure Boot. It will be asked again after a   │ 
    │ reboot.  
***

~$ sudo reboot

After the reboot. I can see a blue screen in boot diagnostics. But I can not do anything. Then wait for several minutes and log in again.

~$ nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

The issue still exists.

Could you please explicitly demonstrate how to do this step: As per my understanding, first check if secure boot is enabled, Since Ubuntu signs its NVIDIA drivers, they work with Secure Boot enabled. When a VM has Trusted Launch and Secure Boot enabled, installing third-party drivers requires MOK for the system to boot.

Finally, if it is possible, can you log in my VM to help me to fix this bug?

Thanks for your help!

Mounika Reddy Anumandla 6,940 Reputation points Microsoft External Staff Moderator

2025-03-21T01:24:13.2033333+00:00
Hi Tianchen Guo,

I understand from your output that VM has Trusted Launch and Secure Boot enabled.

Can you let me know the vm sku size you are using? I will try to reproduce the issue in my lab and see if the issue persists in my environment also.

I would like to know if you have checked that the drivers are properly uninstalled after the cleanup? Run the following command to see if any NVIDIA-related packages are still installed: dpkg -l | grep -i nvidia

Please confirm that the MOK enrollment was successful: sudo mokutil --list-enrolled

If the NVIDIA key is not enrolled, the driver is being blocked.

If MOK is missing, manually enroll it and reboot: sudo mokutil --import /var/lib/shim-signed/mok/MOK.der

I see nvidia-driver-565 is the recommended version: Run

sudo apt install <version>

https://forums.developer.nvidia.com/t/nvidia-smi-has-failed-because-it-couldnt-communicate-with-the-nvidia-driver-make-sure-that-the-latest-nvidia-driver-is-installed-and-running/197141/6
Tianchen Guo 5 Reputation points

2025-03-21T02:16:00.67+00:00
Hi Mounika,

The vm size is

NC4as_T4_v3 GPU 4 28 8 48000 176 (SCSI) Supported A$747.26

No output for dpkg -l | grep -i nvidia after using the above mentioned uninstall commands.

Are there any more tutorials regarding the procedures and the order of doing the MOK, NVIDIA key enrollment, driver installation and etc.?

I have created a new VM with the same setting as the previous one. And then I directly went to the extensions + applications page and installed NVIDIA GPU Driver Extension and it failed after waiting about 15 mins. I have no idea why even when creating a new one without any additional operations, it still failed.
Mounika Reddy Anumandla 6,940 Reputation points Microsoft External Staff Moderator

2025-03-21T09:46:35.3066667+00:00

Hi Tianchen Guo,

I have reproduced the scenario in my lab and faced the same issue. I will discuss the same with the product team. As a temporary workaround, for installing GRID Drivers on NC SKU VMs, you need to use Azure CLI and an ARM template:

https://github.com/Azure/azhpc-extensions/blob/master/NvidiaGPU/resources.json

You can use nvidia-smi to confirm installation at this point. The same has been addressed by Korey here and it worked for him: https://learn.microsoft.com/en-us/answers/questions/2180437/how-to-fix-no-gpu-issue-with-nc-series-on-azure-vm

Try to disable secure boot and install nvidia manually as mentioned in the document:

The GRID driver installation process does not offer any options to skip kernel module build and installation and select a different source of signed kernel modules, so secure boot has to be disabled in Linux VMs in order to use them with GRID, after installing signed kernel modules. https://learn.microsoft.com/en-us/azure/virtual-machines/linux/n-series-driver-setup#install-grid-driver-on-ubuntu-with-secure-boot-enabled

Let me know if you have any further queries!
Mounika Reddy Anumandla 6,940 Reputation points Microsoft External Staff Moderator

2025-03-24T01:42:05.49+00:00

Hi Tianchen Guo,

Just checking in to see if you had a chance to review the response provided to your question.

Let me know if you have any further queries!
Tianchen Guo 5 Reputation points

2025-03-24T06:05:01.2266667+00:00
Hi Mounika,

Thanks for your reply.

"As a temporary workaround, for installing GRID Drivers on NC SKU VMs, you need to use Azure CLI and an ARM template:" I tried but failed even for the new created VM. Perhaps it has some issues in my template. I just used the default one.

"You can use nvidia-smi to confirm installation at this point. The same has been addressed by Korey here and it worked for him: https://learn.microsoft.com/en-us/answers/questions/2180437/how-to-fix-no-gpu-issue-with-nc-series-on-azure-vm". The link doesn't seem to work. It says Bad Request.

I have tried disabling the secure boot and everything worked fine. nvidia-smi is functional by using any of the aforementioned install methods without the secure boot.

I would like to know more about the secure boot. Does this really matter in the VM?

Best,

Tianchen
Mounika Reddy Anumandla 6,940 Reputation points Microsoft External Staff Moderator

2025-03-25T03:55:41.47+00:00

Hi Tianchen Guo,
Just checking in to see if you had a chance to review the response provided to your question.

If you have any further queries, please let me know.

If the information is helpful, please click "Upvote it."
Tianchen Guo 5 Reputation points

2025-03-25T04:29:54.3766667+00:00

Hi Mounika,

Thanks for your response. Currently it works fine without the secure boot. And I will try to install this in secure boot later in the future. Again, thanks for your time and assistance. There is no further queries at the current stage.

Best,

Tianchen

1 answer

Your answer

Mounika Reddy Anumandla 6,940 Reputation points Microsoft External Staff Moderator

2025-03-20T07:12:06.0233333+00:00

Hi Tianchen Guo,

Some users found this workaround helpful to uninstall previous drivers: https://askubuntu.com/questions/206283/how-can-i-uninstall-a-nvidia-driver-completelyAs per my understanding, first check if secure boot is enabled, Since Ubuntu signs its NVIDIA drivers, they work with Secure Boot enabled. When a VM has Trusted Launch and Secure Boot enabled, installing third-party drivers requires MOK for the system to boot.

Ubuntu packages NVIDIA proprietary drivers. Those drivers come directly from NVIDIA and are simply packaged by Ubuntu so that they can be automatically managed by the system. Downloading and installing drivers from another source can lead to a broken system. Moreover, installing third-party drivers requires extra-steps on VMs with TrustedLaunch and Secure Boot enabled. They require the user to add a new Machine Owner Key for the system to boot. Drivers from Ubuntu are signed by Canonical and will work with Secure Boot.

The document should work for you: https://learn.microsoft.com/en-us/azure/virtual-machines/linux/n-series-driver-setup?wt.mc_id=searchAPI_azureportal_inproduct_rmskilling&sessionId=7ec697615f424d5fb668b3e866efa4e5#ubuntu

Hope this helps!

Let me know if you have any further queries!
Mounika Reddy Anumandla 6,940 Reputation points Microsoft External Staff Moderator

2025-03-21T01:24:13.2033333+00:00

Hi Tianchen Guo,

I understand from your output that VM has Trusted Launch and Secure Boot enabled.

Can you let me know the vm sku size you are using? I will try to reproduce the issue in my lab and see if the issue persists in my environment also.

I would like to know if you have checked that the drivers are properly uninstalled after the cleanup? Run the following command to see if any NVIDIA-related packages are still installed: dpkg -l | grep -i nvidia

Please confirm that the MOK enrollment was successful: sudo mokutil --list-enrolled

If the NVIDIA key is not enrolled, the driver is being blocked.

If MOK is missing, manually enroll it and reboot: sudo mokutil --import /var/lib/shim-signed/mok/MOK.der

I see nvidia-driver-565 is the recommended version: Run

sudo apt install <version>

https://forums.developer.nvidia.com/t/nvidia-smi-has-failed-because-it-couldnt-communicate-with-the-nvidia-driver-make-sure-that-the-latest-nvidia-driver-is-installed-and-running/197141/6
Tianchen Guo 5 Reputation points

2025-03-21T02:16:00.67+00:00

Hi Mounika,

The vm size is

NC4as_T4_v3 GPU 4 28 8 48000 176 (SCSI) Supported A$747.26

No output for dpkg -l | grep -i nvidia after using the above mentioned uninstall commands.

Are there any more tutorials regarding the procedures and the order of doing the MOK, NVIDIA key enrollment, driver installation and etc.?

I have created a new VM with the same setting as the previous one. And then I directly went to the extensions + applications page and installed NVIDIA GPU Driver Extension and it failed after waiting about 15 mins. I have no idea why even when creating a new one without any additional operations, it still failed.
Mounika Reddy Anumandla 6,940 Reputation points Microsoft External Staff Moderator

2025-03-21T09:46:35.3066667+00:00

Hi Tianchen Guo,

I have reproduced the scenario in my lab and faced the same issue. I will discuss the same with the product team. As a temporary workaround, for installing GRID Drivers on NC SKU VMs, you need to use Azure CLI and an ARM template:

https://github.com/Azure/azhpc-extensions/blob/master/NvidiaGPU/resources.json

You can use nvidia-smi to confirm installation at this point. The same has been addressed by Korey here and it worked for him: https://learn.microsoft.com/en-us/answers/questions/2180437/how-to-fix-no-gpu-issue-with-nc-series-on-azure-vm

Try to disable secure boot and install nvidia manually as mentioned in the document:

The GRID driver installation process does not offer any options to skip kernel module build and installation and select a different source of signed kernel modules, so secure boot has to be disabled in Linux VMs in order to use them with GRID, after installing signed kernel modules. https://learn.microsoft.com/en-us/azure/virtual-machines/linux/n-series-driver-setup#install-grid-driver-on-ubuntu-with-secure-boot-enabled

Let me know if you have any further queries!
Mounika Reddy Anumandla 6,940 Reputation points Microsoft External Staff Moderator

2025-03-24T01:42:05.49+00:00

Hi Tianchen Guo,

Just checking in to see if you had a chance to review the response provided to your question.

Let me know if you have any further queries!
Tianchen Guo 5 Reputation points

2025-03-24T06:05:01.2266667+00:00

Hi Mounika,

Thanks for your reply.

"As a temporary workaround, for installing GRID Drivers on NC SKU VMs, you need to use Azure CLI and an ARM template:" I tried but failed even for the new created VM. Perhaps it has some issues in my template. I just used the default one.

"You can use nvidia-smi to confirm installation at this point. The same has been addressed by Korey here and it worked for him: https://learn.microsoft.com/en-us/answers/questions/2180437/how-to-fix-no-gpu-issue-with-nc-series-on-azure-vm". The link doesn't seem to work. It says Bad Request.

I have tried disabling the secure boot and everything worked fine. nvidia-smi is functional by using any of the aforementioned install methods without the secure boot.

I would like to know more about the secure boot. Does this really matter in the VM?

Best,

Tianchen
Mounika Reddy Anumandla 6,940 Reputation points Microsoft External Staff Moderator

2025-03-25T03:55:41.47+00:00

Hi Tianchen Guo,
Just checking in to see if you had a chance to review the response provided to your question.

If you have any further queries, please let me know.

If the information is helpful, please click "Upvote it."
Tianchen Guo 5 Reputation points

2025-03-25T04:29:54.3766667+00:00

Hi Mounika,

Thanks for your response. Currently it works fine without the secure boot. And I will try to install this in secure boot later in the future. Again, thanks for your time and assistance. There is no further queries at the current stage.

Best,

Tianchen

Answer 1

Hi Tianchen Guo,
I have investigated on this issue further!

Install GRID driver on Ubuntu with Secure Boot enabled

The GRID driver installation process does not offer any options to skip kernel module build and installation and select a different source of signed kernel modules, so secure boot has to be disabled in Linux VMs in order to use them with GRID, after installing signed kernel modules.

https://learn.microsoft.com/en-us/azure/virtual-machines/linux/n-series-driver-setup#install-grid-driver-on-ubuntu-with-secure-boot-enabled

Installing via Azure CLI: https://learn.microsoft.com/en-us/answers/questions/2180437/how-to-fix-no-gpu-issue-with-nc-series-on-azure-vm

With Secure Boot enabled, all OS boot components (boot loader, kernel, kernel drivers) require trusted publishers signing. Both Windows and select Linux distributions support Secure Boot. If Secure Boot fails to authenticate that the image is signed with a trusted publisher, the VM fails to boot. For more information, see Secure Boot.
For more information: https://learn.microsoft.com/en-us/azure/virtual-machines/trusted-launch

Hope this helps!

Share via

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

1 answer

Your answer