we are facing nvidia driver mismatch issue on ubuntu 22.04 server

Question

Nvidia driver issue

Failed to initialize NVML: Driver/library version mismatch

NVML library version: 535.171

Answer 1

Hi @Nitesh Kumar,

Welcome to the Microsoft Q&A Platform! Thank you for asking your question here.

The reason for this problem is that the version of the kernel mod Nvidia driver is not updated. Generally, restarting the machine can solve the problem. If restarting is not possible for some reason, there is also a way to reload the kernel mod.

First, we should know which drivers are loaded.

lsmod | grep nvidia

You may get

nvidia_uvm 634880 8

nvidia_drm 53248 0

nvidia_modeset 790528 1 nvidia_drm

nvidia 12312576 86 nvidia_modeset,nvidia_uvm

Our final goal is to unload the NVIDIA module, so we should unload the modules dependent on NVIDIA:

sudo rmmod nvidia_drm sudo rmmod nvidia_modeset

sudo rmmod nvidia_uvm

Then, unload NVIDIA:

sudo rmmod nvidia If you encounter an error like rmmod: ERROR: Module nvidia is in use, it indicates that the kernel module is in use.

You should kill the process that is using the module:

sudo lsof /dev/nvidia* Then, kill those processes and continue unloading the kernel modules.

Confirm that you have successfully unloaded the kernel modules:

lsmod | grep nvidia

You should see no output. Then, verify that you can load the correct driver:

nvidia-smi You should see the correct output.

If you have any further queries, do let us know.

If the answer is helpful, please click "Accept Answer" and "Upvote it."