After performing routine system package upgrades on Ubuntu, Debian, or Fedora, running nvidia-smi to monitor GPU temperatures, memory usage, or CUDA workloads frequently fails with the critical error: ‘NVIDIA-SMI has failed because it couldn\’t communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.’ Even though the graphics card worked moments prior, CUDA applications fail to detect compute capability. This mismatch occurs because the active Linux kernel version desynchronised from the compiled Dynamic Kernel Module Support (DKMS) driver modules.
Why Does NVIDIA-SMI Lose Communication with the Driver?
The communication failure stems from kernel-space module discrepancies:
- Kernel Update Without DKMS Rebuild: An automated
apt upgradeinstalled a newer Linux kernel image (e.g. updating from 6.5.0-28 to 6.5.0-35). If the system rebooted or failed to trigger DKMS autoinstall, the running kernel cannot locate matchingnvidia.kobinaries. - Stale Driver Modules in Memory: Old driver modules remain loaded in kernel memory while user-space libraries (like
nvidia-smi) updated to newer package versions. - Nouveau Open-Source Driver Conflict: The default open-source
nouveaugraphics driver grabbed the GPU framebuffer before proprietary NVIDIA modules loaded. - Secure Boot Signature Rejection: UEFI Secure Boot blocked third-party unsigned NVIDIA kernel modules from loading into the Linux kernel ring.
Step 1: Verify Running Kernel and DKMS Module Status
Check whether the running kernel has an active NVIDIA module registered.
- Open your terminal and check the currently active kernel:
uname -r - Check DKMS status across all installed kernel versions:
dkms status - If the output displays
nvidia, X.Y.Z: installedfor an older kernel but showsaddedor nothing for your currentuname -rkernel, the module must be rebuilt.
Step 2: Rebuild NVIDIA DKMS Kernel Module
Compile the NVIDIA kernel module against the currently active kernel headers.
- Ensure header packages matching your kernel are installed:
sudo apt update && sudo apt install -y linux-headers-$(uname -r) build-essential dkms - Trigger DKMS automatic recompilation for all registered modules:
sudo dkms autoinstall - Alternatively, rebuild the specific NVIDIA driver version (replacing
550.54.14with your driver version):sudo dkms install -m nvidia -v 550.54.14 -k $(uname -r)
Step 3: Unload Stale Modules or Reboot
If new modules compiled but user-space tools cannot communicate, unload the stale kernel drivers or perform a clean restart.
- Attempt to unload existing NVIDIA modules (stop display managers first if running a desktop environment):
sudo systemctl stop gdm3 # or lightdm / sddm sudo rmmod nvidia_uvm sudo rmmod nvidia_modeset sudo rmmod nvidia_drm sudo rmmod nvidia - Reload the newly compiled driver:
sudo modprobe nvidia - If modules report ‘in use’, perform a clean system reboot to initialise the new kernel module cleanly:
sudo reboot
Step 4: Check UEFI Secure Boot MOK Key State
If modprobe nvidia returns ‘Operation not permitted’ or ‘Required key not available’, Secure Boot is rejecting unsigned modules.
- Check Secure Boot status:
mokutil --sb-state - If Secure Boot is enabled, sign the module using your machine-owner key (MOK) or disable Secure Boot temporarily in your motherboard UEFI BIOS.
- Run
nvidia-smito verify successful communication and monitor GPU stats.