How to Configure Linux Kernel Soft and Hard Lockup Watchdogs Using sysctl

In high-availability Linux server environments, high-performance computing clusters, and real-time virtualization hosts, kernel stalls can disrupt operations without triggering a traditional kernel panic. When a thread loops indefinitely in kernel space with preemption disabled, or when hardware interrupts fail to service a CPU core, the system experiences soft or hard lockups. To detect, diagnose, and autonomously remediate such lockups, the Linux kernel incorporates built-in watchdog mechanisms: the Soft Lockup Watchdog (which monitors thread responsiveness) and the Hard Lockup Watchdog (driven by Non-Maskable Interrupts / NMI). Using sysctl and procfs, systems engineers can tune watchdog thresholds, configure automated reboot actions on hang, and prevent unexplained silent freezes.

Understanding Soft Lockups vs Hard Lockups in the Linux Kernel

The Linux kernel classifies processor execution stalls into two distinct architectural categories:

• Soft Lockup: Occurs when a task occupies a CPU core in kernel space for longer than the defined threshold without relinquishing execution or scheduling out, whilst interrupts remain enabled. A dedicated per-CPU kernel thread (watchdog/X) increments a local counter; if this thread is starved of execution, the soft lockup detector triggers.
• Hard Lockup: Occurs when a CPU core stops servicing local timer interrupts altogether—often because code is executing inside a critical spinlock with local interrupts disabled (local_irq_disable()). The hard lockup detector relies on performance monitoring counter (PMC) overflows delivering periodic Non-Maskable Interrupts (NMI) that cannot be masked by software.

Auditing Active Watchdog Kernel Parameters via sysctl

To inspect active watchdog configurations, query the kernel namespace using sysctl with administrative privileges:

# Query all kernel watchdog and lockup parameters
sudo sysctl -a | grep -E "watchdog|hung_task"

Key parameters governing lockup detection include:

• kernel.watchdog: Master toggle for all lockup detectors (1 = enabled, 0 = disabled).
• kernel.watchdog_thresh: Time interval in seconds used to evaluate lockups. The default value is typically 10 seconds.
• kernel.nmi_watchdog: Master control for the hardware NMI hard lockup detector.
• kernel.softlockup_panic: Dictates whether a soft lockup triggers an immediate kernel panic (1) or merely logs a warning stack trace to dmesg (0).
• kernel.hardlockup_panic: Dictates whether an unmasked hard lockup forces a kernel panic.

Configuring Watchdog Evaluation Thresholds

The kernel.watchdog_thresh parameter defines the detection baseline. The kernel calculates lockup timeouts using this value:

• Soft lockup timeout: Evaluated as 2 * watchdog_thresh. With a threshold of 10 seconds, a soft lockup is declared if a CPU is unresponsive for 20 seconds.
• Hard lockup timeout: Evaluated directly against the threshold interval.

In virtualised environments or systems with bursty I/O latency, false-positive lockup warnings may occur. To increase the evaluation threshold to 20 seconds (giving a 40-second soft lockup window):

sudo sysctl -w kernel.watchdog_thresh=20

Conversely, in ultra-low-latency real-time applications where rapid failover is mandatory, lowering the threshold ensures faster cluster node fencing.

Enforcing Kernel Panics and Automated Reboots on Lockup

In automated enterprise clusters (such as Kubernetes worker nodes or Ceph storage daemons), a frozen node that fails to panic can hold clustered storage locks indefinitely, stalling cluster failover. Configuring the kernel to panic immediately upon lockup and trigger a hardware reboot restores cluster health automatically.

# Force kernel panic upon soft lockup detection
sudo sysctl -w kernel.softlockup_panic=1

# Force kernel panic upon hard lockup detection
sudo sysctl -w kernel.hardlockup_panic=1

# Trigger automatic hardware reboot 10 seconds after kernel panic
sudo sysctl -w kernel.panic=10

Managing the NMI Watchdog on Intel and AMD Hardware

The hard lockup detector requires hardware performance monitoring counters (PMC) to generate NMIs. On systems where PMCs are heavily utilized by profiling tools (such as perf), or on hypervisors where virtualised NMIs are constrained, you can toggle the NMI watchdog independently:

# Disable NMI watchdog at runtime
sudo sysctl -w kernel.nmi_watchdog=0

# Enable NMI watchdog at runtime
sudo sysctl -w kernel.nmi_watchdog=1

To inspect active NMI interrupt frequency across physical CPU cores:

grep NMI /proc/interrupts

Each core should show an incrementing count of LOC: Local timer interrupts and NMI: Non-maskable interrupts, verifying active watchdog polling.

Persisting Watchdog Configurations Across Reboots

To ensure tuned watchdog parameters apply automatically during early system initialization, persist the directives in a dedicated sysctl configuration drop-in file:

sudo tee /etc/sysctl.d/99-kernel-watchdog.conf << 'EOF'
# Kernel Watchdog and Automated Lockup Remediation
kernel.watchdog = 1
kernel.nmi_watchdog = 1
kernel.watchdog_thresh = 15
kernel.softlockup_panic = 1
kernel.hardlockup_panic = 1
kernel.panic = 10
kernel.panic_on_oops = 1
EOF

Apply the persistent configuration immediately:

sudo sysctl --system

Simulating and Diagnosing Kernel Lockups

To test how the system behaves under kernel stalls, administrators can trigger synthetic hang events using the Linux Magic SysRq interface (ensure testing is conducted in a non-production sandbox):

# Enable SysRq functions
sudo sysctl -w kernel.sysrq=1

# View recent watchdog alerts and kernel backtraces in the ring buffer
sudo dmesg -T | grep -iE "watchdog|lockup|BUG: soft lockup"

When a lockup occurs, the kernel outputs the culprit call trace, processor registers (RIP, RSP), and the offending function name, allowing software engineers to locate deadlock loops or driver stall points.

Proactively configuring kernel watchdogs via sysctl ensures mission-critical Linux deployments recover deterministically from execution freezes while capturing critical diagnostic crash dumps.