How to Configure systemd System and Service Watchdogs in Ubuntu

In production Linux server environments, critical services (like web servers, database engines, or backend queue workers) can occasionally experience deadlocks, infinite event loops, or memory lockups without actually crashing. Because the process PID remains alive, standard service restart policies (like Restart=always) fail to detect that the application is unresponsive. systemd Watchdogs solve this silent freeze dilemma by requiring services to send periodic “heartbeat” signals to systemd. If a daemon fails to ping the supervisor within a defined window, systemd automatically kills and re-initializes the hung process.

Understanding Hardware vs Software Watchdogs in systemd

Linux supports two complementary watchdog layers:

• Hardware Watchdog (System-Level): Leverages motherboard hardware watchdog timers or IPMI BMC chips (via /dev/watchdog). systemd pings the hardware chip. If the Linux kernel itself panics or hard-locks, the hardware timer expires and forces an automatic hardware server reboot.
• Service Watchdog (Application-Level): Governed by WatchdogSec= in individual service unit files. systemd monitors the service process. If the application stops sending WATCHDOG=1 notification pings via sd_notify, systemd restarts the failed unit.

Configuring Service Watchdogs in systemd Unit Files

To enable watchdog supervision for a custom or third-party service daemon, edit or create a drop-in override for the service unit:

Create a systemd unit override using systemctl edit:

sudo systemctl edit my-custom-worker.service

Add the WatchdogSec parameter under the [Service] section:

[Service]
Type=notify
WatchdogSec=30s
Restart=always
RestartSec=5s

With WatchdogSec=30s, systemd expects the managed service to send a keepalive notification at least once every 30 seconds. In practice, well-designed daemons dispatch a heartbeat ping at half that interval (every 15 seconds).

Implementing Watchdog Pings in Daemon Applications

Applications notify systemd using the sd_notify(3) C API, Python libraries (like systemd-python), or shell commands:

In Python Workers:

import systemd.daemon
import time

# Signal systemd that the service has completed startup initialization
systemd.daemon.notify("READY=1")

while True:
    # Perform operational processing work
    process_task_queue()
    
    # Send watchdog heartbeat ping to systemd
    systemd.daemon.notify("WATCHDOG=1")
    
    time.sleep(10)

In Shell Scripts or Simple Binaries:

systemd-notify WATCHDOG=1

Configuring Hardware Watchdogs in system.conf

To guard against total operating system kernel freezes on physical or cloud servers:

1. Open /etc/systemd/system.conf in a text editor:

   sudo nano /etc/systemd/system.conf

2. Locate and configure the watchdog timer directives:

   [Manager]
   RuntimeWatchdogSec=60s
   RebootWatchdogSec=5min
   KExecWatchdogSec=5min

3. Save the file and reload systemd manager configuration:

   sudo systemctl daemon-reload

systemd now continuously feeds /dev/watchdog. If the kernel locks up and stops petting the hardware timer for 60 seconds, the hardware chip triggers an automatic reboot to restore server availability.

Testing and Validating Watchdog Failure Recovery

To simulate an application freeze and verify automatic recovery:

1. Start your watchdog-enabled service:

   sudo systemctl start my-custom-worker.service

2. Freeze the process non-destructively using SIGSTOP:

   sudo kill -SIGSTOP $(systemctl show --property MainPID --value my-custom-worker.service)

3. Monitor the systemd journal logs in real time:

   journalctl -u my-custom-worker.service -f

4. Once the WatchdogSec duration elapses, systemd detects the missed heartbeat, issues an automated SIGABRT or SIGKILL, and restarts the service automatically.