Inspecting Linux system logs via journalctl -xe or dmesg on Ubuntu, Debian, or enterprise Linux servers frequently reveals repetitive kernel warnings: systemd-udevd[1234]: worker [5678] terminated by signal 9 (KILL), followed by alerts that a device event timed out. In severe cases, boot times slow down dramatically, storage drives fail to mount automatically, or network interfaces delay configuration. In modern systemd architecture, systemd-udevd listens for kernel device events (uevents) and executes user-space rules. If a worker script hangs or blocks execution beyond the default 180-second udev timeout, the systemd supervisor issues an uncatchable SIGKILL (signal 9) to terminate the stalled worker.
Why Does systemd-udevd Get Terminated by Signal 9?
The worker execution abort originates from hardware probe deadlocks or misconfigured udev rules:
- Worker Timeout Exceeded (Default 180s): A custom or third-party udev rule executing a synchronous blocking script (e.g. attempting network lookups, interactive prompts, or hanging binary calls) during early boot.
- Failing Storage Devices or Slow Media: Optical drives, SD card readers, or SATA disks with bad sectors causing kernel block layer I/O stalls during partition table probing.
- Out-of-Memory (OOM) Killer Invocations: Extreme physical RAM exhaustion triggering the Linux kernel OOM killer to terminate child udev workers consuming memory.
- Corrupted Initial RAM Filesystem (initramfs): Stale compiled udev rules remaining cached inside initramfs across kernel updates.
Step 1: Identify the Failing Device Event and udev Rule
Isolate the specific device node or subsystem triggering the worker timeout.
- Run
journalctlfiltered specifically for the udev daemon:journalctl -u systemd-udevd --no-pager -n 50 - Look for the line immediately preceding the ‘terminated by signal 9’ alert; it will specify the device path (e.g.
/sys/devices/pci0000:00/.../block/sdbor/dev/bus/usb/001/004). - Check which udev rules apply to that subsystem:
grep -rn "SUBSYSTEM" /etc/udev/rules.d/ /lib/udev/rules.d/
Step 2: Extend udev Event Timeout in Configuration
For storage arrays or specialized FPGA hardware requiring more than 180 seconds to spin up, adjust the supervisor timeout.
- Edit the udev configuration file:
sudo nano /etc/udev/udev.conf - Locate or add the
event_timeoutparameter:event_timeout=300 - Save the file (press Ctrl + O, then Ctrl + X).
- Restart the udev daemon:
sudo systemctl restart systemd-udevd
Step 3: Fix Blocking Synchronous Scripts in Custom Rules
Ensure custom rules execute scripts asynchronously so workers can exit immediately.
- If you have custom rules in
/etc/udev/rules.d/calling external bash scripts, inspect theRUN+=directive. - Never call scripts synchronously if they perform heavy I/O or network calls. Replace direct calls with systemd service triggers:
# Avoid: RUN+="/usr/local/bin/backup.sh" # Use systemd unit spawning instead: TAG+="systemd", ENV{SYSTEMD_WANTS}="backup-device.service" - Reload active rules:
sudo udevadm control --reload-rules && sudo udevadm trigger
Step 4: Update Initial RAM Filesystem and Reboot
Ensure clean udev rules are compiled into the boot image.
- Rebuild initramfs on Debian/Ubuntu:
sudo update-initramfs -u - On RHEL / Rocky / Fedora, run:
sudo dracut -f - Reboot your server and monitor
dmesgto verify that workers complete cleanly without SIGKILL events.