How to Trace Process Signals and Termination in Linux Using killsnoop

Diagnosing unexpected process terminations in production Linux environments can be notoriously difficult when daemons exit abruptly without leaving stack traces or application log entries. While administrators often suspect memory exhaustion, silent terminations are frequently triggered by inter-process signals such as SIGKILL (9), SIGTERM (15), or SIGQUIT (3) sent by orchestrators, watchdog daemons, or rogue maintenance scripts. The eBPF-powered killsnoop utility instruments the kill() system call in real time to reveal exactly who signalled whom.

Understanding Linux Signal Delivery Mechanics

Signals represent asynchronous notifications delivered to processes by the kernel or other user-space processes:

• System Call Vector: Processes send signals to target PIDs via the kill() and tkill() system calls.
• Uncatchable Signals: Signals such as SIGKILL (9) and SIGSTOP (19) bypass user-space signal handlers entirely, immediately halting execution.
• Audit Limitations: Standard system loggers record service state transitions after a daemon dies, but cannot identify which originating process executed the kill command.

Installing the BCC Tracing Toolkit

Ensure the BPF Compiler Collection (BCC) and kernel headers are installed:

# Debian / Ubuntu
sudo apt update
sudo apt install -y bpfcc-tools linux-headers-$(uname -r)

# RHEL / Rocky Linux / Fedora
sudo dnf install -y bcc-tools kernel-devel-$(uname -r)

The executable is available as /usr/sbin/killsnoop-bpfcc or /usr/share/bcc/tools/killsnoop.

Tracing Live Signals with killsnoop

Launch killsnoop with superuser privileges to monitor signal transmissions across the operating system:

sudo killsnoop-bpfcc

When signals are dispatched, killsnoop streams detailed event telemetry:

TIME      PID    COMM             SIG  TPID   RESULT
13:42:01  3104   systemd          15   18420  0
13:42:05  9812   pkill            9    22041  0
13:42:19  1420   dockerd          15   30912  0
13:42:25  4109   monitor.sh       9    18902  -1

The telemetry columns indicate:

• TIME: Timestamp of the kill() system call invocation.
• PID & COMM: Process ID and executable name of the sending process.
• SIG: Numeric signal number (e.g., 9 for SIGKILL, 15 for SIGTERM, 1 for SIGHUP).
• TPID: Target process ID receiving the signal.
• RESULT: Return code from the kernel (0 indicates successful delivery; negative numbers indicate permission or lookup failures, such as -1 for EPERM/ESRCH).

Filtering by Target Process ID or Signal Type

To focus on a specific critical application or signal type:

# Trace signals targeted at a specific PID (e.g., PID 28410)
sudo killsnoop-bpfcc -p 28410

# Trace only fatal SIGKILL (Signal 9) events
sudo killsnoop-bpfcc -s 9

Tracing Failed Signal Attempts and Permission Violations

To audit potential security anomalies where non-privileged processes attempt to terminate system daemons, check the -f flag to report failed signal deliveries:

sudo killsnoop-bpfcc -f

This isolates failed attempts where malicious actors or misconfigured background jobs repeatedly probe processes with unauthorized signal interrupts.