How to Trace Signals and Process Termination in Linux Using killsnoop

In Linux systems administration, unexpected process terminations, silent container crashes, and erratic service restarts present difficult troubleshooting challenges. When a critical database daemon, web server, or background worker disappears without generating a panic stack trace or application error log, standard utilities like ps or top offer no insight because the dead process has already been reaped. Frequently, these terminations occur because another process — such as an automated monitoring watchdog, the systemd init daemon, an orchestrator, or an out-of-memory handler — dispatched a POSIX termination signal (such as SIGKILL or SIGTERM) directly to the target process.

The BPF Compiler Collection (BCC) provides killsnoop, an extended Berkeley Packet Filter (eBPF) tracing utility. By instrumenting the kernel’s sys_kill() entry points, killsnoop traces every inter-process signal in real time with negligible performance overhead, capturing the sending process, target PID, signal number, and system call return status.

Understanding Signal Dispatch and Process Termination Mechanics

In the Linux operating system, processes communicate lifecycle events and asynchronous notifications through signals using the kill() system call. Key POSIX signals frequently responsible for sudden process termination include:

• SIGKILL (Signal 9): Uncatchable, immediate termination enforced directly by the kernel scheduler. The target process cannot execute cleanup routines or log a stack trace.
• SIGTERM (Signal 15): Polite termination request. Well-behaved processes catch this signal to flush buffers, close database sockets, and exit cleanly.
• SIGINT (Signal 2): Interactive interrupt generated by terminal keyboard events (such as pressing Ctrl + C).
• SIGHUP (Signal 1): Hangup signal, traditionally dispatched upon controlling terminal disconnection, often repurposed by daemons to trigger configuration reloads.
• SIGSEGV (Signal 11) and SIGABRT (Signal 6): Memory segmentation faults and abort assertions dispatched by the kernel or standard libraries upon fatal application errors.

killsnoop attaches eBPF kprobes to the kernel’s sys_enter_kill and sys_exit_kill functions. Whenever any process issues a kill syscall across any user account or container namespace, killsnoop extracts execution context before the signal takes effect.

Installing BCC Tools on Modern Linux Distributions

To run killsnoop, install the BCC package corresponding to your Linux distribution.

On Ubuntu and Debian systems, install the package via apt:

sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

(On Ubuntu, BCC tools are prefixed with -bpfcc, e.g. killsnoop-bpfcc.)

On Red Hat Enterprise Linux, Rocky Linux, or Fedora:

sudo dnf install bcc-tools kernel-devel-$(uname -r)

On Arch Linux:

sudo pacman -S bcc bcc-tools linux-headers

Running killsnoop to Capture Signals in Real Time

Launch killsnoop with root privileges in an interactive terminal:

sudo killsnoop-bpfcc

(On RHEL/CentOS, execute sudo /usr/share/bcc/tools/killsnoop.)

When signals are exchanged across the operating system, killsnoop streams structured columnar records:

TIME      PID     COMM             SIG  TPID    RESULT
14:22:01  1248    systemd          15   3812    0
14:22:05  4910    bash             9    4902    0
14:22:12  1       systemd          1    1204    0
14:22:18  8912    dockerd          15   8930    0
14:22:25  2401    monitoring-agent 9    6120    -1

Column definitions explained:

• TIME: Local timestamp of the signal dispatch.
• PID: Process ID of the sender issuing the kill system call.
• COMM: Executable command name of the sending process.
• SIG: POSIX signal number dispatched.
• TPID: Target Process ID receiving the signal.
• RESULT: Return value of the system call (0 indicates success; negative values indicate kernel error codes such as -1 EPERM for permission denied or -3 ESRCH for no such process).

Filtering Signal Traces by Process and Signal Number

In high-throughput environments running hundreds of microservices, tracing all signals can generate excessive noise. killsnoop provides command-line switches to isolate relevant events:

Tracing Only Failed Signal Invocations

To detect unauthorized signal attempts or permission violations where an unprivileged user or rogue process attempts to terminate system daemons, pass the -x flag:

sudo killsnoop-bpfcc -x

This filters out successful operations (RESULT = 0) and displays only failed attempts, such as permission denials (EPERM).

Filtering by Specific Signal Types

To monitor lethal signals specifically (such as SIGKILL / Signal 9) to discover who is force-killing application pods or database workers, pipe the output through awk:

sudo killsnoop-bpfcc | awk '$4 == 9 { print }'

Including Nanosecond Timestamps

For high-precision diagnostics and correlation with kernel dmesg logs, pass the -T flag to include full timestamps:

sudo killsnoop-bpfcc -T

Diagnosing Mystery Terminations and Container Evictions

Consider a scenario where an internal Redis server (PID 5521) dies abruptly every afternoon. Run killsnoop in the background and redirect output to an audit log:

sudo killsnoop-bpfcc -T >> /var/log/killsnoop-audit.log 2>&1 &

When Redis terminates, inspect the log filtered by the target PID:

grep "5521" /var/log/killsnoop-audit.log

The trace reveals the perpetrator:

15:45:10  7810    ansible-playbook 9    5521    0

This immediately proves that an automated Ansible deployment script issued kill -9 to PID 5521, eliminating speculative debugging of kernel bugs or memory leaks.

Using the bpftrace Implementation

If full Python-based BCC dependencies cannot be installed on minimal production containers, trace kill syscalls directly using a single bpftrace one-liner:

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_kill { printf("%s (PID %d) sent signal %d to target PID %d\n", comm, pid, args->sig, args->pid); }'

This provides an immediate, zero-dependency stream of signal exchanges directly from the Linux kernel tracepoints.