How to Trace Process Signals and Inter-Process Signals in Linux Using sigsnoop

When services crash unexpectedly, background daemons terminate without error messages, or microservice containers fail health checks in Linux environments, standard application logs rarely reveal the root cause. Often, processes are being killed externally by POSIX signals—such as SIGKILL (signal 9), SIGTERM (signal 15), or segmentation violations (SIGSEGV). The sigsnoop utility, provided by the BPF Compiler Collection (BCC), attaches eBPF kprobes directly to kernel signal delivery routines, logging every inter-process signal in real time alongside sender and recipient identifiers.

How POSIX Signals Operate and Why They Vanish from Standard Logs

In Unix and Linux architectures, signals provide asynchronous event notifications delivered directly by the kernel or transmitted between processes via the kill(), tkill(), or tgkill() system calls:

• Uncatchable signals: Signals like SIGKILL (9) and SIGSTOP (19) terminate or freeze execution immediately without invoking application signal handlers, leaving zero trace in user-space logs.
• External killers: Out-of-memory killers (OOM killer), container runtimes, process supervisors (systemd), or rogue administration scripts can transmit termination signals to worker threads asynchronously.
• Internal faults: Faulty memory pointers or hardware illegal instructions trigger kernel-generated signals like SIGSEGV (11) and SIGILL (4).

Traditional diagnostic tools like strace can intercept signals received by a single attached process, but introducing strace imposes heavy runtime overhead and requires prior knowledge of which process will receive the signal. sigsnoop observes system-wide signal transitions at native kernel speeds with negligible CPU overhead.

Installing BCC Tools on Debian, Ubuntu, and Enterprise Linux

Ensure that BCC packages and operating system kernel headers are installed:

# On Debian and Ubuntu
sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r)

# On RHEL, Rocky Linux, and Fedora
sudo dnf install bcc-tools kernel-devel-$(uname -r)

After installation, the binary is located at /usr/sbin/sigsnoop-bpfcc on Debian/Ubuntu or /usr/share/bcc/tools/sigsnoop on RPM-based distributions.

Tracing System-Wide Signal Emissions in Real Time

Launch sigsnoop with root privileges to trace all signals exchanged across the operating system:

sudo /usr/sbin/sigsnoop-bpfcc

Whenever a signal is delivered, sigsnoop logs a real-time event record displaying the timestamp, sender process name and PID, signal number and name, target recipient PID, and the system call return result:

TIME(s)    COMM             PID    TID    SIG    TPID   RESULT
0.000000   systemd          1      1      15     18492  0
0.142018   kill             21904  21904  9      19402  0
1.890412   bash             22401  22401  2      22401  0
2.451290   node             23011  23011  11     23011  0

Analysing these event lines yields immediate diagnostic insights:

• Line 1 indicates that systemd (PID 1) transmitted SIGTERM (15) to gracefully shut down worker process 18492 with return code 0 (success).
• Line 2 demonstrates that a manual kill command (PID 21904) forcefully issued uncatchable SIGKILL (9) against target process 19402.
• Line 4 shows a Node.js process encountering a fatal segmentation violation (SIGSEGV 11) within its own thread context.

Advanced Filtering: Isolating Signals, Senders, and Targets

To pinpoint specific troubleshooting scenarios on busy production clusters without capturing routine housekeeping signals, apply command-line switches:

1. Isolating Lethal Signals (SIGKILL and SIGTERM)

To monitor only forceful termination events while ignoring harmless timer or child notifications (like SIGCHLD), filter by specific signal numbers using -s:

sudo /usr/sbin/sigsnoop-bpfcc -s 9,15

2. Filtering by Target Process ID

If a specific critical database or web server daemon keeps crashing mysteriously, monitor all incoming signals directed at its PID:

sudo /usr/sbin/sigsnoop-bpfcc -t 14022

3. Filtering by Originating Sender PID

To inspect what signals a suspected malfunctioning watchdog script or monitoring agent is broadcasting across system processes, pass the sender PID with -p:

sudo /usr/sbin/sigsnoop-bpfcc -p 8412

4. Tracing Failed Signal Deliveries

By default, sigsnoop captures all delivery attempts. To display only failed signal invocations (e.g., where a non-root process attempts to signal a root daemon and receives -EPERM), pass the -f flag:

sudo /usr/sbin/sigsnoop-bpfcc -f

Pairing sigsnoop with core dump analysis provides complete operational transparency into process termination lifecycles.