In high-throughput Linux environments, diagnosing performance degradation, unexpected CPU spikes, or process table exhaustion often requires monitoring how frequently processes fork new child threads and tasks. High-frequency process spawning (such as rogue shell loops, failing microservices, or misconfigured web workers) incurs substantial kernel overhead via page table replication, memory duplication, and context switching.
Traditional utilities such as top, ps, or htop rely on periodic sampling intervals from /proc, inevitably missing short-lived child processes that spawn, run, and terminate between sampling ticks. To capture every process creation event with negligible performance overhead, Linux administrators rely on forksnoop, a specialised tracing utility powered by extended Berkeley Packet Filters (eBPF). This tutorial covers how to install, configure, and use forksnoop to trace process generation, monitor process lifecycles, and diagnose fork latency in Linux.
Understanding How forksnoop and eBPF Work
When a Linux process creates a child copy of itself, it invokes the fork(), vfork(), or clone() system calls. Under traditional tracing using strace or ptrace, attaching to running processes degrades throughput significantly because the kernel context switches repeatedly between user space and kernel space.
forksnoop operates within the kernel space using eBPF kprobes or tracepoints attached to the kernel’s process creation entry points (such as sys_clone or finish_task_switch). It records execution metrics instantly, including the parent process ID (PPID), child process ID (PID), command name, and elapsed execution latency. The compiled metrics are passed to user space via high-speed perf event ring buffers without stalling application execution.
Step 1: Install BCC Tools on Linux
forksnoop is distributed as part of the BPF Compiler Collection (BCC) tools package. Ensure your kernel is version 4.9 or later with eBPF and kprobes enabled.
On Ubuntu and Debian:
sudo apt update
sudo apt install -y bpfcc-tools linux-headers-$(uname -r)
On Debian and Ubuntu, BCC utilities append the -bpfcc suffix to avoid namespace collisions with standard binaries (e.g., forksnoop-bpfcc). To run them without the suffix, add the BCC tools path to your shell environment:
echo 'export PATH=$PATH:/usr/sbin' >> ~/.bashrc
source ~/.bashrc
On RHEL, Rocky Linux, and Fedora:
sudo dnf install -y bcc-tools kernel-devel-$(uname -r)
export PATH=$PATH:/usr/share/bcc/tools
On Arch Linux:
sudo pacman -S bcc bcc-tools linux-headers
Step 2: Basic Process Tracing with forksnoop
To trace every process creation event across the operating system in real time, run forksnoop with elevated administrative privileges:
sudo forksnoop-bpfcc
The console produces a live stream of newly spawned processes:
TIME(s) PCOMM PID PPID RET
0.000000000 bash 18450 18201 18451
0.012481023 git 18451 18450 18452
0.024982110 sed 18453 18451 18454
The output columns display vital diagnostic metrics:
- TIME(s): Relative elapsed time in seconds since tracing started.
- PCOMM: The parent process command name initiating the fork request.
- PID: The process ID of the parent thread that triggered the system call.
- PPID: The grandparent process ID.
- RET: The return value from the kernel. A positive number indicates the new child PID assigned to the newly created process, while a negative integer indicates an execution error.
Step 3: Print Human-Readable Timestamps
When auditing logs or correlating system events with web server access logs or database errors, absolute timestamps are essential. Use the -T flag to output time formatted in HH:MM:SS:
sudo forksnoop-bpfcc -T
TIME PCOMM PID PPID RET
14:15:02 bash 18450 18201 18451
14:15:02 python3 18451 18450 18452
To view timestamps with nanosecond precision for microsecond-level latency audits, add the -t flag instead.
Step 4: Trace and Calculate Process Lifetimes and Latency
One of the most powerful diagnostic capabilities of forksnoop is measuring how long child processes survive. If an application repeatedly spawns processes that exit within a few milliseconds, it causes needless thread contention.
Pass the -d flag to calculate duration in milliseconds:
sudo forksnoop-bpfcc -d
Sample output:
TIME(s) PCOMM PID PPID RET AGE(ms)
0.000000000 make 21044 19200 21045 14.22
0.015024100 gcc 21045 21044 21046 320.18
0.340112340 as 21046 21045 21047 45.02
The AGE(ms) column reflects total process lifespan before termination. Short-lived bursts indicate transient worker threads or high-frequency subshell execution.
Step 5: Filter by Process Name or Specific PID
In high-density production systems, tracing system-wide forks generates voluminous output. Filter tracing to isolate a single parent daemon or application:
Filter by Parent Process Name
Use -n (name) to capture events originating only from a specific binary (e.g., nginx or node):
sudo forksnoop-bpfcc -n nginx
Filter by Specific PID
To inspect a known background service or suspect application daemon, pass the -p parameter with its PID:
sudo forksnoop-bpfcc -p 3452
This restricts eBPF probe execution strictly to thread-creation events triggered by process 3452, filtering out system background noise.
Diagnosing Failed Fork Calls (Resource Exhaustion)
When the kernel exhausts available process slots (reaching kernel.pid_max or exceeding cgroup process limits), fork() calls fail, typically returning -EAGAIN (-11) or -ENOMEM (-12). These failures frequently manifest as intermittent application crashes without descriptive error logs.
Identify failed forks by monitoring negative values in the RET column:
sudo forksnoop-bpfcc | awk '$5 ~ /^-/'
If you observe negative return values, inspect system-wide limits:
# Check maximum system PID capacity
sysctl kernel.pid_max
# Check current user process limit
ulimit -u
Tune the limits accordingly via /etc/sysctl.d/99-limits.conf or systemd service unit configurations to prevent process starvation.