How to Trace Process Creation and Fork Latency in Linux Using forksnoop

In high-throughput Linux environments, diagnosing performance degradation, unexpected CPU spikes, or process table exhaustion often requires monitoring how frequently processes fork new child threads and tasks. High-frequency process spawning (such as rogue shell loops, failing microservices, or misconfigured web workers) incurs substantial kernel overhead via page table replication, memory duplication, and context switching.

Traditional utilities such as top, ps, or htop rely on periodic sampling intervals from /proc, inevitably missing short-lived child processes that spawn, run, and terminate between sampling ticks. To capture every process creation event with negligible performance overhead, Linux administrators rely on forksnoop, a specialised tracing utility powered by extended Berkeley Packet Filters (eBPF). This tutorial covers how to install, configure, and use forksnoop to trace process generation, monitor process lifecycles, and diagnose fork latency in Linux.

Understanding How forksnoop and eBPF Work

When a Linux process creates a child copy of itself, it invokes the fork(), vfork(), or clone() system calls. Under traditional tracing using strace or ptrace, attaching to running processes degrades throughput significantly because the kernel context switches repeatedly between user space and kernel space.

forksnoop operates within the kernel space using eBPF kprobes or tracepoints attached to the kernel’s process creation entry points (such as sys_clone or finish_task_switch). It records execution metrics instantly, including the parent process ID (PPID), child process ID (PID), command name, and elapsed execution latency. The compiled metrics are passed to user space via high-speed perf event ring buffers without stalling application execution.

Step 1: Install BCC Tools on Linux

forksnoop is distributed as part of the BPF Compiler Collection (BCC) tools package. Ensure your kernel is version 4.9 or later with eBPF and kprobes enabled.

On Ubuntu and Debian:

sudo apt update
sudo apt install -y bpfcc-tools linux-headers-$(uname -r)

On Debian and Ubuntu, BCC utilities append the -bpfcc suffix to avoid namespace collisions with standard binaries (e.g., forksnoop-bpfcc). To run them without the suffix, add the BCC tools path to your shell environment:

echo 'export PATH=$PATH:/usr/sbin' >> ~/.bashrc
source ~/.bashrc

On RHEL, Rocky Linux, and Fedora:

sudo dnf install -y bcc-tools kernel-devel-$(uname -r)
export PATH=$PATH:/usr/share/bcc/tools

On Arch Linux:

sudo pacman -S bcc bcc-tools linux-headers

Step 2: Basic Process Tracing with forksnoop

To trace every process creation event across the operating system in real time, run forksnoop with elevated administrative privileges:

sudo forksnoop-bpfcc

The console produces a live stream of newly spawned processes:

TIME(s)            PCOMM            PID    PPID   RET
0.000000000        bash             18450  18201  18451
0.012481023        git              18451  18450  18452
0.024982110        sed              18453  18451  18454

The output columns display vital diagnostic metrics:

  • TIME(s): Relative elapsed time in seconds since tracing started.
  • PCOMM: The parent process command name initiating the fork request.
  • PID: The process ID of the parent thread that triggered the system call.
  • PPID: The grandparent process ID.
  • RET: The return value from the kernel. A positive number indicates the new child PID assigned to the newly created process, while a negative integer indicates an execution error.

Step 3: Print Human-Readable Timestamps

When auditing logs or correlating system events with web server access logs or database errors, absolute timestamps are essential. Use the -T flag to output time formatted in HH:MM:SS:

sudo forksnoop-bpfcc -T
TIME     PCOMM            PID    PPID   RET
14:15:02 bash             18450  18201  18451
14:15:02 python3          18451  18450  18452

To view timestamps with nanosecond precision for microsecond-level latency audits, add the -t flag instead.

Step 4: Trace and Calculate Process Lifetimes and Latency

One of the most powerful diagnostic capabilities of forksnoop is measuring how long child processes survive. If an application repeatedly spawns processes that exit within a few milliseconds, it causes needless thread contention.

Pass the -d flag to calculate duration in milliseconds:

sudo forksnoop-bpfcc -d

Sample output:

TIME(s)            PCOMM            PID    PPID   RET    AGE(ms)
0.000000000        make             21044  19200  21045  14.22
0.015024100        gcc              21045  21044  21046  320.18
0.340112340        as               21046  21045  21047  45.02

The AGE(ms) column reflects total process lifespan before termination. Short-lived bursts indicate transient worker threads or high-frequency subshell execution.

Step 5: Filter by Process Name or Specific PID

In high-density production systems, tracing system-wide forks generates voluminous output. Filter tracing to isolate a single parent daemon or application:

Filter by Parent Process Name

Use -n (name) to capture events originating only from a specific binary (e.g., nginx or node):

sudo forksnoop-bpfcc -n nginx

Filter by Specific PID

To inspect a known background service or suspect application daemon, pass the -p parameter with its PID:

sudo forksnoop-bpfcc -p 3452

This restricts eBPF probe execution strictly to thread-creation events triggered by process 3452, filtering out system background noise.

Diagnosing Failed Fork Calls (Resource Exhaustion)

When the kernel exhausts available process slots (reaching kernel.pid_max or exceeding cgroup process limits), fork() calls fail, typically returning -EAGAIN (-11) or -ENOMEM (-12). These failures frequently manifest as intermittent application crashes without descriptive error logs.

Identify failed forks by monitoring negative values in the RET column:

sudo forksnoop-bpfcc | awk '$5 ~ /^-/'

If you observe negative return values, inspect system-wide limits:

# Check maximum system PID capacity
sysctl kernel.pid_max

# Check current user process limit
ulimit -u

Tune the limits accordingly via /etc/sysctl.d/99-limits.conf or systemd service unit configurations to prevent process starvation.

Get the best tech tips delivered straight to your inbox.

Join thousands of readers mastering Apple, Google, Microsoft, and Linux.