How to Trace Short-Lived Processes in Linux Using execsnoop

In production Linux environments, identifying performance bottlenecks and security anomalies frequently requires auditing command executions and child process launches. While standard monitoring tools such as top, htop, or ps capture long-running processes like databases and web servers, they are blind to transient, short-lived processes. Scripts that execute rapid sub-shells, cron jobs invoking micro-commands, and compilation tasks that spawn thousands of brief compiler passes often consume significant CPU and disk resources before terminating in milliseconds.

Sampling-based utilities inevitably miss these ephemeral executions. The BPF Compiler Collection (BCC) provides execsnoop, an eBPF tracing utility designed to intercept and record every process execution system call in real time. By tracing the execve kernel system call family directly, execsnoop guarantees complete visibility into command executions, arguments, process lineages, and exit failure codes without sampling gaps. This guide shows how to trace short-lived processes in Linux using execsnoop.

How execsnoop Operates via eBPF

The execsnoop utility attaches an extended Berkeley Packet Filter (eBPF) kprobe or tracepoint to the kernel’s sys_enter_execve and sys_exit_execve tracepoints. Whenever any application—whether a bash script, system daemon, or container runtime—invokes an executable via execve(), the kernel generates an event.

Unlike fork() or clone() which duplicate existing memory spaces, execve() transforms a process into a newly loaded binary. Capturing this event allows execsnoop to log the exact command-line arguments, working directory environment references, Parent Process ID (PPID), and execution return status.

Prerequisites and Tool Installation

The utility requires Linux kernel 4.9 or newer with BPF enabled, along with elevated administrative permissions.

Installing on Debian and Ubuntu

1. Update the local APT repository indices:

sudo apt update

2. Install the BCC tools package and kernel headers:

sudo apt install bpfcc-tools linux-headers-$(uname -r)

The executable is available as execsnoop-bpfcc under /usr/sbin/.

Installing on RHEL, Fedora, and Rocky Linux

Install the package using dnf:

sudo dnf install bcc-tools kernel-devel-$(uname -r)

The executable resides in /usr/share/bcc/tools/execsnoop.

Basic Process Tracing with execsnoop

To begin monitoring short-lived process executions across the entire system, run execsnoop as root:

sudo execsnoop-bpfcc

When processes spawn anywhere on the system, execsnoop outputs an event record immediately:

PCOMM            PID    PPID   RET ARGS
git              8921   8410     0 /usr/bin/git status
grep             8922   8921     0 /usr/bin/grep -i modified
sh               8923   1420     0 /bin/sh -c uptime
uptime           8924   8923     0 /usr/bin/uptime
node             8925   3110    -2 /usr/local/bin/worker

Interpreting the Output Columns

• PCOMM: The parent or calling command name executing the binary.
• PID: The Process ID assigned to the newly created process.
• PPID: The Parent Process ID that spawned the child process. This helps trace process hierarchy back to specific cron runners or supervisor daemons.
• RET: The return code of the execve() call. A value of 0 indicates successful execution. Negative integers reflect kernel error numbers (e.g. -2 indicates ENOENT: file not found; -13 indicates EACCES: permission denied).
• ARGS: The full executable path and command-line arguments passed to the program.

Advanced Filtering and Profiling Options

On high-traffic systems, capturing every command execution produces thousands of lines per minute. Use command-line arguments to isolate specific applications and investigate failures.

Tracing Failed Executions Only

Misconfigured cron jobs or corrupted scripts often attempt to invoke missing binaries repeatedly, wasting CPU cycles in silent failure loops. Filter output to show only failed executions using -x or --fails:

sudo execsnoop-bpfcc -x

This highlights missing executables (-2) or permission errors (-13) instantly.

Filtering by Process Name

To audit executions spawned by a specific tool (such as Python or Docker), pass the command name to -n:

sudo execsnoop-bpfcc -n python3

Adding Real-Time Wall-Clock Timestamps

To correlate process launches with system log entries, append the -T flag to output wall-clock timestamps:

sudo execsnoop-bpfcc -T

TIME(s)          PCOMM            PID    PPID   RET ARGS
14:35:10.412019  bash             9104   8410     0 /usr/bin/curl -s https://api.internal/health

Including Process Lineage and User IDs

To identify which system user account launched the command, add the -u flag to display the real User ID (UID):

sudo execsnoop-bpfcc -u 1000

This restricts tracing exclusively to actions performed by user ID 1000.

Diagnostic and Performance Considerations

• Zero Polling Overhead: Unlike shell scripts running ps in a continuous loop, execsnoop executes lightweight eBPF instructions only when an execve call occurs, maintaining minimal system overhead.
• Truncated Command Lines: By default, execsnoop captures the first several argument strings. If an application invokes thousands of characters of arguments, increase the argument buffer using --max-args (e.g. --max-args 64).
• Security Auditing: While execsnoop is an outstanding debugging and performance profiling tool, for tamper-evident compliance logging, pair it with Linux auditd subsystem rules.