How to Trace System Calls and Process Signals in Linux Using perf trace

In Linux system administration and performance engineering, tracing system calls is essential for diagnosing process execution stalls, unhandled exceptions, hidden file I/O bottlenecks, and unexpected signals. While the classical strace utility has long served as the default choice for inspecting syscall invocations, its operational mechanism—based on ptrace(2)—introduces substantial execution overhead, frequently slowing target applications by factors of ten to fifty. The Linux perf tracing subsystem provides a high-throughput, low-overhead alternative: perf trace. Built directly upon Linux kernel tracepoints and eBPF instrumentation, perf trace captures syscalls, arguments, return codes, and context timings with negligible runtime penalty, making it suitable for live production environments.

Understanding perf trace Architecture vs strace

The performance differences between traditional user-space tracers and perf trace stem from their underlying kernel interfaces:

• strace (ptrace-based): Injects breakpoints at every system call entry and exit. The target process pauses, forcing two context switches per syscall while transferring control to the tracing process to copy argument buffers from memory.
• perf trace (tracepoint-based): Connects directly to kernel raw syscall tracepoints (raw_syscalls:sys_enter and raw_syscalls:sys_exit) or dynamic eBPF programs. Data buffers are collected within kernel ring buffers (perf event ring buffers) and memory-mapped to user space asynchronously.
• Latency and Distortion: Because perf trace eliminates synchronous process pauses, race conditions and performance regressions are preserved authentically without the “observer effect” skewing timing telemetry.

Prerequisites and Verifying perf trace Availability

To use perf trace, your system must have the standard Linux performance monitoring packages installed and kernel tracepoints accessible. On Debian and Ubuntu distributions:

# Install linux-tools matching active kernel release
sudo apt update
sudo apt install -y linux-tools-common linux-tools-generic linux-tools-$(uname -r)

# Check perf subsystem privileges
cat /proc/sys/kernel/perf_event_paranoid

If perf_event_paranoid is set to 2 or higher, non-root users cannot trace system-wide events. Set it temporarily for diagnostic sessions or execute commands with root privileges:

sudo sysctl kernel.perf_event_paranoid=1

Tracing Live Commands and Active Processes

The simplest way to use perf trace is to prepend it before an arbitrary binary command:

perf trace ls -la /var/log

The terminal streams formatted system call activity in real time, displaying elapsed timestamps, thread IDs, syscall names, string arguments, and return codes:

     0.041 ( 0.012 ms): ls/418242 close(fd: 3)                                 = 0
     0.057 ( 0.008 ms): ls/418242 openat(dfd: CWD, filename: "/var/log", flags: RDONLY|NONBLOCK|CLOEXEC|DIRECTORY) = 3
     0.069 ( 0.024 ms): ls/418242 getdents64(fd: 3, dirp: 0x55bc73229b40, count: 32768) = 4096
     0.114 ( 0.009 ms): ls/418242 close(fd: 3)                                 = 0
     0.142 ( 0.015 ms): ls/418242 write(fd: 1, buf: "total 8324\ndrw...", count: 83) = 83

Attaching to an Existing Daemon or Process ID

To inspect an already running service (such as an Nginx or PostgreSQL worker thread) without restarting it:

# Attach to PID 10245 and follow thread children
sudo perf trace -p 10245 --all-threads

Press Ctrl + C to detach cleanly from the target process.

Filtering Specific System Calls and Events

In high-throughput services, tracing every system call generates massive terminal output. perf trace allows precise event filtering using the -e parameter.

Tracing File System Access and Open Calls

To audit exclusively which configuration files or sockets a process attempts to open:

sudo perf trace -e openat,open,close -p 10245

Filtering Network I/O and Socket Activity

To capture inbound connections and network read/write operations:

sudo perf trace -e socket,connect,accept,accept4,recvfrom,sendto -p 10245

Tracing Inter-Process Signals

To determine what signals (such as SIGTERM, SIGHUP, or SIGKILL) are being delivered to or emitted by an application:

sudo perf trace -e kill,tgkill,rt_sigprocmask,rt_sigaction -p 10245

Filtering by Syscall Duration and Latency Thresholds

One of the most powerful capabilities of perf trace is latency-based event filtering. When troubleshooting database stalls or slow disk writes, you can configure perf trace to output only system calls that exceed a defined latency threshold:

# Show only system calls taking longer than 5.0 milliseconds
sudo perf trace --duration 5.0 -p 10245

Under this configuration, standard microsecond-level syscalls are dropped silently, instantly surfacing abnormal kernel stalls, locked mutexes, or slow physical block I/O operations.

Generating Syscall Summary and Distribution Statistics

In addition to live streaming, perf trace can aggregate system call statistics over an execution window, providing total invocation counts, accumulated runtimes, and minimum/maximum latencies:

# Run command and print summary table upon completion
perf trace -s tar -czf backup.tar.gz /etc

The generated summary table categorises syscall workload distribution:

 summary: tar (419082), 12543 events, 99.8%

   syscall            calls  errors  total(ms)   min(ms)   avg(ms)   max(ms)
   ─────────────────────────────────────────────────────────────────────────
   read               4892       0     45.210     0.002     0.009     1.240
   write              3110       0     32.180     0.003     0.010     0.890
   newfstatat         2840      12     14.520     0.001     0.005     0.410
   openat             1120       4      8.640     0.003     0.007     0.312
   close              1116       0      3.120     0.001     0.002     0.054

System-Wide Live Syscall Tracing

System administrators can also trace system calls across the entire operating system simultaneously using the -a (system-wide) flag. To monitor file creation and unlinking system-wide for 10 seconds:

# Trace system-wide file unlinking for 10 seconds
sudo perf trace -a -e unlink,unlinkat -- sleep 10

By leveraging perf trace, Linux engineers can inspect process activity, identify blocking system calls, and debug complex multi-threaded software with production-grade efficiency and minimal computational overhead.