How to Use the strace Command in Linux to Trace System Calls and Diagnose Process Failures

When an application in Linux crashes without informative error logs, hangs indefinitely during startup, or fails with generic exit codes, inspecting standard user-space logs is often insufficient. Underneath every compiled binary, script interpreter, or background daemon lies the system call (syscall) interface—the fundamental mechanism through which user-space software requests memory allocation, disk I/O, network sockets, and hardware execution from the Linux kernel. The strace diagnostic utility intercepts and captures these system calls in real time using the kernel’s ptrace subsystem, enabling systems engineers to uncover missing files, blocked network connections, and unhandled permission denials without access to source code.

Understanding Linux System Calls and strace Capabilities

Every time an application interacts with the operating system—whether querying the system clock with clock_gettime, opening a configuration file via openat, or transmitting TCP packets using sendto—it invokes a kernel system call. The table below outlines the primary functional switches available in strace:

Command Switch	Operational Function	Diagnostic Purpose	Performance Impact
-e trace=<set>	Filters specific syscalls or syscall classes.	Isolates file operations (openat, stat), network calls (network), or memory (memory).	Significantly reduces terminal noise and tracing overhead.
-p <PID>	Attaches to a live running process.	Inspects hung daemons or production services without restarting them.	Temporarily suspends and intercepts target thread execution.
-f	Follows child threads and forked processes.	Traces modern multi-threaded architectures (e.g., Nginx workers, JVM processes).	Essential for inspecting background child threads.
-o <filename>	Redirects diagnostic trace stream to a file.	Prevents standard output interference and saves traces for post-incident review.	Minimal; bypasses console buffer contention.
-c	Generates a statistical summary table.	Aggregates total time, error counts, and call volumes across all executed syscalls.	Executes in profiling mode; prints summary upon termination.
-tt	Appends microsecond timestamps.	Identifies latency bottlenecks and exact execution pauses down to the millisecond.	Adds precise chronological markers to each syscall log.
-s <size>	Expands string truncation limits.	Prevents truncation of long file paths, SQL queries, or data payloads (default is 32 bytes).	Increases trace log volume.

Tracing New Binaries and Analyzing Program Startup

To inspect a program from the moment of execution, prepend strace to the command string:

strace -o /tmp/nginx_init.txt -s 256 /usr/sbin/nginx -t

This syntax generates a comprehensive execution log inside /tmp/nginx_init.txt, preserving full path strings up to 256 bytes.

Deciphering the strace Output Anatomy

Each recorded line adheres to a consistent syntax structure:

openat(AT_FDCWD, "/etc/nginx/conf.d/api.conf", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

• openat(...): The intercepted system call name.
• AT_FDCWD, "/etc/nginx/conf.d/api.conf"...: The input arguments passed from the application to the kernel.
• = -1: The return value received from the kernel (a negative value signifies a failure).
• ENOENT (No such file or directory): The symbolic error macro and human-readable explanation provided by the kernel.

Filtering Specific System Call Sets

Tracing every system call generated by a modern binary produces tens of thousands of lines of output within seconds. Filtering by functional categories isolates critical events.

1. Isolating File-Level Access

To identify which configuration files an application searches for upon launch:

strace -e trace=openat,access,stat /usr/bin/python3 script.py

2. Isolating Network Socket Activity

To determine remote hostnames, IP addresses, and ports an application attempts to bind or connect to:

strace -e trace=network -f curl https://api.example.com

This captures socket creation, connect requests, bind operations, and data transmission via sendto and recvfrom.

3. Tracing Process and Thread Spawning

To track when a parent daemon creates child subprocesses:

strace -e trace=process -f ./multiprocess_worker

Attaching to Live Running Daemons with -p

When an existing production daemon—such as a database server or web worker—stops responding to incoming network requests, restarting the process erases critical diagnostic state. Instead, attach directly to its running Process ID (PID).

1. Finding the Target PID

pgrep -u www-data php-fpm

2. Attaching the Tracer

sudo strace -p 14820 -f -tt -o /tmp/php_hung_worker.log

If the worker is hung awaiting a response from an external microservice, the trace immediately highlights an uncompleted syscall:

14:32:01.402195 poll([{fd=7, events=POLLIN}], 1, -1 <unfinished ...>

The <unfinished ...> flag confirms the process is suspended awaiting data on file descriptor 7. Cross-referencing file descriptor 7 via lsof -p 14820 reveals the exact remote IP address causing the stall.

Profiling Performance Bottlenecks with strace -c

Rather than reviewing sequential lines of output, administrators can use strace -c to run a statistical benchmark identifying where an application spends the majority of its execution time.

strace -c -f /usr/bin/git status

Upon program termination, strace prints an aggregated summary table:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 58.12    0.041280           4      9821           stat
 24.15    0.017150           2      7412       120 openat
 11.20    0.007955           3      2500           read
  6.53    0.004640           1      3200           close
------ ----------- ----------- --------- --------- ----------------
100.00    0.071025                 22933       120 total

This output proves whether an application’s delay is CPU-bound, disk-bound (thousands of redundant stat calls), or blocked by memory reallocation.

Troubleshooting Common System Faults with strace

System engineers regularly diagnose the following real-world errors through trace analysis:

• Permission Denied on Existing Files (EACCES): When an application fails with a permission error despite running as the correct user, look for openat(...) = -1 EACCES. The trace reveals whether the permission failure occurred on the file itself or on an unreadable parent directory in the path.
• Missing Shared Libraries (ENOENT): If a binary fails immediately upon execution with cannot open shared object file, filter for trace=openat. The trace shows each directory traversed across the library search path (/lib/x86_64-linux-gnu, /usr/lib, LD_LIBRARY_PATH) before failing.
• Slow Network Timeouts (ETIMEDOUT): When command line tools pause for 30 seconds before executing, trace network activity. If connect(...) = -1 ETIMEDOUT appears against an internal DNS resolver, the system is attempting to resolve queries through an offline nameserver defined in /etc/resolv.conf.