How to Use the strace Command in Linux to Trace System Calls and Diagnose Process Failures

When an application on Linux terminates abruptly without writing to log files, hangs indefinitely on startup, or returns cryptic Permission denied or File not found messages despite correct file permissions, debugging compiled binaries can feel impossible. In Linux, applications interact with the kernel through System Calls (syscalls)—requesting memory allocation, reading files, opening network sockets, and spawning child threads. The strace utility intercepts and records every system call made by a process and the signals received, revealing the exact point of software failure.

Basic Syntax and Fundamental Behavior of strace

You can execute a new command directly under strace, or attach it dynamically to an existing running process using its Process ID (PID):

strace [options] command [args]
sudo strace [options] -p <PID>

By default, strace writes its trace log directly to stderr. Because typical applications invoke hundreds of system calls during initialization, redirecting the output to a dedicated file using -o is strongly recommended:

strace -o /tmp/trace_output.txt ls -l /var/www

Essential strace Flags for Deep System Diagnostics

1. Filtering Specific System Calls (-e trace=)

Rather than drowning in thousands of memory management calls (like mmap and brk), use the -e expression filter to isolate specific categories of kernel interactions:

• -e trace=file: Traces only file access calls (such as openat, stat, access, unlink). Perfect for discovering which missing configuration file caused an app to crash.
• -e trace=network: Traces only network socket calls (like socket, connect, bind, sendto).
• -e trace=process: Traces process lifecycle operations (including fork, clone, execve, wait4).

strace -e trace=openat,access my_application

2. Generating System Call Summary Profiles (-c)

When profiling slow applications or identifying performance bottlenecks, passing -c instructs strace to run in statistical profiling mode. Rather than printing every call sequentially, it accumulates data and outputs a tidy table upon process termination:

strace -c python3 data_processor.py

The output reports the percentage of time spent in each system call, total call counts, total errors, and total microseconds consumed, immediately highlighting whether the bottleneck stems from disk read stalls or memory allocation contention.

3. Recording Wall-Clock Timestamps (-t and -tt)

To identify where a hung process pauses or stalls, prefixing log entries with exact microsecond timestamps allows you to pinpoint the hanging kernel call:

strace -tt -T -o /tmp/hang_audit.txt ./service_daemon

The -tt flag prepends hour, minute, second, and microseconds to each line, while -T reports the exact duration spent inside each discrete system call.

4. Tracing Forked Child Threads and Processes (-f)

Modern applications spawn worker processes and multithreaded pools. By default, strace tracks only the parent process. To trace all child threads and sub-processes spawned via fork() or clone(), pass the -f switch:

strace -f -e trace=file -o /tmp/docker_trace.txt docker build .

Common Diagnostic Patterns with strace

Symptom	Syscall to Inspect	Expected Error in strace Log
Missing Config / Shared Library	openat()	ENOENT (No such file or directory)
Hidden Permission Fault	access(), openat()	EACCES (Permission denied)
Network Connection Hang	connect()	EINPROGRESS or ETIMEDOUT
Storage Disk Full	write()	ENOSPC (No space left on device)