How to Use the strace Command in Linux to Debug Process System Calls and File Access

When an application crashes silently, hangs indefinitely, or returns generic permissions errors without helpful logging, standard debugging commands like ps or top offer little insight. The strace utility sits directly between user-space applications and the Linux kernel, intercepting and recording every system call (such as openat, read, write, connect, and mmap). Mastering strace empowers administrators and engineers to diagnose hidden configuration failures, missing shared objects, and process bottlenecks without modifying source code.

Understanding How strace Intercepts Kernel System Calls

Linux applications execute within an unprivileged user space. Whenever a program needs to interact with hardware, open a file descriptor, allocate heap memory, or open a network socket, it invokes a system call using software interrupts. The strace tool leverages the kernel’s ptrace facility to pause the target process at the entry and exit of every system call, decoding binary registers into human-readable arguments and return values.

Running strace on a New Command or Binary

To trace a program from initialization through termination, precede the normal execution command with strace:

strace -o trace_output.txt ls -la /var/log

The -o flag redirects diagnostic output to a file, preventing strace messages from interleaving with standard program stdout.

Filtering System Calls by Category with the -e Flag

Because an average binary invokes thousands of calls during initialization, filtering output is essential for targeted diagnostics:

• Trace File Operations: Filter strictly for file system interactions (open, stat, unlink) to locate missing files:

  strace -e trace=file nginx -t

• Trace Network Activity: Inspect network socket creation, binding, and connection handshakes:

  strace -e trace=network curl https://digitash.com

• Trace Memory Management: Track page allocations and heap boundaries (brk, mmap):

  strace -e trace=memory ./custom_daemon

• Trace Process Control: Follow process forks, execve replacements, and exit codes:

  strace -e trace=process ./worker_launcher

Attaching strace to an Active Running Process

When an already running daemon or worker hangs, you can attach strace non-destructively to its Process ID (PID):

sudo strace -p 1842 -s 256

Here, the -s 256 flag increases string preview limits from the default 32 bytes to 256 bytes, allowing you to view full SQL queries, file paths, or payload buffers. Press Ctrl + C to detach strace without stopping the monitored daemon.

Tracing Multi-Threaded and Child Fork Processes

Modern services spawn worker pools and threads to handle concurrent requests. By default, strace only tracks the parent PID. To follow all child forks and worker threads, add the -f flag:

sudo strace -f -e trace=openat,access -p 3120

Output lines are automatically prefixed with the thread ID (TID), making it straightforward to correlate specific requests with their corresponding file descriptor access.

Generating Performance Profiling Summaries with -c

Beyond live event tracking, strace can aggregate performance data into an execution histogram:

strace -c python3 script.py

Upon script completion, strace prints a summary table showing:

• The percentage of CPU time spent inside kernel space for each system call.
• The total number of invocations per syscall.
• The total number of failed invocations (errors).
• Total elapsed microseconds per system call category.

If futex or epoll_wait accounts for 90% of the execution time, the application is likely experiencing thread locking contention or waiting on sluggish external socket responses.