How to Use the strace Command in Linux to Troubleshoot and Debug Processes

When an application on Linux crashes unexpectedly, prints a vague error message, or hangs indefinitely without logging details, standard log files often fail to explain what went wrong. The strace utility provides visibility into Linux software by intercepting and recording the system calls made by a process and the signals received by it. Because system calls represent the direct interface between user-space applications and the Linux kernel, strace reveals every file read, memory allocation, network socket connection, and error code in real time.

Whether tracking down a missing configuration file, diagnosing permission denials, or identifying process deadlocks, strace is an essential tool in a system administrator’s toolkit. In this guide, we explore core strace syntax, demonstrate practical debugging filters, trace live running daemons, and analyse common failure patterns.

Understanding Linux System Calls

Every time a program interacts with the physical system—reading a file, opening a network port, spawning a child process, or querying system time—it invokes a kernel system call (syscall). Here are the primary syscall categories intercepted by strace:

System Call Class	Key Syscalls	Diagnostic Value
File Operations	openat, read, write, close, stat	Finds missing files, permission errors, and corrupted path lookups
Process Management	execve, fork, clone, wait4, exit_group	Diagnoses segmentation faults, child termination, and binary launches
Network Sockets	socket, connect, bind, sendto, recvfrom	Uncovers refused connections, DNS delays, and binding conflicts
Memory Allocation	mmap, brk, munmap	Pinpoints out-of-memory limits and buffer allocation failures

When investigating network socket failures reported by strace, pairing your diagnosis with the Netcat command in Linux allows you to verify remote endpoint reachability independently.

Basic Syntax and Tracing a Command

To run any binary executable under strace, prefix the target command with strace:

strace ls -l /tmp

Because system call output is sent to standard error (stderr), it immediately floods your terminal. To capture the diagnostic stream to an external file for analysis, supply the -o flag:

strace -o trace_output.txt ls -l /tmp

Filtering Specific System Calls with -e

Unfiltered traces can generate tens of thousands of lines in seconds. You can isolate specific operations using the -e trace= filtering flag.

1. Tracing Only File Opens

If an application fails stating ‘configuration not found’ but fails to specify the file path, filter by file system calls:

strace -e trace=openat,stat my_application

Lines returning ENOENT (No such file or directory) immediately expose the exact directory paths the binary attempted to search.

2. Tracing Network Connections

To observe all outbound network socket attempts:

strace -e trace=network curl https://example.com

Attaching strace to a Live Running Process (-p)

When an existing daemon or background worker hangs or consumes high CPU, attach strace directly to its Process ID (PID) without restarting the service:

sudo strace -p 14285

If the target process spawns multiple worker threads or forks child processes (such as Nginx, Apache, or Python workers), pass the -f flag to trace all descendants:

sudo strace -f -p 14285

Press Ctrl+C at any time to detach strace cleanly. The running process will continue unaffected.

Profiling Performance and Execution Bottlenecks (-c)

If an application is running slowly and you want an executive summary of time spent inside kernel operations, use the -c summary flag:

strace -c du -sh /var/

Once execution finishes, strace prints an informative statistical table:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 58.12    0.412015           4     98214           newfstatat
 25.10    0.178002           6     28450           getdents64
 12.30    0.087220           3     28450           close
  4.48    0.031780           3     10214           openat
------ ----------- ----------- --------- --------- ----------------
100.00    0.709017                165328           total

This breakdown immediately indicates whether an application is constrained by disk I/O, network waiting, or CPU processing.

For system maintenance after extensive trace debugging, remember to clear accumulated diagnostic logs using journalctl vacuum in Linux to reclaim server storage.

Frequently Asked Questions

Does strace cause performance overhead on production servers?

Yes. Because strace forces context switches between kernel space and user space for every single intercepted syscall, it can slow down the target process by 50% to 90% while attached. Use it sparingly on high-traffic production daemons, or use lightweight eBPF tools like bpftrace where appropriate.

Why do I get ‘Operation not permitted’ when running strace on a process?

Tracing other users’ processes requires root permissions or the CAP_SYS_PTRACE capability. Run sudo strace -p <PID>. Additionally, modern distributions enforce Yama ptrace hardening in /proc/sys/kernel/yama/ptrace_scope.

Can strace reveal passwords or encryption keys?

Yes. If a program reads plain-text credentials or API tokens through read() syscalls, those strings will appear in the trace log. Always inspect and redact trace files before publishing them to public forums.