How to Use strace in Linux to Trace System Calls, Debug Process Hangs, and Audit File Descriptors

Architectural Foundation of strace and the Linux ptrace(2) Subsystem

In the Linux operating system, user-space applications cannot interact directly with physical hardware, disk controllers, or network interfaces. Every input/output operation, memory allocation, process creation, and network socket transmission must cross the user-to-kernel boundary via a System Call (syscall). The canonical diagnostic utility for auditing this boundary is strace.

Under the hood, strace leverages the kernel’s ptrace(2) (process trace) debugging system call. When strace attaches to a target process, the kernel marks the thread with PTRACE_O_TRACESYSGOOD. Upon reaching a system call entry point, the CPU halts execution of the child process, generates a trap signal (SIGTRAP), and transfers context control to strace. Once strace reads the CPU registers to identify the syscall number and memory pointers, it signals the kernel to resume execution. Upon syscall exit, the kernel halts the thread a second time to capture the return value and error code (such as ENOENT or EAGAIN).

Syscall Class	Key Kernel Invocations	strace Filter Expression	Diagnostic Objective
File Operations	openat, read, write, close, stat	-e trace=file	Missing configuration files, permission faults
Network / IPC	socket, connect, bind, sendto, recvfrom	-e trace=network	Blocked sockets, DNS timeouts, API latency
Process Management	clone, fork, execve, wait4, exit_group	-e trace=process	Child process crashes, abnormal termination
Descriptor Lifecycle	dup, dup2, pipe, epoll_create, fcntl	-e trace=desc	File descriptor leaks, pipe buffer exhaustion
Memory & Locking	mmap, brk, mprotect, futex	-e trace=memory,ipc	Memory allocation stalls, thread lock contention

Core strace Syntax and Essential Invocation Flags

Executing strace without parameters against an active application floods the terminal buffer with thousands of unformatted lines. Systems engineers use precise filtering and formatting flags to isolate root causes cleanly.

# Basic invocation: Launch a command and monitor all syscalls
strace -f /usr/bin/my_application

# Attach non-invasively to an existing running process by PID
sudo strace -p 14205

# Track all child processes and spawned threads (-f)
# Prefix each line with microsecond timestamps (-tt)
# Measure execution duration elapsed inside each syscall (-T)
# Output clean log file rather than polluting stderr (-o)
sudo strace -f -tt -T -o /tmp/trace_debug.log -p 14205

Filtering Syscalls and Decoding File Descriptors

To avoid massive trace logs, target specific functional domains. The -e parameter accepts granular filtering expressions.

1. Isolating Missing Configuration Files:

# Trace only file system access calls
strace -e trace=file cat /etc/resolv.conf

# Trace specifically openat and access calls to identify missing shared libraries
strace -e trace=openat,access /usr/local/bin/custom_binary 2>&1 | grep ENOENT

2. Decoding File Descriptors with -y and -yy:

By default, strace prints file descriptors as raw integer handles (e.g., read(3, ...)). Appending -y instructs strace to resolve integer handles into active file paths. Appending -yy resolves socket descriptors into source and destination IP addresses and ports:

# Decode file paths and network sockets inline
sudo strace -y -yy -e trace=network,read,write -p 22014

Output transforms from ambiguous integers to actionable telemetry:

read(3</etc/ssl/certs/ca-certificates.crt>, "...", 4096) = 4096
connect(4<TCP:[192.168.1.50:52412->10.0.0.1:443]>, ... ) = -1 EINPROGRESS

Profiling Latency and Pinpointing Process Bottlenecks (-c)

When an application exhibits sluggish throughput or high CPU usage, strace can generate an aggregate statistical summary profiling total time spent inside each system call.

# Collect execution statistics across 10 seconds of production traffic
sudo strace -c -p 18452

# Alternatively, print call-by-call telemetry AND dump summary upon process exit
sudo strace -C -f -o /tmp/full_profile.log ./data_processor

Upon detaching (Ctrl+C), strace outputs a formatted benchmarking matrix:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 68.42    0.852104        1420       600           futex
 21.15    0.263412          26     10131       142 read
  8.12    0.101150          10     10131           write
  2.31    0.028751         115       250           epoll_wait
------ ----------- ----------- --------- --------- ----------------
100.00    1.245417                 21112       142 total

In this profile, futex accounted for 68.42% of total kernel execution time, immediately diagnosing severe thread lock contention or mutex deadlock.

Debugging Hung Processes: Diagnosing futex and Socket Blocks

When a background daemon stops responding to network requests or health checks, attaching strace reveals precisely what kernel event the thread is awaiting.

1. Identify the hung process thread using ps -eLo pid,tid,comm | grep worker.
2. Attach strace directly to the frozen thread:

   sudo strace -p 31204 -tt -T

3. Analyse the blocked system call:
   • Blocked on futex(0x7ffd..., FUTEX_WAIT_BITSET_PRIVATE, ...): The process is blocked waiting for another thread to release a lock. Inspect sibling threads using sudo strace -f -p <Parent_PID>.
   • Blocked on epoll_wait(5, ...): The application is idling in its event loop waiting for incoming network traffic on descriptor 5. Check network connectivity with ss -lntup | grep <port>.
   • Blocked on read(6<TCP:[...]:443>, ...): The process established an outbound connection to an external API but is stalled waiting for response data because no read timeout was configured in the code.

Managing Security Permissions and Production Overhead

On hardened Linux distributions (including Ubuntu and Debian), non-root users cannot attach strace to processes they own if Yama ptrace controls are enforced.

# Check current Yama ptrace scope
cat /proc/sys/kernel/yama/ptrace_scope

# 0: Classic ptrace permissions (Any process owned by user can be traced)
# 1: Restricted ptrace (Only parent processes can trace children; requires sudo to attach to PID)
# 2: Admin-only attach (Only processes with CAP_SYS_PTRACE can attach)
# 3: No attach (Tracing permanently disabled until reboot)

# Temporarily allow standard user ptrace attachment:
sudo sysctl -w kernel.yama.ptrace_scope=0

Troubleshooting strace Faults and Diagnostic Pitfalls

Observed Error	Probable Root Cause	Technical Diagnostic Method	Remediation Protocol
strace: attach: ptrace(PTRACE_SEIZE, ...): Operation not permitted	Yama security module blocking ptrace or process possesses CAP_SETUID capabilities	Check cat /proc/sys/kernel/yama/ptrace_scope	Execute sudo strace or lower kernel.yama.ptrace_scope to 0
Extreme Performance Degradation (10x-50x Slowdown)	strace context switching CPU for every single I/O loop in high-throughput daemon	Check CPU usage; strace saturates core handling context switches	Never run un-filtered strace in production; use targeted filters (-e trace=network) or eBPF (bpftrace)
Unfinished Syscall Lines (<unfinished ...>)	Multi-threaded process interleaving calls across shared output terminal	Output appears disjointed across multiple threads	Redirect output to file using -o <filename> (strace separates threads cleanly)
Target Process Crashes with SIGTRAP / SIGBUS	Process possesses anti-debugging protection or invalid ptrace detach handling	Inspect dmesg logs for trap faults	Detach cleanly by pressing Ctrl+C; ensure target application is not hardened against ptrace