How to Use strace to Trace System Calls, Signal Handling, and File Descriptor Leaks in Linux

When a Linux daemon crashes without leaving log traces, hangs indefinitely during startup, or consumes 100% CPU time in kernel space, standard application loggers often prove completely unhelpful. The software is executing commands beneath the application layer—making requests to the Linux kernel for file system access, network sockets, memory mapping, and process management. When these interactions fail silently, engineers must inspect the boundary between user space and kernel space.

The definitive utility for inspecting this boundary is strace. By intercepting and recording the system calls (syscalls) invoked by a process and the signals received by it, strace provides unprecedented visibility into black-box binaries, proprietary daemons, and misbehaving services. This guide provides a detailed operational breakdown of deploying strace to diagnose startup stalls, network socket timeouts, file descriptor leaks, and signal handling failures in Linux environments.

---

Architectural Mechanics: How ptrace Intercepts Syscalls

At the architectural core of strace lies the Linux kernel’s ptrace (process trace) subsystem. When strace launches a binary or attaches to a running PID:

Execution Phase	Kernel Mechanism	Technical Event
Process Attachment	PTRACE_ATTACH / PTRACE_SEIZE	The target process is designated as a tracee. The kernel stops the process with SIGSTOP and yields execution control to strace.
Syscall Entry Intercept	PTRACE_SYSCALL	The kernel pauses the tracee immediately before executing any system call, allowing strace to read argument registers (e.g., RDI, RSI, RDX on x86_64).
Kernel Execution & Exit	Syscall Completion Trap	The kernel executes the system call, pauses the tracee again on return, and exposes the return value or errno register (RAX) to strace.
Output Formatting	Symbolic Resolution	strace converts raw numeric registers and memory pointers into human-readable strings, file paths, flags, and error constants (e.g., ENOENT, EACCES).

Production Latency Warning: Because ptrace introduces two context switches for every single system call, running strace can degrade process throughput by 10x to 100x. Always scope strace filters strictly and avoid attaching unfiltered tracers to high-throughput production databases.

---

Step 1: Installing strace and Configuring Capabilities

Install strace across enterprise distributions:

# On Ubuntu / Debian systems:
sudo apt update && sudo apt install -y strace

# On RHEL / AlmaLinux / Rocky Linux / Fedora:
sudo dnf install -y strace

# Verify installation:
strace -V

Security Note on ptrace Scope: Modern Linux kernels restrict non-root users from tracing processes they do not own via the Yama Linux Security Module. To attach to arbitrary processes running under your user account, verify that /proc/sys/kernel/yama/ptrace_scope is set to 1 (standard) or execute strace with sudo.

---

Step 2: Basic Syntax and Essential Diagnostic Flags

The standard syntax for strace accepts either a command to launch or an existing process ID:

strace [OPTIONS] <COMMAND> [ARGS...]
strace [OPTIONS] -p <PID>

Core Diagnostic Flags:

Flag	Operational Role	Why It Is Critical
-f	Follow Forks	Crucial for multi-threaded or multi-process daemons (e.g., Nginx, Apache, PostgreSQL). Traces child threads created via clone or fork.
-e trace=<expr>	Syscall Filtering	Restricts output to specific syscall classes (e.g., -e trace=file, -e trace=network, -e trace=openat,read,write).
-T	Time Spent in Syscalls	Prints the microsecond duration spent inside each system call at the end of the line, instantly spotting blocking I/O calls.
-tt	Microsecond Timestamps	Prefixes each line with the absolute time of day (HH:MM:SS.microseconds) to correlate events with external service logs.
-s <size>	String Truncation Limit	Increases the maximum string size displayed for buffer arguments (default is 32 bytes; set to -s 1024 to inspect full SQL queries or payloads).
-o <filename>	Write to File	Directs strace output to a file, preventing trace logs from polluting stdout/stderr streams.
-c	Statistical Profiling	Suppresses per-call logging and displays a summary table of call counts, percentage execution time, and error totals upon completion.

---

Step 3: Diagnosing Missing Configuration Files and Path Failures

When an application fails during startup with a generic “Configuration error”, trace file access syscalls to see exactly which files it attempts to read:

# Trace file system access during application startup
strace -e trace=file -o startup_trace.log ./my_application

# Filter for failed file opens in the log:
grep 'ENOENT' startup_trace.log

Example trace output:

openat(AT_FDCWD, "/etc/myapp/config.yaml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/deploy/.config/myapp/config.yaml", O_RDONLY) = -1 EACCES (Permission denied)

This instantly reveals that the application first checked /etc/myapp/, fell back to the user’s home folder, and failed due to a permission mismatch (EACCES) on the fallback configuration file.

---

Step 4: Tracking Network Connection Stalls and Timeouts

If an API client or database connection freezes, inspect its socket calls and network latency:

# Attach to a hanging process (PID 4821), filtering network calls and measuring latency
sudo strace -f -e trace=network -T -tt -p 4821

Example trace output:

15:20:01.104291 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 4 <0.000041>
15:20:01.104380 connect(4, {sa_family=AF_INET, sin_port=htons(5432), sin_addr=inet_addr("10.0.1.50")}, 16) = -1 EINPROGRESS <0.000035>
15:20:01.104462 poll([{fd=4, events=POLLOUT}], 1, 10000) = 0 (Timeout) <10.009821>

The trace demonstrates that socket file descriptor 4 attempted a TCP connection to PostgreSQL at 10.0.1.50:5432, which stalled in poll() for exactly 10.009 seconds before timing out. The issue is an upstream firewall drop, not an application crash.

---

Step 5: Detecting File Descriptor (FD) Leaks

When long-running daemons exhaust their maximum open file limit (crashing with EMFILE: Too many open files), track file descriptor allocation and closure:

# Profile file descriptor lifecycle: opening and closing
strace -e trace=openat,close,socket -p <PID> 2>&1 | awk '
/openat|socket/ { open_count++ }
/close\(/ { close_count++ }
{ print "Active FD Differential: " (open_count - close_count) }'

If the differential grows continuously over time without stabilizing, a developer has omitted a close() call on opened files or pooled HTTP client sockets.

---

Step 6: Profiling Application Bottlenecks with Summary Reports

To identify whether a slow command is bound by disk I/O, memory allocations, or context locks, execute a statistical profile:

# Run a compilation or batch script under summary mode
strace -c python3 data_pipeline.py

Example summary table:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 68.42    0.412091         120      3434           futex
 18.25    0.109921          15      7321       120 openat
  8.12    0.048911           8      6102           read
  5.21    0.031380           9      3480           write
------ ----------- ----------- --------- --------- ----------------
100.00    0.602303                 20337       120 total

Here, 68% of the execution time is consumed by futex (fast userspace mutexes), indicating severe multithreading lock contention rather than slow disk operations.

---

Troubleshooting Common strace Operational Faults

Observed Issue	Root Cause	Diagnostic & Remediation Step
strace: attach: ptrace(PTRACE_SEIZE, ...): Operation not permitted	Missing root privileges or restricted by Linux Yama kernel security	Prefix command with sudo. Check Yama security level via sysctl kernel.yama.ptrace_scope. Set temporarily to 0 if non-root tracing is required.
Process freezes or performance degrades catastrophically during trace	Tracing high-frequency syscalls (such as read, write, or gettimeofday) on a live server	Never run naked strace -p <PID> without filters on production nodes. Scope specifically to target syscalls (e.g., -e trace=openat,connect) to minimise overhead.
Child threads are missing from output in multi-threaded application	Omitted the -f (follow forks) parameter upon launching the tracer	Always supply the -f flag (e.g., strace -f -p <PID>) to instruct the kernel to attach breakpoints to threads spawned via clone().
Trace output is truncated with ... on critical strings	Default string buffer truncation limit (32 bytes) reached	Pass the -s <size> argument (e.g., strace -s 2048 -e trace=write ...) to expand the maximum printed string length.

---

Operational Governance and Best Practices

• Redirect Trace Streams to Separate Files: Always pass -o /tmp/strace_output.log rather than piping stdout/stderr. Piping directly can alter application timing or cause deadlocks on stdout buffering.
• Prefer eBPF for Always-On Production Profiling: While strace is unmatched for interactive triage, its ptrace overhead makes it unsuitable for permanent telemetry. For continuous kernel observability, deploy eBPF-based tools (such as bpftrace or BCC scripts like syscount).
• Detach Cleanly: When detaching from a live process, press Ctrl + C. strace intercepts the interrupt, detaches the ptrace hook cleanly, and allows the target process to resume normal uninterrupted execution.