How to Use strace in Linux to Trace System Calls, File Descriptors, and Process Latency Spikes

When a compiled binary crashes on startup with an unhelpful “Segmentation fault” or exits silently without writing anything to standard error, developers and systems engineers often find themselves staring at a black box. Similarly, when a production daemon hangs unpredictably or consumes 100% CPU during peak loads without generating application logs, standard monitoring tools cannot explain what the process is waiting on. In Linux, all applications interact with physical hardware, memory, storage disks, and network interfaces exclusively through System Calls (Syscalls).

The premier diagnostic microscope for intercepting and decoding these kernel transactions is strace. Built on the kernel’s native ptrace(2) debugging interface, strace captures every syscall made by a process—alongside arguments, buffer strings, return codes, and microsecond execution latencies. By leveraging targeted syscall filtering, tracing child threads, and profiling execution time histograms, engineers can diagnose missing configuration files, network deadlocks, and disk I/O bottlenecks without recompiling code or attaching heavy debuggers.

Architectural Overview: The Linux Syscall Boundary and ptrace(2)

strace operates by inserting itself directly into the execution path between userspace application code and the Linux kernel:

Layer	Technical Architecture	Operational Role
Userspace Process	Application Binary & Shared C Libraries (glibc)	Executes business logic in unprivileged Ring 3 CPU mode; transitions to Ring 0 via software interrupts or SYSCALL instructions.
ptrace(2) Subsystem	Kernel Process Tracing Infrastructure	Halts target process execution before entering and immediately after exiting every system call, copying registers to the tracing tool.
Syscall Table	Kernel Dispatcher (Architecture Dependent)	Maps numeric syscall registers (e.g. sys_openat, sys_read, sys_epoll_wait) to kernel handlers.
strace Parser	Format Translation & Decoding Layer	Translates raw memory pointers into human-readable strings, structure flags (e.g. O_RDONLY|O_CLOEXEC), and POSIX error constants (e.g. ENOENT).

Tracing Missing Files and Dynamic Configuration Paths

The most common use case for strace is discovering which missing configuration file, missing shared library (.so), or misconfigured permission is causing an application to fail on boot.

Step 1: Filter Strictly for Filesystem Calls

Running strace without filters emits tens of thousands of lines of output. Constrain the inspection to file opening and access calls using the -e trace flag:

# Trace file opening and access checks for a failing command
strace -e trace=open,openat,access -f ./failing_service

Step 2: Interpret Syscall Return Values

Inspect the terminal output, focusing on negative return codes:

openat(AT_FDCWD, "/etc/custom_app/config.json", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/root/.config/app/settings.yaml", O_RDONLY) = -1 EACCES (Permission denied)
openat(AT_FDCWD, "/usr/share/app/fallback.json", O_RDONLY) = 3

• = -1 ENOENT: The application expected config.json at /etc/custom_app/, but the file does not exist.
• = -1 EACCES: The process located the file, but lacks read permissions under its active user ID.
• = 3: The kernel successfully opened the fallback file, allocating File Descriptor 3.

Tracing Network Sockets and Inter-Process Communication

When an API client hangs indefinitely, trace network socket calls to determine whether the block is caused by DNS resolution, TCP handshake latency, or an unfulfilled HTTP response.

# Trace all network-related syscalls and expand buffer print length to 256 bytes
strace -e trace=network -s 256 -f curl https://internal-api.service.local

Observe the network handshake progression:

socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_TCP) = 4
connect(4, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("10.0.1.50")}, 16) = -1 EINPROGRESS (Operation now in progress)
sendto(4, "GET /v1/health HTTP/1.1\r\nHost: internal-api...\r\n\r\n", 64, MSG_NOSIGNAL, NULL, 0) = 64
recvfrom(4, "HTTP/1.1 200 OK\r\n...", 256, 0, NULL, NULL) = 256

If connect() hangs and returns ETIMEDOUT, an intervening network firewall is dropping SYN packets.

Attaching to Live Production Daemons and Tracing Child Threads

To inspect a live production process (such as a hung Nginx worker or PostgreSQL backend) without terminating it, attach strace to its Process ID (PID).

Step 1: Locate Target PID and Attach

# Identify the PID
pgrep -f "worker_process"

# Attach strace to live process and trace all child threads (-f)
sudo strace -p 14820 -f -o /tmp/daemon_trace.log

Step 2: Safe Detachment

Press Ctrl + C in the terminal session. strace detaches safely via PTRACE_DETACH, allowing the daemon to resume normal execution without interruption.

Profiling Latency Spikes and Syscall Execution Times

When an application suffers from unexplained latency spikes, use strace‘s built-in timing instrumentation to pinpoint kernel bottlenecks.

Method 1: Timing Individual Syscalls with -T

The -T flag records the time spent inside each system call in angle brackets at the end of every line:

strace -T -e trace=read,write ./database_worker

read(4, "...", 8192) = 8192 <0.000015>
write(5, "...", 8192) = 8192 <0.842105>

In this trace, read() finished in 15 microseconds, whereas write() stalled for 842 milliseconds—identifying physical disk write saturation or SAN storage latency.

Method 2: Generating a Syscall Summary Histogram with -c

To produce an aggregated performance profile showing where your process spends the majority of its CPU and I/O time:

# Run command and print summary table upon completion
strace -c ./data_processor

The output categorises execution metrics into a clean diagnostic table:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 78.40    1.245000          12    102000           futex
 15.20    0.241500           5     48000      1200 read
  4.10    0.065100          10      6500           write
  2.30    0.036500           1     24000           clock_gettime
------ ----------- ----------- --------- --------- ----------------
100.00    1.588100                180500      1200 total

Here, 78% of total kernel time is spent inside futex calls, revealing thread lock contention inside application code.

Troubleshooting strace Performance Overhead and Permissions

Consult this diagnostic reference table to prevent performance degradation and permission failures:

Issue / Error Message	Underlying Root Cause	Remediation Procedure
strace: attach: ptrace(PTRACE_SEIZE, ...): Operation not permitted	Linux security feature ptrace_scope prevents unprivileged processes from attaching to other running binaries.	Execute strace using sudo. To allow non-root debugging, set sudo sysctl -w kernel.yama.ptrace_scope=0.
Severe performance degradation when tracing production databases.	ptrace context switches double CPU overhead per syscall, which can severely impact high-throughput services.	Never run unconstrained strace in high-load production. Use specific filters (-e trace=write) or use eBPF tools (such as bpftrace) for zero-context-switch tracing.
Output truncated with ... on long string parameters.	Default string print limit is set to 32 bytes.	Increase string expansion limit via -s 1024 or -s 4096 to capture complete SQL queries or HTTP payloads.
Multi-threaded Go or Java application traces show only one thread.	Command omitted the multi-process/thread follow flag.	Always include the -f flag (strace -f ...) to follow threads created via clone(2) and fork(2).

Mastering strace syscall filtering, execution timing analysis, and histogram profiling transforms Linux debugging into an exact science, allowing developers and DevOps teams to uncover the root causes of application crashes, network timeouts, and storage latency with absolute clarity.