How to Use strace to Trace System Calls and Signals in Linux Terminal

When a compiled binary or third-party service crashes without informative error logs, hangs indefinitely on startup, or fails with generic “permission denied” or “file not found” messages, standard debugging reaches an impasse without access to source code. In Linux, applications perform all hardware interactions, memory allocations, network transmissions, and file access through kernel-level system calls (syscalls). The strace diagnostic utility intercepts and records every system call invoked by a process along with received signals, revealing the exact system interaction that caused an application failure.

Installing strace Across Linux Distributions

The strace package is standard across all mainstream distribution package trees:

# Ubuntu / Debian
sudo apt update && sudo apt install -y strace

# RHEL / Fedora / Rocky Linux
sudo dnf install -y strace

# Arch Linux
sudo pacman -S strace

Tracing Program Execution from Startup

To launch a new program under active kernel tracing:

# Trace all system calls generated by a command
strace ls -l /tmp

# Save execution traces to an external log file for detailed analysis
strace -o /tmp/trace_output.txt my_application

Each output line follows a consistent format: syscall_name(arguments) = return_value. When an operation succeeds, return values typically reflect integers or file descriptors (e.g., = 3). If an operation fails, strace prints the exact errno constant and error text (e.g., = -1 ENOENT (No such file or directory)).

Filtering Specific System Call Families

Because modern programs execute thousands of memory mapping (mmap, brk) and thread coordination calls during startup, raw output can be overwhelming. The -e trace= flag filters specific subsystems:

# Trace only file-related calls (openat, read, write, close, stat)
strace -e trace=file nginx -t

# Trace only network communications (socket, connect, accept, bind)
strace -e trace=network curl https://api.digitash.com

# Trace process lifecycle events (clone, fork, execve, wait4)
strace -e trace=process /usr/local/bin/worker

# Trace IPC signals (kill, sigaction, rt_sigprocmask)
strace -e trace=signal ./app

Attaching strace to Running Daemon Processes

When a production database, web server, or background daemon hangs or consumes 100% CPU without restarting:

1. Identify the process ID (PID) using pgrep or pidof:

   pgrep -f mysqld

2. Attach strace directly to the live process in memory:

   sudo strace -p 1842

3. If the target application spawns multiple threads or child processes, add the -f (follow forks) flag:

   sudo strace -f -p 1842

4. Press Ctrl + C at any time to detach from the process without interrupting its active execution.

Generating Syscall Summary Statistics with Profiling

To diagnose system performance bottlenecks and identify which system calls consume the most kernel time:

strace -c python3 data_processor.py

The -c flag collects summary statistics across execution, printing a formatted table detailing percentage of execution time, total seconds, error counts, and call frequencies per system call.