The Limits of Application Logs
When a Linux application fails—for instance, an Nginx web server hangs indefinitely on startup, or a Python script crashes with an unhelpful “Segmentation Fault” error—the standard troubleshooting methodology is to read the application’s log files. However, application logs are written by the developer; if the developer failed to write robust error-handling code for a specific edge case, the log file will be completely empty.
When you are completely blind, you must look below the application layer and analyze how the application is interacting with the Linux Kernel itself. Every time a program wants to read a file, open a network connection, or allocate RAM, it must execute a System Call (syscall) to ask the kernel for permission to use the hardware.
The strace (System Trace) command allows administrators to intercept, record, and dump every single system call made by a process in real-time. By reading the strace output, you can see exactly which file the application failed to open or exactly which network IP it is desperately trying to reach, completely bypassing the need for application-level logs.
Step 1: Tracing a Command from Startup
If a script is crashing immediately upon execution, you can use strace to wrap the entire launch process.
Suppose a script named data_processor.sh is failing silently. Run it through the tracer:
strace ./data_processor.shThe terminal will instantly flood with thousands of lines of output. Each line represents a single system call.
execve(): The kernel executing the binary.openat(): The application attempting to open a file.mmap(): The application allocating memory.read()/write(): The application moving data.
Look at the very bottom of the massive output, right before the program crashed. You might see a line like this:
openat(AT_FDCWD, "/etc/custom_app/config.json", O_RDONLY) = -1 ENOENT (No such file or directory)This is the smoking gun. The application log was empty, but the kernel trace proves that the application crashed because it tried to open config.json, and the kernel returned an ENOENT error because the file does not exist.
Step 2: Attaching to a Running (Hung) Process
Often, an application doesn’t crash; it simply hangs and stops responding. A Node.js web server might be running and visible in htop, but it refuses to serve web pages.
You can attach strace to an already-running process using its Process ID (PID) via the -p flag.
First, find the PID of the hung application:
pgrep nodeAssume the PID is 4582. Attach the tracer (this requires root privileges):
sudo strace -p 4582If the application is completely frozen, the output might sit still on a single line:
connect(3, {sa_family=AF_INET, sin_port=htons(3306), sin_addr=inet_addr("192.168.1.100")}, 16) ...This reveals the exact root cause: The Node.js application isn’t broken. It has executed a connect() syscall attempting to reach an external MySQL database on port 3306 (at IP 192.168.1.100), and the network firewall is dropping the packets, causing the syscall to hang indefinitely while it waits for a TCP timeout.
Step 3: Filtering the Noise (-e trace=…)
Because a modern application makes tens of thousands of system calls per second, running a bare strace on a busy database server will overload your terminal and slow down the application.
You can use the -e (expression) flag to filter the trace, instructing the kernel to only intercept specific types of system calls.
If you only care about file I/O errors (e.g., figuring out which specific log file an application is writing to), filter by open and read calls:
strace -e trace=open,read -p 4582If you are debugging a complex networking issue, filter by the network subsystem. This will only show calls like socket(), bind(), connect(), and recv():
strace -e trace=network -p 4582Step 4: Timing and Profiling (-c and -T)
If an application is simply slow (not crashed or hung), strace can be used as a powerful profiling tool to see exactly where the CPU is wasting time.
Use the -c (count) flag. Instead of dumping every single syscall to the screen, strace will quietly monitor the application and, when you press Ctrl+C, print a highly formatted statistical summary table.
sudo strace -c -p 4582The table will show exactly how many times each syscall was executed, the total time spent executing them, and the error count. If you see that 95% of the application’s execution time is spent blocked on futex() calls, you know the application is suffering from severe thread-locking and contention issues.
Alternatively, if you want to see the exact microsecond duration of every individual syscall inline, append the -T flag to a standard trace.
Conclusion
strace is the ultimate diagnostic weapon of last resort. When application developers fail to write proper logging, or when complex interactions between an executable and the Linux kernel cause inexplicable hangs, the ability to intercept raw system calls provides administrators with absolute, irrefutable visibility into the exact mechanical failure of the software.