The Black Box Problem
One of the most frustrating experiences in Linux administration is when an application simply fails to run, and the log files are completely empty. The program might crash silently, hang indefinitely, or exit with a generic “Permission Denied” error, offering absolutely no clues about what actually went wrong.
When you encounter a “black box” failure like this, standard troubleshooting tools are often useless. You need to look under the hood and see exactly how the application is interacting with the Linux kernel. This is where the strace command becomes invaluable.
Every time a program does anything meaningful—opens a file, allocates memory, connects to a network, or prints to the screen—it must ask the Linux kernel for permission. It does this by making a System Call (syscall). The strace command intercepts and records every single system call an application makes in real-time, allowing you to see exactly where the failure is occurring.
Step 1: Installing strace
While standard on many distributions, strace might not be installed by default on minimal server builds.
On Debian/Ubuntu-based systems: sudo apt install strace
On RHEL/CentOS-based systems: sudo dnf install strace
Step 2: Tracing a Basic Command
The simplest way to use strace is to pass the failing command directly as an argument.
Let’s say you have a custom script called backup_tool that is mysteriously failing. Run:
strace ./backup_tool
Your screen will immediately fill with hundreds or thousands of lines of output. This can be overwhelming. Each line represents a single system call. You will see functions like execve(), openat(), read(), and close().
The key to reading strace output is to look at the very end of the output, right before the program crashed. Furthermore, look at the right side of the equals sign (=) on each line. This is the return value of the system call. A negative number (usually accompanied by an error code like ENOENT or EACCES) indicates a failure.
Step 3: Troubleshooting a “File Not Found” Error (ENOENT)
A very common silent failure occurs when a program expects a configuration file to exist, but it doesn’t. strace will expose this instantly.
Scan the output for lines containing ENOENT (No such file or directory).
openat(AT_FDCWD, "/etc/backup_tool.conf", O_RDONLY) = -1 ENOENT (No such file or directory)
This single line tells you everything you need to know. The application attempted to open a file named /etc/backup_tool.conf, and the kernel replied that the file does not exist. You now know exactly how to fix the application.
Step 4: Troubleshooting a “Permission Denied” Error (EACCES)
Similarly, a program might fail because it lacks the necessary permissions to read a file or bind to a network port.
Look for lines containing EACCES (Permission denied).
openat(AT_FDCWD, "/var/log/backup.log", O_WRONLY|O_CREAT) = -1 EACCES (Permission denied)
Here, the program attempted to create or write to a log file, but the Linux kernel blocked it. You simply need to adjust the directory permissions or run the application as a different user.
Step 5: Filtering Output
Because strace captures everything, the output can be too noisy. You can filter the trace to only show specific types of system calls using the -e flag.
To only trace file operations (like open, read, and write):
strace -e trace=file ./backup_tool
To only trace network operations:
strace -e trace=network ./backup_tool
Step 6: Tracing an Already Running Process
If an application like a web server or database is currently hanging (but already running in the background), you can attach strace to its live Process ID (PID) using the -p flag.
First, find the PID (e.g., using htop or ps aux | grep nginx), then attach:
sudo strace -p 12345
Press Ctrl+C to detach when you have collected enough diagnostic data. The application will continue running normally.