The Observability Revolution
Historically, if a Linux system administrator wanted to understand exactly what was happening inside the kernel—for example, measuring the exact microsecond latency of a specific disk I/O operation or capturing packets dropping at the firewall layer—they had to write custom kernel modules in C, compile them, and inject them into the live kernel. This was highly dangerous; a single typo in a kernel module would cause a catastrophic kernel panic, instantly crashing the server.
eBPF (Extended Berkeley Packet Filter) revolutionized Linux observability. eBPF is essentially a highly secure, sandboxed virtual machine running directly inside the Linux kernel. It allows administrators to write small programs (often in C or Rust) and attach them to almost any event in the kernel (system calls, network packets, tracepoints) without modifying the kernel source code or risking a crash. If an eBPF program attempts to perform an unsafe memory operation, the kernel’s built-in verifier instantly rejects it.
While writing raw eBPF code is complex, you can leverage the BCC (BPF Compiler Collection) toolkit, which provides a massive suite of pre-written eBPF tools accessible via Python wrappers.
Step 1: Installing the BCC Toolkit
To use eBPF, you must install the BCC tools and the Linux kernel headers matching your exact running kernel version.
On Ubuntu/Debian:
sudo apt-get update
sudo apt-get install bpfcc-tools linux-headers-$(uname -r)On RHEL/CentOS/Rocky Linux:
sudo dnf install bcc-tools kernel-headers-$(uname -r) kernel-devel-$(uname -r)(Note: The tools are often installed into /usr/sbin/ or /usr/share/bcc/tools/).
Step 2: Tracing Disk I/O Latency (biolatency)
Traditional tools like iostat show average disk usage. eBPF allows you to see the exact microsecond latency of every single read/write operation as a histogram.
Execute the biolatency tool (which attaches an eBPF program to the block device tracepoints):
sudo biolatency-bpfcc -mLet it run for 10 seconds, then press Ctrl+C. The kernel will output a histogram showing exactly how many disk operations took 1 millisecond, how many took 10 milliseconds, etc. If you see operations taking over 100ms, you have definitive proof of a failing hard drive or a saturated SAN connection.
Step 3: Tracing Executed Commands (execsnoop)
If you suspect a compromised web application is spawning malicious bash shells, traditional logging might miss short-lived processes. eBPF can attach directly to the execve() system call.
sudo execsnoop-bpfccThe terminal will instantly stream every single command executed on the entire server, regardless of the user, along with the process ID (PID) and the exact arguments passed to the command.
Step 4: Tracing Network Drops (dropwatch)
If your server is experiencing mysterious network latency, the kernel might be dropping packets silently before they even reach the firewall rules.
You can use the eBPF dropwatch tool to monitor the exact kernel function where the packet died:
sudo dropwatchAt the dropwatch> prompt, type start. It will output the exact kernel memory addresses and function names (like tcp_v4_rcv or ip_error) where packets are being discarded, allowing advanced engineers to pinpoint highly obscure routing bugs.
The Power of the Sandbox
Because these tools are running inside the eBPF sandbox, they introduce near-zero overhead. You can run execsnoop or biolatency on a massive production database server handling 10,000 queries per second without causing any noticeable performance degradation, giving you unprecedented X-ray vision into the Linux operating system.