Monitoring the Linux kernel dynamically without modifying source code or rebooting the system is a critical requirement for advanced performance profiling, debugging, and security observability. The Linux kernel provides a mechanism called kprobes (Kernel Probes) to achieve exactly this. When combined with eBPF (Extended Berkeley Packet Filter), kprobes allow administrators to safely attach programmable logic to almost any kernel function.
This guide explains how kprobes function architecturally, how to deploy them for dynamic function tracing, and how to use eBPF to hook into kernel space safely to capture arguments and return values.
Understanding kprobes Architecture
A kprobe works by dynamically modifying the executing kernel code in memory. When you register a kprobe for a specific kernel function, the kprobe subsystem replaces the first few bytes of that function’s instructions with a breakpoint instruction (e.g., int3 on x86 architectures).
When the CPU hits this breakpoint, a trap occurs, halting normal execution and handing control over to the kprobe handler. The handler can inspect CPU registers (to read function arguments) or memory. Once the handler finishes executing, the kernel executes the original instruction that was replaced, and normal function execution resumes.
There are three types of probes available in the kernel:
- kprobes: Hooks into the entry point of a kernel function.
- kretprobes: Hooks into the return point of a kernel function (to capture return values or measure execution time).
- uprobes: User-space probes that hook into user-space applications (e.g., intercepting
malloc()in libc).
Prerequisites for eBPF and kprobes
To write and compile eBPF kprobes, you need the standard BPF Compiler Collection (BCC) toolchain installed. BCC abstracts much of the complex C and Python boilerplate required to interact with the eBPF subsystem.
Install the BCC tools on an Ubuntu/Debian system:
sudo apt update
sudo apt install bpfcc-tools linux-headers-$(uname -r) python3-bpfcc
You can verify that kprobes are enabled in your kernel by checking the configuration file:
grep CONFIG_KPROBES /boot/config-$(uname -r)
This should return CONFIG_KPROBES=y.
Step 1: Identifying a Kernel Function to Trace
Before you can attach a kprobe, you must identify the exact name of the kernel function you wish to trace. The kernel exposes a list of available functions via the /proc/kallsyms file.
For example, if you want to trace whenever a new process is spawned, you might look for functions related to execve (the system call used to execute a program).
sudo grep sys_execve /proc/kallsyms
On modern 64-bit systems, the system call function is usually named __x64_sys_execve.
Step 2: Writing the eBPF kprobe in C and Python
We will write a short script using the BCC library in Python. The C code (the eBPF program) will be compiled on the fly and injected into the kernel. The Python code will read the output from user space.
Create a file named trace_execve.py:
#!/usr/bin/env python3
from bcc import BPF
# Define the eBPF program in C
bpf_program = """
#include <uapi/linux/ptrace.h>
#include <linux/sched.h>
// Define a C struct to hold the data we want to pass to user space
struct data_t {
u32 pid;
char comm[TASK_COMM_LEN];
};
// Define a BPF perf event array to send data to Python
BPF_PERF_OUTPUT(events);
// The kprobe handler function
int trace_sys_execve(struct pt_regs *ctx) {
struct data_t data = {};
// Get the Process ID
data.pid = bpf_get_current_pid_tgid() >> 32;
// Get the current command (the program name)
bpf_get_current_comm(&data.comm, sizeof(data.comm));
// Submit the data to the perf buffer
events.perf_submit(ctx, &data, sizeof(data));
return 0;
}
"""
# Initialise BPF
b = BPF(text=bpf_program)
# Attach the kprobe to the kernel function
# Replace __x64_sys_execve with the correct symbol for your architecture if necessary
b.attach_kprobe(event=b.get_syscall_fnname("execve"), fn_name="trace_sys_execve")
print("Tracing execve() calls... Press Ctrl+C to exit.")
# Define a Python function to process the events emitted by the kernel
def print_event(cpu, data, size):
event = b["events"].event(data)
print(f"New Process Detected - PID: {event.pid} | Command: {event.comm.decode('utf-8', 'replace')}")
# Tell BCC to use our Python function when data arrives in the perf buffer
b["events"].open_perf_buffer(print_event)
# Loop and wait for events
while True:
try:
b.perf_buffer_poll()
except KeyboardInterrupt:
exit()
Step 3: Executing and Testing the kprobe
Because kprobes inject code directly into the kernel, running the script requires root privileges (specifically CAP_SYS_ADMIN).
sudo python3 trace_execve.py
Once the script is running, open a new terminal and run some commands (e.g., ls, cat, or top). You will immediately see output in your tracing script displaying the PID and the command name of every process being executed system-wide.
When you press Ctrl+C, the BCC library automatically detaches the kprobe, restores the original kernel instruction, and unloads the eBPF program, ensuring zero lingering performance impact.
Step 4: Using kretprobes to Capture Return Values
Sometimes, knowing that a function was called is not enough; you need to know if it succeeded or failed. This requires a kretprobe.
To attach a kretprobe using BCC, you simply use the attach_kretprobe method instead of attach_kprobe. Inside the C code, the handler accesses the return value via PT_REGS_RC(ctx).
// C Code for a kretprobe
int trace_return(struct pt_regs *ctx) {
int ret = PT_REGS_RC(ctx);
bpf_trace_printk("Function returned: %d\\n", ret);
return 0;
}
In Python:
b.attach_kretprobe(event=b.get_syscall_fnname("execve"), fn_name="trace_return")
Conclusion
Linux kprobes, especially when combined with the safety and programmability of eBPF, offer a revolutionary approach to kernel observability. By dynamically hooking into live kernel functions, administrators can build custom intrusion detection systems, profile latency bottlenecks, and debug production systems in real-time without the risk of kernel panics or the need for intrusive kernel modules.