How to Use bpftrace in Linux to Trace Kernel Function Latency, VFS Calls, and Syscall Arguments

Pinpointing performance bottlenecks in modern Linux infrastructure requires observability beyond coarse system metrics like CPU utilization or load averages. When production services experience sporadic latency spikes, diagnosing whether the delay originates in disk I/O, socket buffer saturation, file system locking, or kernel memory allocation traditionally involved invasive kernel rebuilds or high-overhead tracing tools. bpftrace revolutionises Linux performance analysis by providing a high-level, expressive tracing language powered by extended Berkeley Packet Filter (eBPF).

Operating entirely in-kernel without kernel modification, bpftrace compiles high-level scripting logic down into sandboxed eBPF bytecode verified by the Linux kernel verifier. This enables system reliability engineers, performance specialists, and kernel developers to safely inspect live function calls, measure syscall latency down to nanoseconds using power-of-two histograms, and dissect internal kernel function arguments in production environments with virtually zero overhead.

Architectural Overview: The bpftrace Tracing Engine

bpftrace interacts with the Linux kernel tracing infrastructure across several distinct probe layers:

Probe Type	Kernel Mechanism	Operational Role	Safety & Stability
kprobe / kretprobe	Kernel Dynamic Probes	Attaches dynamically to arbitrary internal kernel function entries and returns.	High capability; non-stable interface across differing Linux kernel releases.
tracepoint	Kernel Static Tracepoints	Hooks into stable, pre-defined trace instrumentation points compiled into the kernel.	Stable API across kernel updates; highly recommended for production scripts.
uprobe / uretprobe	User-Space Dynamic Probes	Instruments user-space executable binaries and shared libraries (e.g. libc, openssl).	Inspects unstripped userland function entries, arguments, and return values.
profile / interval	Timer Interrupts	Fires on all CPUs at a set frequency (e.g. 99 Hz) or periodic intervals (e.g. 1s) for sampling.	Used for off-CPU analysis, CPU profiling, and periodic summary aggregations.
software / hardware	Performance Monitoring Counters	Instruments hardware PMC events (cache misses, branch mispredictions, page faults).	Direct microarchitectural insight into hardware-level pipeline stalls.

Installing bpftrace and Verifying Kernel Subsystems

bpftrace requires Linux kernel 4.9 or higher with eBPF support enabled (CONFIG_BPF_SYSCALL=y, CONFIG_BPF_JIT=y, CONFIG_FTRACE=y). Modern distributions package bpftrace directly.

Step 1: Install bpftrace and Kernel Debugging Headers

On Ubuntu and Debian systems, install the tracing suite via apt:

# Update package repositories and install bpftrace
sudo apt update
sudo apt install -y bpftrace linux-headers-$(uname -r) bpfcc-tools

Step 2: Verify Kernel Tracepoint Accessibility

Confirm that your user has root privileges and that debugfs or tracefs is mounted:

# Verify mount point of tracefs
mount | grep -E "(debugfs|tracefs)"

# Test basic probe discovery
sudo bpftrace -l 'tracepoint:syscalls:sys_enter_*' | head -n 10

Mastering Essential bpftrace One-Liners

bpftrace allows powerful diagnostic one-liners that deliver immediate insights directly from the command line.

1. Summarising Syscalls by Process Name

Identify which processes are generating the highest frequency of system calls:

# Count system calls per process command name across a 5-second window
sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); } interval:s:5 { exit(); }'

2. Measuring VFS Read Latency with Power-of-Two Histograms

VFS (Virtual File System) reads represent the bridge between application requests and filesystem drivers. Visualise VFS read duration in microseconds using the built-in hist() accumulator:

# Measure vfs_read latency in microseconds and plot as log2 histogram
sudo bpftrace -e '
kprobe:vfs_read {
    @start[tid] = nsecs;
}
kretprobe:vfs_read /@start[tid]/ {
    $duration_us = (nsecs - @start[tid]) / 1000;
    @vfs_read_latency_us = hist($duration_us);
    delete(@start[tid]);
}'

Press Ctrl + C after capturing traffic. bpftrace outputs an ASCII log2 distribution chart detailing min, max, and bucketed frequency counts.

3. Tracing File Open Paths with Process and Return Codes

Capture file opening requests across the system in real time, inspecting arguments passed to do_sys_openat2:

# Trace file opens including filename and process ID
sudo bpftrace -e '
tracepoint:syscalls:sys_enter_openat {
    printf("%-6d %-16s %s\n", pid, comm, str(args.filename));
}'

Constructing an Advanced bpftrace Diagnostic Script

For persistent monitoring, save multi-line bpftrace programs into executable script files ending in .bt.

Step 1: Write the Block I/O Latency Inspector

Create a script named block_io_trace.bt to profile block device queuing delays and request sizes:

nano block_io_trace.bt

Add the following code:

#!/usr/bin/env bpftrace

BEGIN
{
    printf("Tracing block I/O latency... Hit Ctrl-C to end.\n");
}

/* Capture timestamp upon I/O request insertion into queue */
tracepoint:block:block_io_start
{
    @start[args.sector] = nsecs;
}

/* Measure time elapsed upon I/O request completion */
tracepoint:block:block_io_done
/@start[args.sector]/
{
    $latency_ms = (nsecs - @start[args.sector]) / 1000000;
    @io_latency_ms[args.rwbs] = hist($latency_ms);
    @io_size_bytes[args.rwbs] = hist(args.bytes);
    delete(@start[args.sector]);
}

END
{
    printf("\n--- Block Device Latency & Size Breakdown ---\n");
}

Step 2: Execute and Analyse Script Output

# Make script executable and run with elevated root permissions
chmod +x block_io_trace.bt
sudo ./block_io_trace.bt

The program separates read (R), write (W), and discard (D) operations, plotting distinct latency profiles for each I/O type.

Troubleshooting bpftrace Execution and Verifier Errors

Consult this diagnostic matrix to resolve probe attachment errors, BPF verifier rejections, and runtime memory limits:

Error Message	Underlying Root Cause	Remediation Procedure
Permission denied / Failed to initialize bpftrace	Process lacks CAP_BPF and CAP_PERFMON (or root privileges), or kernel restricts unprivileged BPF.	Execute with sudo. Verify sysctl restrictions via sudo sysctl kernel.unprivileged_bpf_disabled=1.
Attaching to kprobe failed: No such file or directory	The kernel function targeted by kprobe has been inlined, renamed, or stripped in the running kernel.	Check available functions in /proc/kallsyms via grep <function_name> /proc/kallsyms. Prefer static tracepoints which remain stable across releases.
BPF program too large / Verifier limit exceeded	Script includes unbounded loops or exceeds maximum BPF instruction limits (1 million instructions).	Refactor loops into unrolled logic or map reductions. Avoid storing deep string structures inside associative maps.
Lost N events / Buffer overflow	Probe fires faster than the user-space daemon can drain the perf event ring buffer.	Increase buffer size using BPFTRACE_PERF_RB_PAGES=64 bpftrace .... Filter events inside the kernel predicate clause (e.g. /args.bytes > 4096/) to reduce event generation volume.
Failed to mmap tracefs / debugfs not mounted	Kernel debug filesystem is unmounted or restricted in container environments.	Mount tracefs manually: sudo mount -t tracefs nodev /sys/kernel/tracing or grant the container --privileged host access.

Leveraging bpftrace transforms Linux performance engineering from guesswork into deterministic, empirical analysis, unlocking real-time visibility into kernel internals with unprecedented precision and minimal overhead.