How to Inject Dynamic Kernel and User-Space Tracepoints in Linux Using perf probe

Debugging intermittent latency spikes, tracing unexpected function arguments, and profiling kernel drivers often requires inspecting running code without modifying source files or recompiling binaries. While static Linux tracepoints (such as sched:sched_switch or syscalls:sys_enter_read) provide comprehensive instrumentation for common kernel operations, they cannot inspect private kernel routines or user-space library functions. Traditionally, adding custom hooks required writing C kernel modules or compiling complex eBPF programs.

The standard Linux kernel performance subsystem includes a native dynamic instrumentation tool: perf probe. Leveraging the Linux kernel’s kprobes (kernel dynamic probes) and uprobes (user-space dynamic probes) subsystems alongside DWARF debugging symbols, perf probe allows systems engineers to insert tracepoints into any arbitrary instruction offset or function entry point on the fly. You can capture local variables, inspect function return codes, and record event stacks with zero runtime recompilation. This guide explains how perf probe functions, how to install required debugging symbol packages, and how to inject and record dynamic tracepoints across kernel and user-space binaries in Linux.

Understanding perf probe Architecture: kprobes and uprobes

Dynamic probing relies on hardware breakpoint traps and instruction replacement managed by the Linux kernel:

• kprobes (Kernel Probes): Injects an instruction breakpoint into running kernel memory. When the processor executes the probed memory address, the CPU triggers an exception, saves registers, passes control to the perf event handler to capture function arguments, and transparently resumes execution.
• kretprobes (Kernel Return Probes): Hooks the return address of a kernel function, allowing performance engineers to capture execution duration and return status codes (e.g. tracking whether an operation succeeded or returned -EACCES).
• uprobes (User Probes): Injects dynamic breakpoints into user-space executable binaries or shared libraries (e.g. /lib/x86_64-linux-gnu/libc.so.6 or custom C/Go/Rust binaries), tracing application execution without attaching an intrusive debugger like GDB.
• DWARF Debugging Symbols: If debugging symbols (packaged as linux-image-*-dbgsym on Debian/Ubuntu or kernel-debuginfo on RHEL/Fedora) are installed, perf probe can resolve source line numbers, function parameters, and local variable names directly from C source code.

Once injected, probes appear in the standard perf event registry under the probe: or probe_libc: namespaces and can be sampled using perf record or perf trace.

Installing Prerequisites and Debugging Symbols

Dynamic symbol resolution requires matching Linux kernel tooling and debuginfo symbols.

On Ubuntu and Debian systems, ensure linux-tools is installed, and add the debug symbols repository if line-level variable inspection is required:

# Install standard perf tools
sudo apt update
sudo apt install -y linux-tools-common linux-tools-generic linux-tools-$(uname -r)

# Optional: Enable Ubuntu ddebs repository for kernel debug symbols
sudo apt install -y ubuntu-dbgsym-keyring
echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse" | \
sudo tee /etc/apt/sources.list.d/ddebs.list
sudo apt update
sudo apt install -y linux-image-$(uname -r)-dbgsym

On RHEL, AlmaLinux, or Fedora systems:

sudo dnf install -y perf kernel-devel
sudo debuginfo-install -y kernel-$(uname -r)

Listing Available Probing Locations in the Kernel

To inspect functions available for dynamic instrumentation within the active kernel, use perf probe -F (filter functions):

# Find all network socket receive functions in the kernel
perf probe -F "sock_recv*"

# Find VFS filesystem write functions
perf probe -F "vfs_write*"

If debugging symbols are installed, you can inspect the exact source code lines and available local variables for any target kernel function using -L (list lines):

perf probe -L vfs_write

The output highlights specific source line offsets within the C source tree:

<vfs_write@/build/linux/fs/read_write.c:0>
      0  ssize_t vfs_write(struct file *file, const char __user *buf, size_t count, loff_t *pos)
      1  {
      2      ssize_t ret;
      3
      4      if (!(file->f_mode & FMODE_WRITE))
      5          return -EBADF;
      6      if (!(file->f_mode & FMODE_CAN_WRITE))
      7          return -EINVAL;

To discover what function arguments and local variables are accessible at line 0 or line 4, use -V (view variables):

perf probe -V vfs_write

Injecting Dynamic Kernel Probes (kprobes)

Once a target function and variable list are identified, inject a dynamic probe into the kernel event subsystem.

Adding a Basic Function Entry Probe

To monitor when do_sys_openat2 (the primary kernel syscall implementation behind open() and openat()) is invoked, add a dynamic probe:

sudo perf probe --add do_sys_openat2

The utility registers the probe and assigns it an event name:

Added new event:
  probe:do_sys_openat2 (on do_sys_openat2)

You can now use it in all perf tools, such as:

    perf record -e probe:do_sys_openat2 -aR sleep 1

Capturing Function Arguments and File Descriptors

To capture the actual file descriptor or memory pointer passed to a function, specify variable names during probe creation:

# Capture the file descriptor and write length in vfs_write
sudo perf probe --add 'vfs_write file count'

If full DWARF debug symbols are unavailable, you can extract CPU hardware registers directly based on the system ABI (on x86-64, the first three function arguments reside in %di, %si, and %dx):

# Probe using raw CPU architecture register arguments (%di = 1st arg, %si = 2nd arg)
sudo perf probe --add 'vfs_write_raw=vfs_write %di %si'

Adding a Function Return Probe (kretprobe)

To measure function exit status and return values (e.g. how many bytes vfs_read returned or whether an operation failed), append %return to the probe definition:

sudo perf probe --add 'vfs_read_ret=vfs_read%return $retval'

Injecting User-Space Dynamic Probes (uprobes)

In addition to probing kernel internals, perf probe can dynamically trace user-space applications, databases, and standard shared libraries without recompiling.

To probe memory allocations inside the standard C library (libc.so.6), provide the path to the shared binary via -x:

# Find target libc library path
LIBC_PATH=$(ldconfig -p | grep "libc.so.6" | awk '{print $4}' | head -n 1)

# Add a user probe on malloc entry capturing the requested allocation size
sudo perf probe -x $LIBC_PATH --add 'malloc_entry=malloc size=%di'

# Add a return probe on malloc capturing the allocated memory address
sudo perf probe -x $LIBC_PATH --add 'malloc_return=malloc%return $retval'

Recording and Inspecting Probed Events

Once dynamic tracepoints are registered, they operate identically to static tracepoint events and can be sampled using perf record.

To record 10 seconds of activity across all CPU cores for your new probe:

sudo perf record -e probe:do_sys_openat2,probe:vfs_write -a -g -- sleep 10

To inspect the recorded trace stream with argument values, use perf script:

sudo perf script

The output prints each probe hit with high-precision microsecond timestamps, process IDs, command names, and argument values:

            cat 14820 [002] 12480.192841: probe:do_sys_openat2: (ffffffff812401a0)
          nginx 15201 [001] 12480.193120: probe:vfs_write: (ffffffff8123fa40) file=0xffff88810a42b100 count=1024
       postgres 16042 [003] 12480.194012: probe:vfs_write: (ffffffff8123fa40) file=0xffff88810b19c240 count=8192

To view an aggregated statistical summary of event counts and calling processes, use perf report:

sudo perf report --stdio

Listing and Cleaning Up Dynamic Probes

Dynamic probes consume minor kernel memory tracking structures and should be removed after diagnostic investigations are complete.

To view all currently active dynamic probes registered on the system:

sudo perf probe -l

Output lists all registered kprobes and uprobes:

  probe:do_sys_openat2 (on do_sys_openat2)
  probe:vfs_write      (on vfs_write with file count)
  probe_libc:malloc_entry (on malloc in /usr/lib/x86_64-linux-gnu/libc.so.6)

To delete an individual probe by name:

sudo perf probe -d probe:do_sys_openat2

To remove all registered dynamic probes across kernel and user-space binaries in a single command:

sudo perf probe --del '*'

Verify that all dynamic probes are cleared by running perf probe -l, returning your Linux kernel to its default un-instrumented state.