How to Trace Kernel Events in Real Time in Linux Using trace-cmd and Ftrace

Diagnosing mysterious latency spikes, scheduling delays, and hardware interrupt bottlenecks requires deep visibility into the Linux kernel execution path. While modern eBPF tools offer high programmability, the official built-in kernel tracer, Ftrace, remains the gold standard for native, low-overhead kernel profiling across all Linux platforms. However, interacting with Ftrace directly through the debugfs or tracefs virtual file interface (reading and echoing into /sys/kernel/tracing/) is cumbersome and error-prone. The official user-space companion utility, trace-cmd, provides a streamlined command-line interface to configure tracepoints, record multi-CPU execution buffers, and generate comprehensive latency reports.

Understanding the Ftrace Architecture and trace-cmd Mechanics

Ftrace is integrated directly into mainline Linux kernels, operating without requiring custom kernel modules or third-party background daemons:

• Function Tracer: Hooks into the compiler’s -pg profiling prologue on every kernel function, allowing administrators to trace execution flow.
• Function Graph Tracer: Records both entry and exit boundaries of kernel functions, calculating exact execution durations and rendering hierarchical call trees.
• Static Tracepoints: Pre-instrumented kernel events located across the scheduler, networking stack, block I/O layer, and system call handlers.
• trace-cmd Workflow: trace-cmd configures these ring buffers, streams binary execution data into a compact trace.dat file, and formats human-readable reports on demand.

Installing trace-cmd Across Linux Distributions

Install trace-cmd using your distribution’s standard package manager:

On Debian and Ubuntu

sudo apt update
sudo apt install trace-cmd

On Fedora, RHEL, and AlmaLinux

sudo dnf install trace-cmd

On Arch Linux

sudo pacman -S trace-cmd

Discovering Available Kernel Plugins and Events

Before recording a trace session, inspect the tracer plugins and events supported by your active kernel:

1. Listing Available Tracers

trace-cmd list -t

Common outputs include function, function_graph, blk, mmiotrace, and nop.

2. Searching for Specific Event Subsystems

Ftrace organizes thousands of static tracepoints into subsystems (such as sched, net, syscalls, and block). List all available events or filter using grep:

trace-cmd list -e | grep sched
trace-cmd list -e | grep sys_enter

Recording Live Kernel Traces

Because tracing interacts with low-level kernel ring buffers, all trace-cmd recording commands require administrative privileges.

1. Tracing Scheduler Context Switches

To investigate whether CPU scheduling delays are causing application stutter, record the sched_switch event for 5 seconds:

sudo trace-cmd record -e sched:sched_switch sleep 5

The utility captures binary event buffers from all active CPU cores and writes them to trace.dat.

2. Tracing System Calls for a Specific Process

To trace every system call executed by a specific program (e.g. an Nginx web worker or database instance), attach to its PID using -P:

sudo trace-cmd record -e syscalls -P 14820 sleep 10

3. Generating a Function Call Graph

To visualize the exact execution pathway and timings inside a specific kernel subsystem (e.g., ext4 file writes), utilize the function_graph tracer filtered by function name (-g):

sudo trace-cmd record -p function_graph -g ext4_file_write_iter sleep 3

Analysing Trace Data with trace-cmd report

Once recording concludes, decode the binary trace.dat file into a structured analytical report:

trace-cmd report

Sample output from a function_graph session:

       nginx-14820 [002] 12450.320401: funcgraph_entry:      |  ext4_file_write_iter() {
       nginx-14820 [002] 12450.320403: funcgraph_entry:      |    generic_file_write_checks() {
       nginx-14820 [002] 12450.320404: funcgraph_exit:       |      0.820 us   |    }
       nginx-14820 [002] 12450.320406: funcgraph_entry:      |    ext4_da_write_begin() {
       nginx-14820 [002] 12450.320412: funcgraph_exit:       |      5.640 us   |    }
       nginx-14820 [002] 12450.320420: funcgraph_exit:       |   19.240 us   |  }

The report provides microsecond-accurate timings, highlighting slow function branches and showing exact call hierarchy.

Filtering and Profiling Event Statistics

To summarise total invocations and latencies without scrolling through hundreds of thousands of individual lines, execute the profile mode:

sudo trace-cmd record -p function_graph -l "do_sys_open*" sleep 5
trace-cmd report --profile

This generates an aggregate table listing Hit Count, Total Time, Average Time, and Maximum Execution Latency per kernel routine.

Leveraging trace-cmd provides systems engineers with unparalleled diagnostic clarity into Linux kernel behaviour without requiring code re-compilation or heavy external instrumentation.