How to Trace Linux Kernel Functions and Call Graphs Using perf ftrace

Debugging Linux kernel performance bottlenecks, device driver latency spikes, or unexpected filesystem pauses often requires observing internal kernel function calls in real time. The Linux kernel features ftrace, an internal tracing framework built directly into the kernel core. While system engineers traditionally interface with ftrace by manually mounting the debugfs or tracefs virtual filesystem and writing configuration strings to control files in /sys/kernel/tracing/, this low-level interface is cumbersome and prone to syntax errors during fast-paced incident triage.

To simplify kernel function inspection, the standard Linux performance analysis suite includes perf ftrace. This tool provides a streamlined command-line interface to ftrace engines—specifically the function tracer and the hierarchical function_graph tracer—allowing developers and administrators to trace subroutines, generate call trees, apply symbol filters, and calculate kernel execution durations with zero manual filesystem manipulation.

Understanding Function Tracers and Graph Trees

The perf ftrace utility leverages two primary tracing engines supported by the kernel:

• Function Tracer (-t function): Records every invocation of selected kernel routines. Each output record displays the timestamp, CPU core, process PID, target function name, and the parent function that initiated the call.
• Function Graph Tracer (-t function_graph): Records both function entry and function return events. It outputs a formatted C-style hierarchical call tree complete with execution duration timestamps (in microseconds), making it trivial to spot which inner subroutine consumed excessive CPU time.
• Trace Filters (-T and -G): Prevent trace buffer overflows by restricting tracing to specific functions (e.g. vfs_read, ext4_file_write_iter, or tcp_v4_rcv) or tracing the complete call tree starting from a designated root function.

Prerequisites and Kernel Tracing Support

Executing perf ftrace requires root privileges and a kernel compiled with ftrace support (standard in Ubuntu Server and production enterprise distributions).

Install the required perf userspace packages and verify kernel support:

# Install Linux performance tools
sudo apt update
sudo apt install -y linux-tools-common linux-tools-$(uname -r)

# Verify ftrace availability in the running kernel
cat /sys/kernel/tracing/available_tracers

Confirm that both function and function_graph appear in the available tracers list. Verify that kernel symbol resolution is accessible:

# Check kallsyms access permissions
sudo sysctl kernel.kptr_restrict=0

Tracing Single Kernel Functions with perf ftrace trace

To inspect when and how often a specific kernel function executes across the operating system, use the -T (target function) argument:

# Trace invocations of the Virtual File System read handler
sudo perf ftrace trace -T vfs_read

The console produces a real-time stream of executions:

# tracer: function
#
# entries-in-buffer/entries-written: 142/142   #P:8
#
#           TASK-PID     CPU#  TIMESTAMP  FUNCTION
#              | |         |       |         |
          sshd-14285   [002] 1845.102341: vfs_read <-ksys_read
          sshd-14285   [002] 1845.102349: vfs_read <-ksys_read
       systemd-1       [000] 1845.104112: vfs_read <-ksys_read
         nginx-18920   [005] 1845.105220: vfs_read <-ksys_read

The <- notation immediately identifies the caller (e.g. ksys_read called vfs_read), allowing you to trace code flow back toward user-space system calls.

Generating Hierarchical Call Trees with the Function Graph Tracer

To diagnose why a specific subsystem experiences latency, trace the complete call graph originating from a designated root function using -G (graph function) paired with the function_graph tracer:

# Trace hierarchical call tree for network packet reception
sudo perf ftrace trace -t function_graph -G tcp_v4_rcv

The output visualises nested function execution with duration timestamps:

 2)               |  tcp_v4_rcv() {
 2)               |    tcp_v4_do_rcv() {
 2)   0.140 us    |      tcp_v4_inbound_md5_hash();
 2)               |      tcp_rcv_established() {
 2)   0.210 us    |        tcp_ack();
 2)   0.180 us    |        tcp_data_queue();
 2)   1.240 us    |      }
 2)   2.150 us    |    }
 2)   3.410 us    |  }

Execution durations longer than 10 microseconds are flagged with an exclamation mark (!) and durations longer than 100 microseconds are flagged with an asterisk (*), highlighting slow hardware accesses or spinlock delays instantly.

Filtering Traces to a Specific Process ID or CPU Core

In high-traffic production environments, tracing kernel functions globally can produce overwhelming trace volumes. Narrow the capture to a specific process or CPU core:

# Trace VFS operations exclusively for process PID 14285
sudo perf ftrace trace -p 14285 -T "vfs_*"

# Trace block I/O dispatch exclusively on CPU core 3
sudo perf ftrace trace -C 3 -T blk_mq_dispatch_rq_list

Wildcards (*) are fully supported when targeting function names, allowing you to monitor an entire family of subroutines (such as all ext4 filesystem operations using -T "ext4_*").

Setting Execution Latency Filters

When tracking intermittent kernel latency spikes, logging thousands of fast subroutines wastes buffer memory. Configure a minimum latency threshold using -D (delay duration threshold) to report only operations exceeding a specified duration:

# Only display function graphs taking longer than 50 microseconds
sudo perf ftrace trace -t function_graph -G do_sys_openat2 --graph-opts depth=4

The --graph-opts depth=4 argument limits the recursion depth of the call tree, preventing deeply nested internal kernel routines from cluttering the diagnostic view.

By leveraging perf ftrace, Linux system administrators and kernel engineers gain immediate, non-invasive insight into low-level execution paths without the complexity of raw tracefs configuration.