How to Trace Kernel Function Latency Histograms in Linux Using funclatency

Pinpointing performance bottlenecks in production Linux environments often requires examining how long individual kernel routines take to complete. While aggregate execution counters indicate how frequently an event occurs, they fail to reveal latency variance—such as whether slow calls are caused by occasional severe tail latency or consistent minor delays. The funclatency tool, part of the BPF Compiler Collection (BCC), solves this by attaching eBPF kprobes and kretprobes to measure execution durations and render visual power-of-two or linear latency histograms directly in your terminal.

Understanding How funclatency Profiles Kernel Functions

When you instrument a kernel function using traditional tracing tools like ftrace or raw perf, logging every single event to user space introduces substantial overhead. By contrast, funclatency leverages in-kernel BPF maps. When a designated function is entered, a BPF kprobe records a high-resolution timestamp keyed to the current thread ID. When the function returns, a kretprobe calculates the delta between the entry and exit timestamps, records the elapsed time in an internal histogram bucket, and deletes the temporary entry.

Because histogram calculations happen inside the kernel, thousands of function invocations can be processed every second with minimal CPU overhead. Much like measuring thread scheduling delays with offcputime or assessing queue delays via wakeuptime, funclatency delivers detailed visibility into tail latencies that average metrics invariably conceal.

Installing Prerequisites and BCC Tools

To run funclatency, your system requires kernel headers matching your active Linux release and the BCC tools package. On Debian and Ubuntu systems, run the following command in terminal:

sudo apt update && sudo apt install -y bpfcc-tools linux-headers-$(uname -r)

On Red Hat Enterprise Linux, Fedora, or Rocky Linux, install the package using dnf:

sudo dnf install -y bcc-tools kernel-devel-$(uname -r)

Depending on your distribution, BCC binaries are typically installed to /usr/sbin/ or symlinked with a -bpfcc suffix (such as funclatency-bpfcc). Ensure your user account possesses sudo or CAP_BPF privileges.

Tracing Basic Kernel Function Latency

To record latency distributions for a specific kernel function, pass the function name as an argument. For instance, to evaluate the execution duration of virtual filesystem read operations (vfs_read), execute:

sudo funclatency vfs_read

The utility attaches probes immediately and runs quietly in the background. Allow it to sample live traffic for several seconds or minutes, then press Ctrl + C to terminate collection and display the distribution:

Tracing 1 functions for "vfs_read"... Hit Ctrl-C to end.
^C
     nsecs               : count     distribution
         0 -> 1           : 0        |                                        |
         2 -> 3           : 0        |                                        |
         4 -> 7           : 0        |                                        |
         8 -> 15          : 12       |*                                       |
        16 -> 31          : 84       |******                                  |
        32 -> 63          : 512      |****************************************|
        64 -> 127         : 289      |**********************                  |
       128 -> 255         : 94       |*******                                 |
       256 -> 511         : 18       |*                                       |
       512 -> 1023        : 3        |                                        |

The output displays elapsed time intervals in nanoseconds on the left, the count of occurrences in the centre, and an ASCII bar chart on the right showing the distribution curve.

Customising Time Units and Interval Reporting

By default, funclatency displays durations in nanoseconds. For longer operations like block I/O or network socket timeouts, tracking nanoseconds creates excessively granular tables. Use flags to adjust time units:

• -u: Display durations in microseconds (μs).
• -m: Display durations in milliseconds (ms).

To capture disk sync latency across filesystem writebacks in milliseconds, run:

sudo funclatency -m sync_filesystem

If you want to monitor latency continuously without manually pressing Ctrl + C, provide an output interval in seconds followed by an optional iteration count:

sudo funclatency -m -i 5 do_sys_openat2 3

This command outputs a fresh latency histogram every 5 seconds, repeating 3 times before automatically exiting, making it ideal for automated benchmark logging.

Filtering Latency Traces by Process and Thread

In multi-tenant or containerised environments, tracing a global kernel function can mix data from unrelated background workloads. You can narrow down latency collection using process identifiers:

sudo funclatency -p 14820 -u vfs_write

This restricts the trace to PID 14820, ensuring you only measure disk write latencies generated by your designated database or web server application. If you also wish to examine memory subsystem behaviour under load, pairing this analysis with slabratetop helps establish whether cache memory pressure is contributing to downstream function delays.

Using Wildcards to Trace Related Kernel Routines

Frequently, an operation is serviced by multiple interrelated routines. Rather than running separate tracing sessions, funclatency accepts POSIX regular expressions or wildcards:

sudo funclatency -u 'tcp_send*'

When multiple matching functions are probed simultaneously, funclatency prints separate, clearly labelled latency histograms for each matching symbol (such as tcp_sendmsg and tcp_send_ack), allowing you to contrast network protocol delays across internal execution paths.