How to Trace and Count VFS Calls in Linux Using vfscount

In high-throughput Linux environments, storage I/O latency and filesystem contention represent primary performance bottlenecks for databases, web caches, and application runtimes. When diagnosing input/output performance, engineers often inspect device-level counters using tools like iostat. However, block device statistics only capture operations that reach physical storage controllers, completely bypassing interactions handled directly by the Virtual Filesystem Switch (VFS) and the Linux page cache.

The Linux VFS provides an abstraction layer that handles file operations across diverse backing filesystems, including ext4, XFS, Btrfs, and network mounts like NFS. Understanding the aggregate volume and distribution of VFS calls helps pinpoint whether an application is generating excessive filesystem traversals, directory lookups, or unbuffered reads. The BPF Compiler Collection (BCC) provides vfscount, an eBPF tracing utility that counts VFS operations in-kernel with near-zero overhead. This guide shows how to trace and count VFS calls in Linux using vfscount.

Understanding How vfscount Functions

The vfscount utility uses extended Berkeley Packet Filters (eBPF) to dynamically attach kprobes to core VFS kernel functions matching the pattern vfs_*. These include essential filesystem operations:

• vfs_read: Read requests dispatched by user-space applications.
• vfs_write: Write requests committed to memory buffers or disk.
• vfs_open: File handle opening and descriptor allocation.
• vfs_create: Creation of regular filesystem nodes.
• vfs_fsync: Explicit file synchronisation flushes to persistent media.
• vfs_statx: Metadata queries and file attribute inspections.

Rather than streaming individual trace events to user space (which incurs substantial context-switching overhead), vfscount maintains an in-kernel BPF hash map, incrementing integer counters directly inside kernel space. When execution completes, it dumps the aggregated totals cleanly to the terminal.

Installation and System Prerequisites

The tool requires Linux kernel 4.9 or newer with BPF support enabled, along with root or CAP_SYS_ADMIN capabilities.

Installing on Debian and Ubuntu

1. Update the local APT repository cache:

sudo apt update

2. Install the BCC tools package and corresponding kernel headers:

sudo apt install bpfcc-tools linux-headers-$(uname -r)

On Debian and Ubuntu systems, the utility is installed as vfscount-bpfcc under /usr/sbin/.

Installing on RHEL, Fedora, and Rocky Linux

Install the package via dnf:

sudo dnf install bcc-tools kernel-devel-$(uname -r)

The tool is located at /usr/share/bcc/tools/vfscount.

Tracing VFS Calls System-Wide

To start monitoring VFS call frequency across the entire system, execute vfscount with administrative privileges:

sudo vfscount-bpfcc

The command attaches probes to all active vfs_* entry points and begins aggregating counts. Allow it to run for your desired profiling interval (e.g. 10 to 30 seconds), then press Ctrl+C to stop tracing and print the report:

Tracing... Hit Ctrl-C to end.
^C
ADDR             FUNC                          COUNT
ffffffffb120c1a0 vfs_fsync                         4
ffffffffb120a440 vfs_create                       18
ffffffffb1209b10 vfs_open                        482
ffffffffb120a890 vfs_statx                      1240
ffffffffb1208920 vfs_write                      8921
ffffffffb1208450 vfs_read                      45120

Interpreting the Output Columns

• ADDR: The kernel virtual memory address where the probed VFS function resides.
• FUNC: The exact VFS kernel function hooked by the eBPF probe.
• COUNT: The total number of invocations observed across all system processes during the profiling window.

Analysing Workload Characteristics with vfscount

The count distribution provides immediate diagnostic insights into system workload behaviour:

1. Read-Heavy vs Write-Heavy Workloads

Comparing vfs_read against vfs_write reveals the real application I/O balance. If vfs_read is extremely high while block device statistics (via iostat) remain flat, your application is reading extensively from page cache memory without generating physical disk reads.

2. Excessive Metadata Traversal

High volumes of vfs_statx, vfs_getattr, or vfs_open relative to actual read/write operations indicate that an application or container runtime is performing excessive path checking, directory scanning, or unoptimised file existence verification (common in build pipelines and file indexers).

3. Synchronous Flush Contention

A high count for vfs_fsync indicates that databases or logging frameworks are constantly forcing synchronous disk flushes. Because fsync blocks user-space execution until data hits physical non-volatile storage, excessive counts point directly to transaction commit bottlenecks.

Performance and Operational Considerations

• Production-Safe eBPF Counting: Because vfscount updates in-kernel hash maps rather than sending per-event records over perf rings, CPU overhead remains negligible even when handling hundreds of thousands of operations per second.
• Kernel Function Inlining: On custom or bleeding-edge kernels where specific static VFS functions might be inlined by the compiler, kprobes cannot attach to inlined symbols. The core API functions (vfs_read, vfs_write, vfs_open) are standard non-inlined kernel symbols and always trace reliably.
• Complementary Tools: Use vfscount for macro-level frequency audits. If you identify an unexpected spike in specific functions, follow up with fileslower to identify individual slow operations or filetop to attribute calls to specific Process IDs.