How to Trace Filesystem Sync Latency and Flush Calls in Linux Using syncsnoop

Storage performance issues in production Linux systems often stem not from sustained read or write volume, but from periodic, blocking filesystem synchronisation calls. When an application invokes sync(), syncfs(), fsync(), fdatasync(), or msync(), the kernel flushes dirty page cache buffers directly to non-volatile block storage. While necessary for data integrity and database transaction durability, these synchronisation calls force calling threads to block until the underlying hardware acknowledges data persistence. Using syncsnoop, an extended Berkeley Packet Filter (eBPF) diagnostic tool from the BCC toolkit, administrators can trace flush calls system-wide, pinpointing the processes and storage latencies responsible for unexpected application freezes.

How Filesystem Flush Calls Impact Linux System Performance

Linux employs aggressive write-back page caching to maximise I/O throughput. When an application writes data, the kernel marks memory pages as “dirty” and queues them for asynchronous flushing by background kworker threads. However, database management engines (such as PostgreSQL, MySQL, and SQLite), message brokers, and container runtimes explicitly issue fsync() to guarantee that write-ahead logs (WAL) reach persistent storage before acknowledging transactions.

A rogue application calling global sync() forces every dirty buffer across the entire operating system to flush to disk simultaneously. Under heavy write workloads, this creates a “flush storm” that saturates the block device queue, driving disk latency from fractions of a millisecond to tens of seconds. Standard diagnostic utilities like iostat or vmstat report high I/O wait times but cannot identify which specific PID or system call initiated the flush. syncsnoop solves this observability gap by attaching eBPF kprobes directly to kernel entry and return points for sync syscalls.

Installing the BCC Tools and syncsnoop on Linux

syncsnoop is packaged within the BPF Compiler Collection (BCC). To install it on modern Debian, Ubuntu, and Red Hat distributions, ensure kernel headers matching your running kernel are installed alongside the tooling:

On Ubuntu and Debian systems:

sudo apt update
sudo apt install -y bpfcc-tools linux-headers-$(uname -r)

On RHEL, AlmaLinux, and Rocky Linux systems:

sudo dnf install -y bcc-tools kernel-devel-$(uname -r)

On Ubuntu/Debian distributions, BCC executables are appended with a -bpfcc suffix and installed into /usr/sbin/. Verify that syncsnoop is accessible:

which syncsnoop-bpfcc || which /usr/share/bcc/tools/syncsnoop

Running syncsnoop to Trace Real-Time Sync Calls

To trace every synchronisation call as it occurs across all running processes, execute syncsnoop-bpfcc with administrative privileges:

sudo syncsnoop-bpfcc

The tool monitors system calls continuously, displaying a timestamped trace line whenever a synchronisation syscall returns:

TIME(s)            COMM             PID     CALL
2.415891           postgres         18421   fsync
3.104285           jbd2/nvme0n1p2-8 312     fdatasync
4.892104           dockerd          1490    fsync
5.210492           systemd-journal  642     fdatasync
8.114298           backup-script.sh 28912   sync

Key output fields include:

• TIME(s): Relative elapsed time in seconds since the trace began.
• COMM: Name of the executable or kernel thread invoking the synchronisation call.
• PID: Process ID responsible for the call.
• CALL: Specific system call executed (sync, syncfs, fsync, fdatasync, or msync).

Tracing Duration and Measuring Flush Latency

Identifying which process triggers a sync is only half the diagnosis; understanding how long the thread was blocked waiting for storage hardware is equally crucial. To record the round-trip latency of each flush call, pass the -d (duration) flag:

sudo syncsnoop-bpfcc -d

The output now includes an additional column recording execution duration in milliseconds:

TIME(s)            COMM             PID     CALL       LAT(ms)
1.149204           postgres         18421   fsync      0.42
2.841029           sqlite3          24810   fsync      1.89
5.918402           heavy-writer.py  29104   sync       4280.15
6.012948           redis-server     1512    fdatasync  18.40

In this example, heavy-writer.py called sync(), blocking the system for over 4.2 seconds while flushing accumulated write caches. Any transactional workload running concurrently during this window would experience immediate throughput degradation.

Filtering and Tracing Specific Processes

If you are diagnosing an isolated service—such as an enterprise database server or container engine—filtering out background OS flushes simplifies telemetry analysis. While BCC’s syncsnoop monitors system-wide events by default, bpftrace provides a lightweight one-liner alternative to filter by process name or PID while capturing call latency:

sudo bpftrace -e '
tracepoint:syscalls:sys_enter_fsync,
tracepoint:syscalls:sys_enter_fdatasync,
tracepoint:syscalls:sys_enter_sync {
    @start[tid] = nsecs;
}
tracepoint:syscalls:sys_exit_fsync,
tracepoint:syscalls:sys_exit_fdatasync,
tracepoint:syscalls:sys_exit_sync /@start[tid]/ {
    $duration_us = (nsecs - @start[tid]) / 1000;
    if ($duration_us > 5000) {
        printf("%-16s PID %-6d syscall completed in %d us\n", comm, pid, $duration_us);
    }
    delete(@start[tid]);
}'

This script selectively records and prints only synchronisation calls that exceed 5 milliseconds, immediately isolating performance outliers.

Mitigating Flush Storms and Tuning Virtual Memory Buffers

If syncsnoop reveals that frequent flushes are causing severe disk contention, investigate the Linux virtual memory dirty page ratios. By default, Linux may allow dirty memory to accumulate up to 20% of system RAM before forcing synchronous background write-backs. When sync() is called, flushing gigabytes of cached data overwhelms NVMe and SATA write queues.

Inspect your current dirty memory sysctl parameters:

sysctl vm.dirty_ratio vm.dirty_background_ratio vm.dirty_expire_centisecs

To smooth out write patterns and avoid massive periodic sync pauses on systems with large amounts of RAM (e.g. 64GB or more), adjust the dirty thresholds from percentages to fixed byte limits:

sudo sysctl -w vm.dirty_background_bytes=268435456   # 256 MB
sudo sysctl -w vm.dirty_bytes=1073741824              # 1 GB

This configuration forces the kernel to write dirty pages to storage continuously in smaller increments, dramatically reducing the maximum latency measured by syncsnoop during explicit sync operations.