High-throughput database engines and write-intensive microservices depend on predictable block I/O latency. However, rogue maintenance scripts, misconfigured database checkpoints, or unthrottled backup jobs frequently execute global sync() or aggressive fsync() system calls, forcing the kernel page cache to flush dirty pages synchronously to physical disk. These operations saturate block storage controller queues, causing severe application latency spikes. The eBPF-based syncsnoop utility instruments storage flush calls in real time to reveal exactly which processes trigger disk flushes.
Understanding Linux Filesystem Sync Operations
The Linux kernel offers distinct system calls to flush cached filesystem buffers:
sync(): Commits all dirty filesystem buffers, inodes, and superblocks across every mounted filesystem to underlying physical media. It affects the entire operating system.fsync(fd): Flushes all modified data and metadata associated with a specific open file descriptor to disk.fdatasync(fd): Flushes file payload blocks to disk while omitting non-essential metadata changes (such as access timestamps), reducing seek overhead.syncfs(fd): Flushes all dirty buffers on the specific mounted filesystem that contains the designated file descriptor.
Installing the BCC Tracing Toolkit
Ensure the BPF Compiler Collection (BCC) is installed on your Linux distribution:
# Ubuntu / Debian
sudo apt update
sudo apt install -y bpfcc-tools linux-headers-$(uname -r)
# RHEL / Rocky Linux / Fedora
sudo dnf install -y bcc-tools kernel-devel-$(uname -r)
The binary is located in /usr/sbin/syncsnoop-bpfcc or /usr/share/bcc/tools/syncsnoop.
Tracing Live Disk Sync Calls with syncsnoop
Execute syncsnoop with superuser privileges to begin monitoring system-wide sync activity:
sudo syncsnoop-bpfcc
When processes execute sync operations, syncsnoop streams real-time telemetry:
TIME(s) COMM PID CALL
14:10:02.128491 syslogd 982 fsync
14:10:05.482012 postgres 14802 fdatasync
14:10:18.892104 backup_job.sh 29014 sync
14:10:22.019482 mysqld 16022 fdatasync
The output columns indicate:
- TIME(s): Fractional wall-clock timestamp of the sync call.
- COMM & PID: The executable name and process ID responsible for initiating the flush.
- CALL: The specific system call invoked (
sync,fsync,fdatasync, orsyncfs).
Diagnosing Global sync() Outliers
Global sync() invocations are especially dangerous in multi-tenant container or virtualized hosts because one container can stall I/O for all neighboring workloads:
- Run
syncsnoopand filter specifically for global flushes:sudo syncsnoop-bpfcc | grep -w "sync" - Identify offending cron jobs or diagnostic scripts invoking the
/bin/synccommand directly. - Replace broad global sync scripts with targeted
fdatasyncroutines within application code to isolate disk flushing strictly to relevant data tables.
Optimizing Kernel Dirty Page Flushes
If disk flushing causes severe queue saturation, adjust kernel page writeback parameters to spread dirty page flushes evenly over time:
# Check current dirty writeback ratios
sysctl vm.dirty_background_ratio
sysctl vm.dirty_ratio
# Lower thresholds to force smaller, continuous background writebacks
sudo sysctl -w vm.dirty_background_ratio=5
sudo sysctl -w vm.dirty_ratio=10
Re-running syncsnoop confirms whether flush bursts are successfully smoothed, preventing block device saturation.