How to Trace System Sync Calls and Storage Flushes in Linux Using syncsnoop

High-throughput database engines and write-intensive microservices depend on predictable block I/O latency. However, rogue maintenance scripts, misconfigured database checkpoints, or unthrottled backup jobs frequently execute global sync() or aggressive fsync() system calls, forcing the kernel page cache to flush dirty pages synchronously to physical disk. These operations saturate block storage controller queues, causing severe application latency spikes. The eBPF-based syncsnoop utility instruments storage flush calls in real time to reveal exactly which processes trigger disk flushes.

Understanding Linux Filesystem Sync Operations

The Linux kernel offers distinct system calls to flush cached filesystem buffers:

  • sync(): Commits all dirty filesystem buffers, inodes, and superblocks across every mounted filesystem to underlying physical media. It affects the entire operating system.
  • fsync(fd): Flushes all modified data and metadata associated with a specific open file descriptor to disk.
  • fdatasync(fd): Flushes file payload blocks to disk while omitting non-essential metadata changes (such as access timestamps), reducing seek overhead.
  • syncfs(fd): Flushes all dirty buffers on the specific mounted filesystem that contains the designated file descriptor.

Installing the BCC Tracing Toolkit

Ensure the BPF Compiler Collection (BCC) is installed on your Linux distribution:

# Ubuntu / Debian
sudo apt update
sudo apt install -y bpfcc-tools linux-headers-$(uname -r)

# RHEL / Rocky Linux / Fedora
sudo dnf install -y bcc-tools kernel-devel-$(uname -r)

The binary is located in /usr/sbin/syncsnoop-bpfcc or /usr/share/bcc/tools/syncsnoop.

Tracing Live Disk Sync Calls with syncsnoop

Execute syncsnoop with superuser privileges to begin monitoring system-wide sync activity:

sudo syncsnoop-bpfcc

When processes execute sync operations, syncsnoop streams real-time telemetry:

TIME(s)            COMM             PID    CALL
14:10:02.128491    syslogd          982    fsync
14:10:05.482012    postgres         14802  fdatasync
14:10:18.892104    backup_job.sh    29014  sync
14:10:22.019482    mysqld           16022  fdatasync

The output columns indicate:

  • TIME(s): Fractional wall-clock timestamp of the sync call.
  • COMM & PID: The executable name and process ID responsible for initiating the flush.
  • CALL: The specific system call invoked (sync, fsync, fdatasync, or syncfs).

Diagnosing Global sync() Outliers

Global sync() invocations are especially dangerous in multi-tenant container or virtualized hosts because one container can stall I/O for all neighboring workloads:

  1. Run syncsnoop and filter specifically for global flushes:
    sudo syncsnoop-bpfcc | grep -w "sync"
  2. Identify offending cron jobs or diagnostic scripts invoking the /bin/sync command directly.
  3. Replace broad global sync scripts with targeted fdatasync routines within application code to isolate disk flushing strictly to relevant data tables.

Optimizing Kernel Dirty Page Flushes

If disk flushing causes severe queue saturation, adjust kernel page writeback parameters to spread dirty page flushes evenly over time:

# Check current dirty writeback ratios
sysctl vm.dirty_background_ratio
sysctl vm.dirty_ratio

# Lower thresholds to force smaller, continuous background writebacks
sudo sysctl -w vm.dirty_background_ratio=5
sudo sysctl -w vm.dirty_ratio=10

Re-running syncsnoop confirms whether flush bursts are successfully smoothed, preventing block device saturation.

Get the best tech tips delivered straight to your inbox.

Join thousands of readers mastering Apple, Google, Microsoft, and Linux.