Storage subsystem performance on Linux is fundamentally shaped by the size distribution of disk I/O requests. While Solid-State Drives (SSDs) and Non-Volatile Memory Express (NVMe) devices handle high input/output operations per second (IOPS), their real-world transfer efficiency varies dramatically between small random requests (such as 4 Kbyte blocks) and large sequential transfers (such as 128 Kbyte or 1 Mbyte extents). Standard administrative utilities like iostat display aggregated average request sizes (e.g. avgrq-sz or rareq-sz), but averages hide multimodal access patterns where an application mixes tiny metadata updates with large data buffers. Using bitesize, an extended Berkeley Packet Filter (eBPF) diagnostic utility from the BCC toolkit, systems engineers can trace real-time block device request size histograms grouped per process.
Block I/O Size Distributions and Storage Hardware Efficiency
When user-space processes perform file operations, the virtual filesystem (VFS) and block layer break down data into bios and request structures before dispatching them to the device driver queue. Hardware storage controllers, flash translation layers (FTLs), and rotational media perform substantially better when processing large sequential blocks because overhead per transaction is amortised across larger payload volumes.
Conversely, workloads dominated by small 512-byte to 4 Kbyte requests generate significant CPU interrupt and submission queue overhead. Knowing the exact histogram of I/O request sizes per process is critical for tuning application buffer pools, filesystem block sizes, database write-ahead log (WAL) block limits, and kernel readahead parameters. bitesize instruments the block layer tracepoint block_rq_issue using kernel eBPF probes, tabulating exact power-of-two size distributions directly in kernel memory before outputting clean visual histograms.
Installing the BCC Tools and bitesize on Linux
bitesize is maintained within the BPF Compiler Collection (BCC). To install the required tooling alongside matching kernel headers on modern enterprise Linux distributions:
On Ubuntu and Debian systems:
sudo apt update
sudo apt install -y bpfcc-tools linux-headers-$(uname -r)On RHEL, AlmaLinux, and Rocky Linux systems:
sudo dnf install -y bcc-tools kernel-devel-$(uname -r)On Debian-derived systems, BCC tools reside in /usr/sbin/ with a -bpfcc suffix. Verify that the tool is accessible:
which bitesize-bpfcc || which /usr/share/bcc/tools/bitesizeRunning bitesize to Capture I/O Size Histograms
To trace block I/O requests system-wide across all active processes, run bitesize-bpfcc with root privileges:
sudo bitesize-bpfccLet the tool trace for several seconds while typical application traffic is active, then terminate the capture using Ctrl + C. The tool summarizes block request size distributions across each active process:
Tracing block device I/O... Hit Ctrl-C to end.
^C
Process Name = jbd2/nvme0n1p2-8
Kbytes : count distribution
0 -> 1 : 0 | |
2 -> 3 : 0 | |
4 -> 7 : 148 |****************************************|
8 -> 15 : 12 |*** |
16 -> 31 : 4 |* |
Process Name = postgres
Kbytes : count distribution
0 -> 1 : 0 | |
2 -> 3 : 2 | |
4 -> 7 : 8 |* |
8 -> 15 : 942 |****************************************|
16 -> 31 : 18 | |
Process Name = dd
Kbytes : count distribution
64 -> 127 : 0 | |
128 -> 255 : 4120 |****************************************|Interpreting Histogram Output and Identifying Bottlenecks
The resulting output provides valuable architectural insights into process I/O characteristics:
Process Name: Identifies the executable or kernel worker issuing the block requests.Kbytes: Power-of-two size buckets (e.g. 4 to 7 Kbytes, 8 to 15 Kbytes, 128 to 255 Kbytes).count: The total number of block device requests falling into each size bucket.distribution: Visual ASCII bar graph depicting relative frequency within that process.
In the example above, the PostgreSQL process issues the vast majority of its I/O in the 8 -> 15 Kbyte range, matching PostgreSQL’s default 8 Kbyte relational block size. Meanwhile, jbd2 (the ext4 journaling daemon) writes primarily in 4 -> 7 Kbyte transactions. If a bulk analytical query or backup task displays thousands of tiny 0 -> 1 or 4 -> 7 Kbyte writes rather than large multi-kilobyte bursts, the application may be lacking proper output buffering or read-ahead prefetching.
Correlating I/O Size with Storage Latency
Small I/O requests become problematic primarily when they compete with large block transfers on saturated queues. To correlate request size patterns with actual block I/O latency, combine bitesize with its companion BCC tool biosnoop:
sudo biosnoop-bpfcc -Q | head -n 20biosnoop displays the sector count, process ID, and device queue latency in real time for every individual block request, allowing you to observe whether large 128 Kbyte requests are starving smaller 4 Kbyte metadata updates on the underlying SSD.
Optimising Block Request Sizes in Linux
If bitesize reveals sub-optimal block size distributions for sequential processing, adjust kernel block device parameters to encourage larger merged requests.
Inspect the maximum hardware request size supported by your storage controller:
cat /sys/block/nvme0n1/queue/max_hw_sectors_kbCheck the active software request limit:
cat /sys/block/nvme0n1/queue/max_sectors_kbIf max_sectors_kb is configured significantly lower than max_hw_sectors_kb (e.g. 512 instead of 1280 or 2048), increase it to allow the block layer to assemble larger contiguous requests:
echo 1024 | sudo tee /sys/block/nvme0n1/queue/max_sectors_kbAdditionally, for sequential read-heavy streaming workloads, increase the block device readahead buffer:
sudo blockdev --setra 2048 /dev/nvme0n1After adjusting these parameters, re-run bitesize-bpfcc to verify that block request distributions shift into higher Kbyte buckets, maximising device throughput and reducing CPU queue management overhead.