How to Monitor Device Mapper I/O Latency and Performance in Linux Using dmstats

Modern Linux enterprise storage relies extensively on the Device Mapper framework to manage Logical Volume Manager (LVM2) partitions, dm-crypt encrypted disks (LUKS), software RAID arrays, multipath devices, and thin-provisioned pools. While traditional system performance tools like iostat or vmstat report aggregated input/output (I/O) statistics across entire physical disks, they cannot provide fine-grained latency and throughput metrics for individual logical volumes or sub-regions within a virtual disk. The dmstats command-line utility, maintained as part of the core Device Mapper toolset, provides storage engineers and system administrators with direct kernel-level instrumentation to measure I/O latency, queue depths, and read/write histograms across device mapper targets.

Understanding Device Mapper Statistics (dmstats)

The Linux kernel device mapper driver includes built-in statistics gathering capabilities managed via dmstats:

• Granular Regions: Instead of reporting single average figures for a multi-terabyte drive, dmstats allows you to divide a device mapper target into arbitrary step-sized monitoring regions (e.g. tracking latency per gigabyte or per logical volume segment).
• Latency Histograms: Captures detailed response-time histograms, revealing microsecond-level latency spikes and I/O bottlenecks that disappear in macroscopic averages.
• Zero-Overhead Dormancy: By default, kernel statistics collection is inactive, consuming zero CPU cycles until a monitoring region is explicitly created via dmstats create.
• Target Agnostic: Operates seamlessly across standard LVM logical volumes, dm-crypt encrypted devices, multipath storage, and software RAID targets.

Discovering Active Device Mapper Targets

Before creating statistics regions, list the active device mapper targets on your system using dmsetup ls or lvs:

sudo dmsetup ls

The command outputs registered device names and major/minor numbers (e.g. ubuntu--vg-ubuntu--lv, luks-data, or vg0-lv_database).

You can also inspect existing dmstats regions across all mapped devices:

sudo dmstats list

If no regions have been configured, the output will return empty headers.

Creating a Monitoring Region with dmstats create

To begin collecting I/O metrics on a specific logical volume, create a monitoring region. To monitor an entire device mapper volume as a single aggregated unit:

sudo dmstats create ubuntu--vg-ubuntu--lv

To divide a volume into smaller monitoring segments (for instance, creating 1 GB step regions across a fast database volume to identify which sector ranges experience heavy write activity):

sudo dmstats create --step 1G ubuntu--vg-ubuntu--lv

Verify that the region was created and note its assigned Region ID (typically 0 for the first region):

sudo dmstats list ubuntu--vg-ubuntu--lv

Configuring Latency Histograms for Microsecond Profiling

Average latency metrics can mask severe performance hitches caused by garbage collection or block allocation delays. You can configure precise latency histogram boundaries when creating a region using the --bounds option.

For example, to track read and write latencies categorized into buckets of 1ms, 5ms, 10ms, 25ms, 50ms, and 100ms:

sudo dmstats create --bounds 1ms,5ms,10ms,25ms,50ms,100ms ubuntu--vg-ubuntu--lv

Displaying Real-Time Metrics with dmstats print and report

To view the captured metrics, use dmstats print or dmstats report:

sudo dmstats report ubuntu--vg-ubuntu--lv

The report displays standard I/O metrics tailored to your storage regions:

• RgID: Region Identifier.
• Read/Write Sectors: Total sectors read and written within the region.
• r/s and w/s: Read and write requests processed per second.
• r_await and w_await: Average time (in milliseconds) read and write requests spent waiting in queue and being serviced.
• %util: Percentage of CPU and I/O time during which requests were in flight.

To display continuous live updates (similar to iostat 2), use the --interval option:

sudo dmstats report --interval 2 ubuntu--vg-ubuntu--lv

Clearing and Deleting Monitoring Regions

Once performance profiling or troubleshooting is complete, clear or delete regions to prevent unnecessary kernel tracking:

To reset statistics counters back to zero without deleting the monitoring region:

sudo dmstats clear --regionid 0 ubuntu--vg-ubuntu--lv

To delete a specific monitoring region:

sudo dmstats delete --regionid 0 ubuntu--vg-ubuntu--lv

To delete all active regions across every mapped storage device:

sudo dmstats delete --allregions --allprograms

Verify that all counters are cleared:

sudo dmstats list