In high-performance computing and enterprise database architectures (like SAP HANA or Redis Enterprise), the ultimate bottleneck is storage latency. Traditional NVMe solid-state drives sit on the PCIe bus. While fast, data must travel from the CPU, across the PCIe controller, into the SSD controller, and into NAND flash. This traversal incurs latency measured in microseconds.
Non-Volatile Dual In-line Memory Modules (NVDIMMs) collapse this architecture. An NVDIMM physically plugs directly into the motherboard’s DDR4/DDR5 RAM slots. It connects directly to the CPU’s memory controller, achieving latency measured in nanoseconds. Crucially, unlike standard volatile RAM, NVDIMMs contain onboard NAND flash and a supercapacitor. If the server loses power, the supercapacitor instantly flushes the data from the volatile DRAM into the non-volatile NAND. When power is restored, the data is pushed back into RAM.
To leverage NVDIMMs, you cannot just treat them like a standard hard drive. You must configure the Linux kernel to address them as Persistent Memory (PMEM) using Direct Access (DAX).
This guide explains how to initialize and mount NVDIMM persistent memory in Linux.
Understanding the PMEM Architecture
The Linux kernel interacts with NVDIMMs using the libnvdimm subsystem. There are two primary ways to expose this memory to applications:
- Sector Mode (fsdax): The NVDIMM is formatted with a filesystem (like XFS or ext4). The crucial difference is that the filesystem is mounted with the
-o dax(Direct Access) flag. This tells the kernel to completely bypass the page cache. Applications read and write directly to the physical silicon on the DIMM slot via memory-mapped files (mmap). - Devdax Mode (devdax): The memory is exposed as a raw character device (e.g.,
/dev/dax0.0). Databases (like Oracle or proprietary in-memory trading systems) bypass the filesystem entirely and map the physical memory directly into their application address space.
Step 1: Installing the ndctl Utility
The primary tool for managing NVDIMM namespaces is ndctl.
sudo apt-get update
sudo apt-get install ndctl
Verify that the Linux kernel has detected the physical NVDIMM hardware populated in the motherboard slots:
ndctl list --regions
The output will list the physical regions of persistent memory detected by the ACPI NFIT (NVDIMM Firmware Interface Table). You should see a region (e.g., region0) with a size corresponding to your NVDIMM hardware.
Step 2: Creating a PMEM Namespace (fsdax)
By default, raw persistent memory must be carved into logical namespaces before it can be used, similar to partitioning a traditional drive.
We will create a namespace configured for File System Direct Access (fsdax):
sudo ndctl create-namespace --mode=fsdax --region=region0
The system will output a JSON block confirming the creation of a block device, typically named /dev/pmem0.
Step 3: Formatting the NVDIMM for DAX
Because NVDIMMs are wildly faster than standard SSDs, you must use a modern filesystem that explicitly supports the DAX extensions. Both XFS and ext4 support DAX.
Format the block device with XFS:
sudo mkfs.xfs -f -d su=2m,sw=1 /dev/pmem0
Create a mount point:
sudo mkdir -p /mnt/nvdimm_db
Step 4: Mounting with the DAX Flag
This is the most critical step. If you mount the filesystem normally, the Linux kernel will cache the data in standard DDR RAM before writing it to the NVDIMM, introducing massive software latency and completely defeating the purpose of the hardware.
You must explicitly mount the device using the dax option:
sudo mount -o dax /dev/pmem0 /mnt/nvdimm_db
Verify that DAX is actively enabled on the mount point:
mount | grep dax
You should see /dev/pmem0 on /mnt/nvdimm_db type xfs (rw,relatime,attr2,dax,inode64,noquota).
Step 5: Testing Persistent Memory Performance
To verify the extreme low-latency performance of the DAX mount, you can use the fio (Flexible I/O Tester) benchmarking utility.
sudo fio --name=dax_test --filename=/mnt/nvdimm_db/testfile \
--ioengine=mmap --rw=randwrite --bs=4k --numjobs=1 --size=1G \
--direct=1 --sync=1
Because the ioengine is set to mmap (memory-mapped) and the filesystem is mounted with DAX, fio will write data directly into the CPU’s L1/L2 caches, which are instantly flushed directly onto the physical NVDIMM silicon sitting on the DDR bus. The latency will be drastically lower than an equivalent test run on an NVMe PCIe drive.
Conclusion
NVDIMM Persistent Memory fundamentally blurs the line between volatile RAM and non-volatile storage. By leveraging the libnvdimm kernel subsystem and DAX-enabled filesystems, systems architects can provide high-frequency trading platforms and massive in-memory databases with storage tiers operating at the blistering speed of the DDR memory bus, all while guaranteeing perfect data persistence across power failures.