Architectural Foundation of SAN Multipathing and SCSI ALUA
Enterprise storage architectures connected via Fibre Channel (FC), iSCSI, or NVMe over Fabrics (NVMe-oF) deploy redundant physical paths—spanning dual Host Bus Adapters (HBAs), redundant SAN fabric switches, and dual active storage controllers. When an Ubuntu Server attaches to a Storage Area Network (SAN) Logical Unit Number (LUN) across redundant fabrics, the Linux SCSI mid-layer enumerates each physical route as a discrete block device (for example, detecting /dev/sdb, /dev/sdc, /dev/sdd, and /dev/sde for a single physical storage volume). Accessing these raw paths directly leads to severe filesystem corruption.
To consolidate redundant paths into a unified, high-availability block device, Linux implements Device-Mapper Multipathing (DM-Multipath) managed by the multipathd daemon. DM-Multipath abstracts individual SCSI paths into a resilient virtual block device (e.g., /dev/mapper/mpatha or WWID alias). Furthermore, modern enterprise SAN arrays (such as Dell PowerStore, NetApp ONTAP, HPE Alletra, and Pure Storage) operate using Asymmetric Logical Unit Access (ALUA, SPC-3 standard). ALUA designates specific controller ports as Active/Optimised (direct low-latency bus access to disk cache) while marking alternate ports as Active/Non-Optimised (traversing inter-controller backplanes). DM-Multipath actively prioritises Active/Optimised paths, failing over to non-optimised routes instantaneously should a primary fabric link sever.
| Multipath Path State | ALUA Hardware Classification | Kernel I/O Handling | Routing Precedence |
|---|---|---|---|
| Active / Optimised | Direct access via owning controller | Dispatches 100% of read/write SCSI commands | Highest priority (Priority score: 50) |
| Active / Non-Optimised | Access via peer controller backplane | Standby; accepts I/O only if optimised paths fail | Secondary priority (Priority score: 10) |
| Standby / Transitioning | Passive or target port swapping state | Rejects commands with CHECK CONDITION | Offline standby |
| Unavailable (Dead) | Physical link severed (cable/HBA down) | Marked failed; excluded from path selection | Zero I/O dispatch |
Installing and Initialising DM-Multipath on Ubuntu Server
The multipathing daemon and kernel device-mapper modules are packaged within multipath-tools:
# Update APT repository and install multipath toolchain
sudo apt update && sudo apt install -y multipath-tools
# Load the device-mapper multipath kernel module
sudo modprobe dm-multipath
# Ensure the module loads automatically on system boot
echo "dm-multipath" | sudo tee -a /etc/modules
Constructing an Enterprise /etc/multipath.conf Configuration
The master configuration file resides at /etc/multipath.conf. A well-tuned configuration must balance path grouping, define failure timeouts, and enable ALUA hardware priority checkers.
sudo tee /etc/multipath.conf << 'EOF'
defaults {
# Generate persistent user-friendly names (e.g., /dev/mapper/mpatha)
user_friendly_names yes
# Enable automatic device-mapper queue finding
find_multipaths yes
# Path selection algorithm across active path groups
path_selector "round-robin 0"
# Grouping policy based on SCSI ALUA priority values
path_grouping_policy group_by_prio
# Hardware priority checker querying ALUA target port groups
prio alua
# Background path health checker
path_checker tur
# Healthcheck polling interval in seconds
polling_interval 5
# Retry I/O when all paths fail before reporting error (prevents FS read-only crash)
no_path_retry 12
# Flush on last del
flush_on_last_del yes
}
# Blacklist local operating system drives (e.g., internal NVMe boot drives)
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^(hd|xvd|vd)[a-z]"
devnode "^nvme.*"
}
# Explicit LUN aliases bound to World Wide Identifiers (WWID)
multipaths {
multipath {
wwid "36001405b8a97c41e88e404b9d0e21a4f"
alias database_vol01
}
}
EOF
Applying Configuration and Updating Initial RAM Filesystem
Because multipath devices must be initialized during early boot (especially if root or database partitions reside on SAN storage), recompile the kernel initramfs image:
# Restart and enable the multipath daemon
sudo systemctl restart multipathd
sudo systemctl enable multipathd
# Update the Linux initial RAM filesystem (initramfs)
sudo update-initramfs -u -k all
Auditing Active Multipath Topologies and Failover Health
To inspect aggregated LUN topologies, active path groups, and ALUA priority rankings, use multipath -ll:
# Display comprehensive multipath topology
sudo multipath -ll
Example healthy ALUA topology output:
database_vol01 (36001405b8a97c41e88e404b9d0e21a4f) dm-0 DELL,PowerStore
size=2.0T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| |- 0:0:1:1 sdb 8:16 active ready running
| `- 1:0:1:1 sdd 8:48 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
|- 0:0:2:1 sdc 8:32 active ready running
`- 1:0:2:1 sde 8:64 active ready running
Interpretation:
prio=50 status=active: Pathssdbandsddrepresent Active/Optimised links directly into the primary storage controller. All I/O currently load-balances across these two physical cables.prio=10 status=enabled: Pathssdcandsderepresent Active/Non-Optimised paths via the peer controller. They remain primed in standby, ready to assume I/O within milliseconds if both primary paths fail.
Interactive Management via multipathd CLI
The multipathd daemon features an interactive control shell for live operations without restarting daemons:
# Query path states and error rates
sudo multipathd show paths
# Show summary status of all multipath device maps
sudo multipathd show maps status
# Force an immediate path re-check across all adapters
sudo multipathd reconfigure
Troubleshooting Common SAN Multipathing Faults
| Observed Fault | Underlying Technical Cause | Diagnostic Verification Method | Remediation Protocol |
|---|---|---|---|
| Infinite I/O Hang When Primary SAN Controller Fails | features "1 queue_if_no_path" enabled without defining a finite no_path_retry counter |
Processes enter uninterruptible D state; I/O blocks indefinitely waiting for path recovery |
Set no_path_retry 12 (retries for 60 seconds before failing gracefully) in defaults |
| Root / Boot Disk Inadvertently Claimed by Multipath | Blacklist section omitted or internal drive WWID was not excluded from /etc/multipath.conf |
Boot disk appears as /dev/mapper/mpatha; GRUB loader errors on reboot |
Add internal SATA/NVMe drive controllers to the blacklist stanza in /etc/multipath.conf |
| Path Priority Inversion (All Paths Assigned Equal Priority 1) | SCSI ALUA hardware handler (hwhandler="1 alua") missing, or storage array ALUA mode disabled |
multipath -ll shows all paths grouped under single tier with prio=1 |
Set prio alua in multipath.conf; verify host mode on storage array is configured for Linux / ALUA |
| Path Flapping / Rapid Failover Churn | Intermittent optical fibre link noise causing path_checker tur timeouts |
Kernel log (dmesg) floods with: checker failed path sdb followed by path restored |
Clean optical SFP transceivers; replace patch cables; adjust switch port error thresholds |