How to Implement Linux OverlayFS for Immutable Root Filesystems

In high-security environments, IoT edge devices, or kiosk systems, maintaining an absolutely pristine operating system is critical. If malware infects the system, or an administrator accidentally deletes a critical configuration file, the system must be capable of recovering to a known-good state instantly upon reboot.

The solution is an Immutable Root Filesystem. Historically, this was achieved by mounting the entire / filesystem as strictly Read-Only. However, a purely read-only filesystem breaks thousands of Linux applications that expect to be able to write temporary files, update /var/log, or modify their own configurations.

OverlayFS solves this paradox. It allows you to mathematically stack a writable temporary filesystem (held in RAM) on top of the read-only root filesystem. The applications believe they are writing to the disk, but they are actually writing to the temporary RAM overlay. When the system reboots, the RAM is cleared, the overlay is destroyed, and the pristine, untouched read-only root filesystem remains.

This guide explains the architecture of OverlayFS and how to configure a robust immutable root implementation.

Understanding OverlayFS Architecture

OverlayFS operates by defining specific directories (called layers):

1. The Lower Directory (lowerdir): This is your actual, read-only root filesystem (e.g., the block device /dev/sda1). OverlayFS will never write data here.
2. The Upper Directory (upperdir): This is a writable filesystem. For an immutable system, this is usually a tmpfs (a filesystem stored entirely in volatile RAM).
3. The Work Directory (workdir): An empty directory used internally by OverlayFS to prepare files before they are shifted to the upper directory. It must reside on the same filesystem as the upperdir.
4. The Merged Directory (merged): This is the mount point presented to the operating system. When a user runs ls /, they are looking at the Merged directory.

When an application reads a file, OverlayFS looks in the Upper directory first. If it’s not there, it seamlessly passes the read down to the Lower directory. When an application writes or deletes a file, the change is recorded exclusively in the Upper directory, masking the underlying Lower file.

Step 1: The Initramfs Challenge

You cannot simply mount OverlayFS over the root filesystem (/) while the system is actively running. By the time the standard OS boot process reaches your fstab file, the kernel has already mounted the real root filesystem and spawned systemd.

To implement an immutable root, you must intercept the boot process before the real root is mounted. This requires modifying the initramfs (the tiny, temporary RAM filesystem the kernel loads immediately after the bootloader).

Step 2: Installing the Overlayroot Package

Manually rewriting initramfs scripts to mount tmpfs, construct the OverlayFS layers, and pivot the root via switch_root is incredibly complex and prone to kernel panics.

Fortunately, Canonical provides a battle-tested package called overlayroot (available on Ubuntu and Debian-based systems) that injects all the necessary shell scripts directly into the initramfs.

Install the package:

sudo apt-get update
sudo apt-get install overlayroot

Step 3: Configuring Overlayroot

The configuration for overlayroot is managed in a single configuration file.

sudo nano /etc/overlayroot.conf

By default, the package is disabled. To force the root filesystem into an immutable, RAM-backed overlay, modify the following line:

overlayroot="tmpfs"

This simple configuration tells the initramfs scripts to:

1. Mount the real root filesystem (/dev/sda1) as Read-Only.
2. Mount a tmpfs in RAM.
3. Use OverlayFS to merge them.
4. Boot the system using the merged filesystem as /.

Save the file.

Step 4: Rebooting into Immutability

Restart the server:

sudo reboot

When the system comes back online, log in and verify the mounts:

mount | grep overlayroot

You will see that / is mounted as type overlay, with the upperdir pointing to a tmpfs location.

Testing the Immutability:

1. Create a file on the root filesystem: sudo touch /THIS_FILE_WILL_DISAPPEAR.
2. Verify it exists: ls /.
3. Reboot the system: sudo reboot.
4. Log back in and check: ls /. The file is completely gone. The system has restored itself to its pristine state.

Step 5: Modifying the Immutable System (Disabling the Overlay)

If the system resets on every reboot, how does an administrator install security patches or update configurations?

You have two options:

Option 1 (The Chroot Method): The overlayroot package mounts the actual, underlying read-only block device to a hidden directory, usually /media/root-ro. You can mount it as read-write, chroot into it, perform your updates, and reboot.

sudo mount -o remount,rw /media/root-ro
sudo chroot /media/root-ro
# Run apt-get update, etc.
exit
sudo reboot

Option 2 (The Boot Parameter Method): If you have physical or out-of-band console access to the GRUB bootloader, you can temporarily disable the overlay for a single boot sequence.

When the GRUB menu appears, press e to edit the boot parameters. Add the following to the end of the Linux kernel line:

overlayroot=disabled

Press F10 to boot. The system will boot normally, mounting the physical disk as read-write. You can perform maintenance, and the next standard reboot will automatically re-engage the immutable overlay.

Conclusion

Implementing an immutable root filesystem via OverlayFS fundamentally alters the security posture of a Linux endpoint. By decoupling the runtime environment from the physical storage, administrators can guarantee absolute consistency, immune to drift, configuration errors, and persistent malware, ensuring the system boots in a mathematically verifiable state every single time.