How to Extend System Binaries and Libraries in Ubuntu Using systemd-sysext

Modern Linux deployments and immutable containerised host distributions frequently lock down the root filesystem or mount /usr and /opt as read-only partitions to guarantee operating system integrity and prevent configuration drift. While this architecture enhances security and system reliability, it complicates scenarios where system administrators and developers need to deploy debugging utilities, temporary binary toolchains, or vendor-specific drivers without modifying the base disk image. In Ubuntu, systemd-sysext provides a native, robust mechanism to dynamically overlay system extension images onto /usr and /opt at runtime using OverlayFS.

Rather than relying on invasive package installations or complex container runtimes, systemd-sysext inspects standalone disk images, loop devices, or raw directory trees, matches their metadata against the host operating system release, and combines their contents into a unified hierarchy. When debugging or servicing is complete, the extension layer can be merged, refreshed, or unmerged instantly without leaving lingering files on the underlying filesystem.

Understanding How systemd-sysext Architecture Works

The systemd-sysext utility operates by mounting an OverlayFS instance where the host’s existing /usr and /opt directories act as the lower read-only layers, while the extension images provide additional lower directories stacked on top. Because OverlayFS synthesises these directory layers in the virtual filesystem layer, running processes observe a unified directory structure containing all original system binaries alongside the newly introduced tools.

To preserve binary compatibility and prevent library mismatch issues, each extension image must declare an extension release metadata file matching the host’s OS identity. This metadata is stored inside the image at /usr/lib/extension-release.d/extension-release.IMAGE_NAME and must match the ID, VERSION_ID, or SYSEXT_LEVEL attributes found in the host system’s /etc/os-release file. If the OS version does not match, systemd-sysext rejects the image to avoid corrupting runtime execution environments.

Inspecting Current Extension Status and Hierarchy

Before preparing or activating extensions, inspect the current runtime configuration and discover whether any extensions are currently active on your Ubuntu host. Open a terminal and run the status query command:

systemd-sysext status

If no extensions are currently mounted, the command indicates that no extensions are merged. You can also view the search paths that systemd-sysext scans for image files and directories by executing:

systemd-sysext list

By default, systemd-sysext scans the following predefined filesystem paths in hierarchical order:

• /etc/extensions/ — Host-specific extensions intended for local administrator overrides.
• /run/extensions/ — Ephemeral extensions created during runtime or network provisioning.
• /var/lib/extensions/ — Persistent system extensions stored across system reboots.
• /usr/lib/extensions/ — Operating system vendor extensions integrated with factory provisioning.

Building a Valid System Extension Directory Structure

A system extension image can be supplied as a raw directory hierarchy, a SquashFS image (.raw), an ext4 disk image, or an encrypted block image. For local testing, building a directory hierarchy inside /var/lib/extensions/ provides the most straightforward approach.

First, verify the host release values that the extension must declare. Inspect your host release identifiers:

grep -E '^(ID|VERSION_ID)=' /etc/os-release

Create a dedicated extension directory inside /var/lib/extensions/ for a custom utility bundle named dev-tools:

sudo mkdir -p /var/lib/extensions/dev-tools.raw.d/usr/bin
sudo mkdir -p /var/lib/extensions/dev-tools.raw.d/usr/lib/extension-release.d

Next, generate the required extension release metadata file. The file name must exactly match the base name of the extension folder or image:

sudo bash -c 'cat <<EOF > /var/lib/extensions/dev-tools.raw.d/usr/lib/extension-release.d/extension-release.dev-tools
ID=ubuntu
VERSION_ID=24.04
EOF'

To verify the dynamic binary extension mechanism, place an executable script or binary within the extension’s binary path:

sudo bash -c 'cat <<EOF > /var/lib/extensions/dev-tools.raw.d/usr/bin/audit-helper
#!/bin/bash
echo "Dynamic extension tool executed successfully from /usr/bin on Ubuntu."
EOF'
sudo chmod +x /var/lib/extensions/dev-tools.raw.d/usr/bin/audit-helper

Merging Extensions into the Live Filesystem

Once your extension hierarchy and metadata file are populated, instruct systemd-sysext to inspect available extensions and merge them into the live system hierarchy:

sudo systemd-sysext merge

The command mounts an OverlayFS instance over /usr and /opt. You can verify that the extension is active by running:

systemd-sysext status

The output will display the active extension name, its source directory, and the target mount points. Confirm that your newly merged binary is immediately accessible through the standard system execution path without modifying the underlying root partition:

which audit-helper
audit-helper

Refreshing and Unmerging Extensions Safely

When changes are made to extension files or new images are dropped into /var/lib/extensions/, you can reload the overlay hierarchy dynamically without disrupting unaffected processes:

sudo systemd-sysext refresh

The refresh command unmerges existing overlays, re-evaluates the configured directories, and re-merges the updated layers atomically.

To completely detach all extensions and restore /usr and /opt to their pure underlying filesystem state, execute the unmerge command:

sudo systemd-sysext unmerge

Run a verification check to ensure that the overlay has been dismantled cleanly:

which audit-helper

The command will report that the binary is no longer present in /usr/bin, confirming that the base system remains pristine and untouched.

Automating Extension Activation at Boot

If your system requires extensions to be available automatically during host boot, enable the native systemd service unit designed for extension management:

sudo systemctl enable systemd-sysext.service

During the system boot sequence, systemd executes systemd-sysext merge before standard target services start, ensuring that all dependent services find necessary binaries and shared libraries in /usr and /opt without manual administrator intervention.