How to Configure Offline System Updates in Ubuntu Using systemd-system-update-generator

In enterprise Linux systems management and mission-critical server environments, applying fundamental operating system updates—such as glibc upgrades, systemd runtime updates, core kernel security patches, and UEFI firmware capsules—while the system is running live services presents considerable risks. In-place live updates can lead to shared library version mismatches, in-memory process crashes, or half-configured packages if user-space applications invoke newly replaced binary files during package unpack stages. To solve this architectural vulnerability, modern Linux distributions including Ubuntu implement the standardized systemd offline update mechanism. Orchestrated by systemd-system-update-generator and system-update.target, this framework reboots the operating system into an isolated, non-interactive execution mode to apply pending updates before automatically rebooting into the upgraded environment.

Understanding Offline Updates and systemd-system-update-generator

The systemd offline update specification defines a declarative handshake between package managers (such as apt, PackageKit, fwupd, or unattended-upgrades) and the system initialization manager:

1. Staging: While the operating system is running, the update tool downloads package archives, firmware blobs, and installation scripts into dedicated staging directories (typically under /var/lib/PackageKit/ or /var/lib/fwupd/).
2. Signaling: The tool creates a symbolic link or marker file at the root filesystem path: /system-update.
3. Reboot and Generation: The system is instructed to reboot. During early boot, before mounting local storage volumes or starting background services, systemd-system-update-generator inspects the root directory. If /system-update exists, the generator dynamically redirects the default boot target from default.target (e.g., graphical.target or multi-user.target) to system-update.target.
4. Execution: In system-update.target, minimal services launch to execute the staged update scripts in complete isolation. Network daemons, graphical desktop environments, and user sessions are prohibited from starting.
5. Completion and Cleanup: Once updates complete, the update service unlinks /system-update and immediately reboots the computer into the newly updated operating system.

Verifying System Generator Status on Ubuntu

The systemd-system-update-generator executable is a built-in system binary located within systemd’s system-generators directory. You can confirm its existence on Ubuntu using ls:

ls -l /lib/systemd/system-generators/systemd-system-update-generator

In addition, verify the definition of the target unit responsible for coordinating update execution:

systemctl cat system-update.target

The unit definition outlines that system-update.target isolates the system, ensuring that only units declaring DefaultDependencies=no and explicit dependencies on the update target execute during the update phase.

Staging Offline Firmware Updates Using fwupd

The most common native application of the offline update mechanism on modern Ubuntu machines is UEFI and hardware device firmware flashing via fwupd.

To check for pending device firmware updates across motherboards, SSD controllers, and Thunderbolt docks:

sudo fwupdmgr refresh
sudo fwupdmgr get-updates

When you trigger a firmware installation requiring an offline update:

sudo fwupdmgr update

The fwupd daemon stages the firmware capsule in the EFI System Partition (ESP), creates the /system-update symlink pointing to the firmware staging directory, and prompts for a system reboot. Upon rebooting, systemd-system-update-generator diverts boot execution directly to the firmware update agent.

Creating a Custom Offline Package Update Service

Systems administrators can leverage this exact mechanism to execute custom package installations, disk migrations, or security hardening scripts during an isolated reboot.

First, create an isolated systemd service unit designed to execute exclusively within system-update.target:

sudo nano /etc/systemd/system/custom-offline-upgrade.service

Populate the unit file with the following configuration:

[Unit]
Description=Custom Isolated Offline System Upgrade
Documentation=man:systemd.offline-updates(7)
DefaultDependencies=no
Requires=sysinit.target
After=sysinit.target
Before=system-update.target
SuccessAction=reboot
FailureAction=reboot

[Service]
Type=oneshot
ExecStart=/usr/local/bin/apply-staged-updates.sh

Next, create the update script that unpacks staged packages and cleans up the marker file:

sudo nano /usr/local/bin/apply-staged-updates.sh
sudo chmod +x /usr/local/bin/apply-staged-updates.sh

Add the following bash logic:

#!/bin/bash
set -euo pipefail

echo "Executing offline upgrade in isolated boot environment..."

# Execute package installation
if [ -d /var/cache/apt/staged-packages ]; then
    dpkg -i /var/cache/apt/staged-packages/*.deb || apt-get install -f -y
    rm -rf /var/cache/apt/staged-packages
fi

# Mandatory: Remove the /system-update symlink to allow normal boot on next cycle
rm -f /system-update

echo "Offline upgrade completed successfully. Rebooting..."

Triggering the Offline Update Cycle

To initiate the update cycle, download your deb package archives into the staging directory, enable your service for the update target, create the marker symlink, and reboot:

# Enable the service for system-update.target
sudo ln -sf /etc/systemd/system/custom-offline-upgrade.service /etc/systemd/system/system-update.target.wants/custom-offline-upgrade.service

# Create the trigger symlink pointing to the staging directory
sudo ln -sf /var/cache/apt/staged-packages /system-update

# Initiate system reboot
sudo systemctl reboot

During the reboot, systemd-system-update-generator parses /system-update, bypasses graphical desktop and multi-user services, executes custom-offline-upgrade.service, unlinks /system-update, and triggers a clean reboot into your fully updated Ubuntu system.

Auditing and Troubleshooting Offline Update Execution

If an offline update appears to fail or reboot prematurely, inspect the system journal of the previous boot cycle immediately after logging back into Ubuntu:

journalctl -b -1 -u custom-offline-upgrade.service -u system-update.target

Verify that the service ran to completion and that exit status codes confirm clean execution without unhandled script exceptions.