How to Configure Automated Boot Assessment and Rollback in Ubuntu Using systemd-bless-boot

Deploying automated kernel upgrades and system image updates on headless Ubuntu servers or remote edge appliances carries inherent risk. If a newly installed kernel triggers a hardware panic, fails to mount the root filesystem, or breaks essential network connectivity, the server can become completely unreachable, requiring costly on-site manual recovery or out-of-band console access.

To eliminate this vulnerability, the systemd ecosystem incorporates an automated boot assessment and rollback framework powered by systemd-boot and the systemd-bless-boot subsystem. By implementing boot counting, the bootloader tracks consecutive boot attempts. If a newly booted kernel reaches a verified stable state, systemd-bless-boot marks the boot as successful. If the system crashes, hangs, or reboots before completing verification, the boot counter decrements until it hits zero, triggering an automated fallback to the previously known working kernel.

Architecture of Automated Boot Assessment

The systemd boot assessment specification operates across three cooperative components:

• systemd-boot (UEFI Bootloader): Parses Boot Loader Specification (BLS) entry files located in the EFI System Partition (ESP) at /boot/efi/loader/entries/. Boot counting is enabled by appending a counter suffix to the entry filename (for example, ubuntu-kernel-6.8.0-31-generic+3-0.conf, indicating 3 remaining attempts and 0 failed attempts).
• Kernel Execution and User-Space Handoff: When the bootloader executes an entry with boot counting enabled, it decrements the remaining attempt counter before transferring execution control to the Linux kernel.
• systemd-bless-boot.service: A systemd background service that runs late in the boot process (after reaching boot-complete.target). Once critical system services are operational, the service executes systemd-bless-boot good, renaming the BLS configuration file to remove the counter or mark it permanently valid.

If a kernel panic or hardware freeze interrupts the boot sequence, the system reboots. Upon the third failed attempt, systemd-boot automatically skips the broken entry and launches the older fallback kernel entry.

Auditing Bootloader Status with bootctl

Before configuring boot counting, confirm that your Ubuntu installation uses systemd-boot or supports the Boot Loader Interface by querying bootctl:

# Query the current bootloader status and entry list
bootctl status

The output reveals whether the UEFI firmware and bootloader report boot loader variables:

System:
      Firmware: UEFI 2.80
   Secure Boot: enabled
  Boot Loader: systemd-boot 255.4-1ubuntu8
Features:
  Boot Loader Specification: yes
  Automatic Boot Assessment: supported

To inspect currently available boot loader entries:

# List all configured boot entries
bootctl list

Inspecting systemd-bless-boot Service State

To verify the active operational state of the boot blessing utility, execute systemd-bless-boot status:

# Inspect the boot blessing status
systemd-bless-boot status

The utility inspects the active boot entry and reports its validation status:

Current Boot:
     Entry: ubuntu-kernel-6.8.0-31-generic.conf
    Status: good (marked permanently clean)

If the system is currently evaluating a provisional boot with remaining attempts, the status reports indeterminate along with the count of remaining tries:

Current Boot:
     Entry: ubuntu-kernel-6.8.0-35-generic+3-0.conf
    Status: indeterminate (3 tries left)

Enabling Automated Boot Blessing on Boot Completion

Ubuntu systems managing automated boot assessment utilize systemd-bless-boot.service, which triggers automatically when the multi-user target successfully completes:

# Verify and enable the automatic boot blessing service
sudo systemctl status systemd-bless-boot.service
sudo systemctl enable systemd-bless-boot.service

To ensure that the boot is blessed only after critical custom services (such as Docker, custom database engines, or enterprise VPNs) are fully operational, create a systemd drop-in override for boot-complete.target:

# Create drop-in directory for boot completion target
sudo mkdir -p /etc/systemd/system/boot-complete.target.d/

# Ensure network and database services are up before blessing
sudo tee /etc/systemd/system/boot-complete.target.d/override.conf << 'EOF'
[Unit]
Wants=nginx.service postgresql.service
After=nginx.service postgresql.service
EOF

Manually Marking Boots as Good or Bad

During manual kernel maintenance or when scripting custom automated validation tests, you can explicitly bless or reject the running boot:

# Manually bless the running boot entry as permanently successful
sudo systemd-bless-boot good

# Manually mark the current boot as bad to trigger fallback on next reboot
sudo systemd-bless-boot bad

Marking the boot as bad immediately sets the remaining attempt counter to 0 within the BLS configuration entry. When the server reboots, systemd-boot immediately bypasses the faulty kernel and loads the fallback configuration without requiring operator intervention.

Configuring automated boot assessment via systemd-bless-boot ensures maximum availability for mission-critical Ubuntu servers, enabling resilient, self-healing updates across enterprise infrastructure.