How to Configure Boot Success Verification in Ubuntu Using systemd-boot-check-no-failures

Modern Linux servers and edge devices frequently depend on automated, unattended operating system updates. When applying kernel patches or major system upgrades, an automated mechanism is necessary to verify that the newly booted system reaches a healthy state before permanently committing to that boot configuration. If essential services fail during startup, the system should ideally revert to a previously known good kernel or snapshot. In systemd-based Linux systems such as Ubuntu, this automated health verification is implemented via systemd-boot-check-no-failures.service, which gates boot completion and works alongside systemd-bless-boot and modern boot loaders to automate rollback.

Role of systemd-boot-check-no-failures in Boot Assessment

The standard systemd boot assessment architecture operates through a cooperative handoff between the boot loader (such as systemd-boot or GRUB with EFI variable support) and user-space validation daemons. When a new kernel or OS version boots, the boot loader decrements an attempt counter stored in EFI NVRAM or boot loader configuration files. If the machine restarts repeatedly without the counter being cleared, the boot loader detects a boot loop and falls back to a previous kernel entry.

systemd-boot-check-no-failures.service acts as the critical user-space gatekeeper. Ordered after default.target and all critical system services, it inspects systemd’s internal transaction log. If any system service or mount point has entered a failed state, systemd-boot-check-no-failures immediately fails its own execution. This failure blocks the subsequent activation of boot-complete.target and prevents systemd-bless-boot.service from marking the current boot as “good”, allowing watchdog timers or administrative reboot handlers to initiate a rollback.

Verifying Service Availability and Unit Dependencies

On Ubuntu systems, the unit definition is installed as part of the core systemd package. To inspect the unit file and its configuration status, open a terminal and execute:

systemctl status systemd-boot-check-no-failures.service

To inspect the unit dependencies and ordering constraints, view the unit file details using systemctl cat:

systemctl cat systemd-boot-check-no-failures.service

The unit definition includes strict ordering directives:

[Unit]
Description=Check if any units failed during boot
Documentation=man:systemd-boot-check-no-failures.service(8)
DefaultDependencies=no
After=default.target
Before=boot-complete.target

[Service]
Type=oneshot
ExecStart=/lib/systemd/systemd-boot-check-no-failures
RemainAfterExit=yes

Because it is ordered After=default.target, it only runs after all standard multi-user services, networking daemons, and storage mounts have finished initialising.

Enabling and Integrating Boot Verification into boot-complete.target

By default, systemd-boot-check-no-failures.service is not enabled out of the box on standard desktop installations. To enforce strict boot success validation on servers and production appliances, link the service to boot-complete.target:

sudo systemctl enable systemd-boot-check-no-failures.service

Next, ensure that boot-complete.target is pulled in during standard startup. On systems utilising automatic boot assessment, verify that boot-complete.target is enabled or required by your primary target:

sudo systemctl status boot-complete.target

If you also utilise systemd-bless-boot (which marks the boot loader entry as good in EFI variables), ensure it is enabled alongside verification:

sudo systemctl enable systemd-bless-boot.service

Testing and Simulating Boot Service Failures

To test how the verification daemon behaves during an active failure, you can run the binary directly from the terminal with administrative privileges:

sudo /lib/systemd/systemd-boot-check-no-failures

If all system units are healthy, the command exits silently with return code 0. To verify the return code:

echo $?

Now, simulate a failed unit by creating a temporary faulty service:

sudo systemd-run --unit=test-failure.service /bin/false

Check the failure state using systemctl:

systemctl is-failed test-failure.service

Now re-run the verification binary:

sudo /lib/systemd/systemd-boot-check-no-failures

The command immediately outputs an error and exits with a non-zero code:

Unit test-failure.service failed.
Failed to check system state: Unit test-failure.service is failed.

Clean up the simulated failure by resetting unit states:

sudo systemctl reset-failed test-failure.service

Configuring Service Overrides and Excluding Non-Critical Units

In real-world environments, certain non-critical services (such as peripheral user services, optional Bluetooth daemons, or transient telemetry agents) may occasionally fail without compromising the core utility of the server. By default, systemd-boot-check-no-failures treats any failed system unit as fatal.

To prevent non-essential services from failing the boot check, create a systemd drop-in override to customise the check criteria. Launch the editor:

sudo systemctl edit systemd-boot-check-no-failures.service

You can adjust the execution timeout, add pre-execution scripts to reset known benign failures, or configure fallback logging:

[Service]
TimeoutSec=30s
ExecStartPre=/bin/sh -c 'systemctl reset-failed non-critical-agent.service 2>/dev/null || true'

Reload systemd manager configuration to apply the drop-in:

sudo systemctl daemon-reload

Integrating with Automated Watchdogs and Health Scripts

To achieve end-to-end self-healing infrastructure, pair systemd-boot-check-no-failures with a hardware watchdog or software reboot trigger. If systemd-boot-check-no-failures.service fails during boot, you can configure an OnFailure= handler in the unit drop-in:

[Unit]
OnFailure=reboot-to-fallback.service

Where reboot-to-fallback.service executes an automated fallback script or signals the BMC/IPMI controller. This ensures that an unviable kernel or corrupted software deployment cannot leave an unattended remote system unresponsive.