How to Enable Early Boot Debug Shells in Ubuntu Using systemd-debug-generator

Diagnosing critical operating system boot failures in Ubuntu—such as unmountable root filesystems, stalled systemd ordering cycles, broken device mappers, or hanging network generators—frequently presents a frustrating challenge. When a server fails before launching the graphical display manager, virtual terminal logins (getty), or the OpenSSH daemon, standard remote management and local interactive login prompts remain inaccessible. To provide systems administrators with immediate emergency access during the boot process, systemd includes systemd-debug-generator. This early-stage generator parses kernel command-line arguments passed from GRUB during boot and automatically instantiates a privileged root debug shell bound to Virtual Terminal 9 (TTY9). By enabling early debug shells, Linux administrators can inspect active unit transactions, review journal streams, and resolve deadlocks in real time.

Understanding systemd-debug-generator and the Debug Shell

During the earliest moments of system initialization—prior to loading user-defined targets and services—the systemd manager executes generator binaries located in /lib/systemd/system-generators/. These generators inspect hardware state, configuration tables, and kernel parameters, dynamically creating dependency symlinks and transient service units in /run/systemd/generator/.

The systemd-debug-generator is specifically responsible for inspecting the kernel command line (read from /proc/cmdline) for debugging directives. When it detects arguments such as systemd.debug_shell, it generates an early dependency symlink that pulls debug-shell.service into the initial boot transaction. Unlike standard emergency or rescue targets that halt the boot sequence, debug-shell.service runs concurrently in the background as an unauthenticated Bash shell operating with full root privileges on /dev/tty9. The primary operating system continues booting normally, allowing administrators to switch to TTY9 at any point to observe, diagnose, or manipulate services as they initialize.

Enabling the Early Debug Shell Temporarily via GRUB

For immediate incident triage on an unresponsive or hanging Ubuntu server, enabling the debug shell temporarily via the GRUB boot menu avoids permanent configuration modifications.

1. Reboot the Ubuntu machine.
2. At the GRUB bootloader menu, highlight the desired Ubuntu kernel entry and press e to enter edit mode.
3. Locate the line starting with linux (which specifies the kernel image and boot arguments such as quiet splash).
4. Move your cursor to the end of the line, append a space, and add the following kernel argument:

   systemd.debug_shell

5. Press Ctrl + X or F10 to boot the system with the modified command line.

As the kernel hands control over to systemd, systemd-debug-generator parses systemd.debug_shell and launches debug-shell.service on TTY9.

Switching to the Root Debug Shell on TTY9

Once the boot sequence commences, you can switch immediately to Virtual Terminal 9 by pressing the keyboard shortcut:

Ctrl + Alt + F9

You will be presented immediately with a root shell prompt (root@ubuntu:~#) without being prompted for a username or password. This unauthenticated access is deliberate, designed to ensure administrators can recover systems even when PAM, authentication databases, or user accounts are damaged or unmounted.

Investigating Boot Hangs and Unit Deadlocks

From the TTY9 debug shell, execute standard systemd inspection commands while the rest of the operating system attempts to boot:

Inspecting Active Boot Jobs

To identify which services are currently running, queued, or blocking the boot pipeline, query active systemd jobs:

systemctl list-jobs

If a service is stuck waiting for a hardware device or timeout (such as waiting for an unavailable network mount or cryptographic key), list-jobs highlights the blocking unit with a state such as running or waiting.

Inspecting Real-Time System Journal Logs

To monitor boot log messages as they are recorded by systemd-journald:

journalctl -f -b

To isolate critical errors and warning messages generated during the current boot cycle:

journalctl -b -p 3 -xb

Cancelling Blocking Jobs to Resume Booting

If a non-critical unit (such as a remote NFS automount or custom backup script) is deadlocking the boot transaction, cancel the specific job or stop the service directly from TTY9:

systemctl cancel <job_id>
systemctl stop problematic-service.service

Once cancelled, systemd immediately resumes processing dependent targets, progressing through multi-user.target or graphical.target.

Configuring Permanent Debug Shell for Staging Environments

In lab testbeds, continuous deployment runners, or kernel development environments where repeated reboots require ongoing diagnosis, you can enable debug-shell.service persistently.

Enable the service unit using systemctl:

sudo systemctl enable debug-shell.service

Alternatively, to ensure the kernel command line persistently instructs systemd-debug-generator on every boot, edit /etc/default/grub:

sudo nano /etc/default/grub

Append systemd.debug_shell to GRUB_CMDLINE_LINUX_DEFAULT:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash systemd.debug_shell"

Regenerate the GRUB configuration file:

sudo update-grub

Critical Security Hardening Considerations

Because debug-shell.service exposes an unauthenticated root terminal on Ctrl + Alt + F9, leaving it enabled in production environments creates a severe physical and console security vulnerability. Anyone with physical access to the server keyboard, KVM over IP, or hypervisor console can gain full root administrative access without credentials.

Always disable the service and remove the kernel parameter immediately after resolving boot issues:

sudo systemctl disable --now debug-shell.service

Verify that the unit is inactive and masked against accidental activation:

systemctl is-active debug-shell.service