How to Inspect and Debug System Event Loops Using systemd-analyze dump in Linux

When troubleshooting frozen system transitions, unkillable services, or unexpected dependency deadlocks in Linux, standard tools like systemctl status or journalctl often fail to reveal what the init system is doing internally. At PID 1, systemd maintains an intricate internal state machine consisting of event loops, asynchronous D-Bus subscriptions, job queues, timer deadlines, and serialized file descriptors. The systemd-analyze dump command extracts an exhaustive, plain-text memory serialization of PID 1’s runtime state, giving systems administrators and kernel engineers an unprecedented diagnostic window into the core service manager.

Understanding the Internal State of PID 1

Unlike legacy SysV init scripts that executed sequential, blocking shell scripts, systemd functions as an event-driven transaction manager. PID 1 relies on an sd-event loop that continuously multiplexes incoming I/O from Unix sockets, inotify filesystem watches, POSIX signal queues, and timer events.

Every unit file configured on the system—whether a .service, .socket, .mount, or .timer—is represented internally as an instantiated C structure within PID 1’s address space. When services transition between inactive, activating, active, and deactivating states, jobs are placed in an internal dependency graph. If a transaction becomes deadlocked or an internal timeout fails to fire, systemd-analyze dump exposes the precise memory state, active transactions, and pending callbacks within the daemon.

Generating a Complete systemd Serialization Dump

Because dumping the internal state machine accesses sensitive IPC structures and extracts detailed process memory graphs, the operation requires root privileges.

To inspect the entire internal state of PID 1, open a terminal and execute:

sudo systemd-analyze dump

The command streams a voluminous textual dump to standard output. On a typical production workstation or server, this report spans tens of thousands of lines. To analyze it effectively, pipe the output into a pager or redirect it into a timestamped file for offline inspection:

sudo systemd-analyze dump > /tmp/systemd-state-$(date +%Y%m%d-%H%M%S).txt

You can then inspect the generated file using less or search specific sections using standard text utilities.

Inspecting Pending and Running Jobs in the Transaction Queue

When an operating system appears hung during boot, reboot, or service reload, the root cause is almost always a pending job awaiting a prerequisite condition that never resolves. The beginning of the dump report outlines the active job queue.

Search for the jobs header within the dump:

sudo systemd-analyze dump | grep -A 15 "Jobs:"

The output presents a list of outstanding jobs along with their state, ID, and target unit:

Jobs:
-> Job 452:
   Unit: nginx.service
   Type: restart
   State: running
   Waiting for:
   -> Job 453:
      Unit: php-fpm.service
      Type: start
      State: waiting

This hierarchy clearly indicates which job is blocking the queue, allowing you to identify the specific prerequisite service preventing completion.

Auditing Unit States and Descriptors

Following the job list, systemd-analyze dump catalogues every active and loaded unit on the system. For each unit, it details low-level runtime properties that do not appear in normal systemctl show summaries.

To inspect the detailed internal state of a specific service (for example, sshd.service), extract its block from the dump file:

awk '/^-> Unit sshd.service:/{flag=1; print; next} /^-> Unit/{flag=0} flag' /tmp/systemd-state-*.txt

The extracted block reveals vital internal parameters:

• Description and LoadState: Confirms if the unit is loaded cleanly from disk or generated dynamically in memory.
• ActiveState and SubState: Exposes the micro-state of the internal state machine (such as auto-restart, stop-sigterm, or condition-failed).
• UnitFileState: Indicates enablement status across targets.
• MainPID and ControlPID: Tracks currently monitored processes, including ephemeral child scripts executing in ExecStartPre or ExecStop.
• File Descriptors: Lists raw file descriptors held open by PID 1 for the service’s standard I/O sockets.

Analyzing the Watchdog and Timer Queues

Service watchdog failures occur when a daemon fails to send a keep-alive ping via sd_notify("WATCHDOG=1") before the configured WatchdogSec= deadline. When troubleshooting watchdog triggers or systemd timer anomalies, the dump report reveals the exact hardware and software timer expirations calculated by the event loop.

Filter the dump output for timer subscriptions:

grep -E "(Watchdog|NextElapse|Timer)" /tmp/systemd-state-*.txt -B 2 -A 4

This allows you to verify whether a service’s watchdog timer is actively running, the timestamp when the last watchdog ping was received by PID 1, and the precise moment the supervisor will issue a forced SIGABRT.

Diagnosing D-Bus Client Subscriptions and Locks

PID 1 communicates with external utilities (such as systemctl, polkit, and custom management agents) over the private D-Bus socket located at /run/systemd/private. If a buggy management daemon opens an unbuffered D-Bus connection or leaks event listeners, PID 1’s memory consumption can increase.

Search the dump file for active D-Bus tracking entries:

grep -A 10 "Subscribed:" /tmp/systemd-state-*.txt

This exposes all external clients that have registered subscriptions to unit state change signals. If an orphaned process or unresponsive remote management agent is holding open hundreds of subscriptions, it will be prominently displayed here, enabling you to terminate the rogue client.

Comparing Dumps to Identify State Leaks Across Restarts

When performing system daemon reloads via systemctl daemon-reload or in-place supervisor re-executions via systemctl daemon-reexec, PID 1 serializes its entire state to a temporary file descriptor in /run and reads it back into memory after re-execution.

If you suspect that a service reload leaves orphaned state or lingering file descriptors behind, capture a baseline dump, execute the reload, capture a second dump, and compare them:

sudo systemd-analyze dump > /tmp/state-before.txt
sudo systemctl daemon-reload
sudo systemd-analyze dump > /tmp/state-after.txt
diff -u /tmp/state-before.txt /tmp/state-after.txt | grep -E "^[+-]->"

The resulting diff pinpoints units that altered their internal graph or accumulated new dependencies during the reload cycle, providing the deepest possible insight into systemd’s operational mechanics.