How to Audit Systemd Configuration Overrides and Drop-in Files in Ubuntu Using systemd-delta

Maintaining complex Linux environments often involves customizing system services, modifying daemon timeout values, or overriding package-provided configuration defaults. The systemd service manager implements a strict configuration hierarchy that reads files across vendor distributions (/usr/lib/systemd/), local system administration directories (/etc/systemd/), and ephemeral runtime directories (/run/systemd/). When services misbehave, identifying which drop-in snippets or shadowed files are altering default behaviour can require manual file audits. The systemd-delta diagnostic utility automates this discovery by scanning all configuration paths and identifying masked, extended, redirected, and overridden files.

Understanding the systemd Configuration Hierarchy

To avoid package updates overwriting local customizations, systemd processes configuration files according to an established priority order:

1. /etc/systemd/system/: Local administrator customizations. Highest priority.
2. /run/systemd/system/: Transient runtime units created dynamically by running daemons. Overrides vendor packages.
3. /usr/lib/systemd/system/ (or /lib/systemd/system/): Upstream package manager files provided by Ubuntu repositories. Lowest priority.

When an administrator creates an override in /etc/ with the same name as a file in /usr/lib/, the local file shadows the vendor file entirely. Alternatively, creating a drop-in directory (such as nginx.service.d/override.conf) appends or modifies specific directives without replacing the whole file. systemd-delta tracks these relationships and displays the exact architectural diff between vendor defaults and local state.

Running a Full System Audit with systemd-delta

To audit all configuration overrides across your Ubuntu installation, open a terminal and execute:

systemd-delta

The command parses the directory hierarchy and outputs an annotated list categorizing how local files diverge from vendor baselines. Each entry is tagged with a status type:

[MASKED]     /etc/systemd/system/apt-daily.service -> /dev/null
[OVERRIDDEN] /etc/systemd/system/systemd-resolved.service -> /lib/systemd/system/systemd-resolved.service
[EXTENDED]   /lib/systemd/system/docker.service
             /etc/systemd/system/docker.service.d/http-proxy.conf
[REDIRECTED] /etc/systemd/system/syslog.service -> /lib/systemd/system/rsyslog.service

4 overridden configuration files found.

This report highlights vital system modifications:

• MASKED: Units symlinked to /dev/null to prevent activation manually or via dependencies.
• OVERRIDDEN: A whole vendor file in /lib/ is superseded by a complete copy in /etc/.
• EXTENDED: The base unit is augmented by one or more drop-in configuration snippets in a .d/ directory.
• REDIRECTED: A unit alias pointing to another service.

Filtering Audits by Delta Type

In large enterprise deployments, viewing every extended or redirected unit can produce noisy output. The --type= (or -t) parameter limits output to specific override classifications, including masked, equivalent, redirected, overridden, extended, or unchanged.

To locate only masked services that may be preventing services from starting, run:

systemd-delta --type=masked

To locate services that have been completely overridden—which carries the risk of missing upstream bug fixes when the base Ubuntu package is updated—execute:

systemd-delta --type=overridden

Inspecting Inline Differences with Embedded Diffs

Identifying that a file is overridden does not explain which parameters changed. The --diff= flag instructs systemd-delta to invoke the diff utility inline, displaying the exact line-by-line differences between the vendor package unit and your local administrator override.

To view contextual diffs across all overridden units, execute:

systemd-delta --diff=yes --type=overridden

The output outputs standard unified diff syntax:

--- /lib/systemd/system/ssh.service
+++ /etc/systemd/system/ssh.service
@@ -9,4 +9,5 @@
 ExecStartPre=/usr/sbin/sshd -t
-ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
+ExecStart=/usr/sbin/sshd -D -o LogLevel=VERBOSE $SSHD_OPTS
 Restart=on-failure
+RestartSec=10s

This pinpoint accuracy reveals whether a local file merely adjusts a restart counter or introduces major security modifications.

Targeting Specific Subsystems and Units

By default, systemd-delta audits all systemd configuration hierarchies, including journald, logind, resolved, timesyncd, and udev rules. You can restrict the audit to a specific subsystem by providing path prefixes as trailing arguments.

To audit only core unit files in the system manager:

systemd-delta systemd/system

To check for overrides applied specifically to network management services:

systemd-delta systemd/network

To audit user-level service units configured under personal home directories:

systemd-delta --user

Reverting Unwanted Overrides Cleanly

If systemd-delta exposes an obsolete override that was created during temporary debugging and is no longer needed, you can restore package defaults cleanly.

To remove a drop-in override:

sudo rm -r /etc/systemd/system/service-name.service.d/

To unmask a masked service:

sudo systemctl unmask service-name.service

To remove a full file override and return to vendor settings:

sudo rm /etc/systemd/system/service-name.service

After modifying any configuration files on disk, always reload systemd to apply the changes to PID 1:

sudo systemctl daemon-reload

Running systemd-delta periodically provides administrators with complete visibility over server configuration drift, ensuring custom adjustments remain documented, safe, and maintainable.