How to Integrate Custom Scripts and Daemons with systemd Using systemd-notify in Ubuntu

When orchestrating custom background daemons, automated backup scripts, database initializers, or internal web services in Ubuntu, configuring a standard systemd service unit with Type=simple or Type=forking often creates subtle race conditions. In a Type=simple service, systemd assumes the service has successfully started the instant the main process is spawned by the kernel. If dependent services—such as an API gateway or reporting worker—rely on that service being fully initialized, listening on a network socket, or pre-populating database schemas, dependent services fail prematurely before initialization completes.

The native systemd notification protocol solves this problem through Type=notify service units. By executing the systemd-notify command within custom shell scripts, Python daemons, or compiled binaries, background processes communicate directly with systemd over a dedicated Unix domain socket ($NOTIFY_SOCKET). Processes can signal exact startup readiness, broadcast real-time status strings visible in systemctl status, send watchdog keep-alive heartbeats, and report orderly shutdown stages.

Understanding the systemd-notify Architecture

When systemd starts a service declared with Type=notify, it provisions a temporary, private Unix domain datagram socket and passes its filesystem path to the child process via the NOTIFY_SOCKET environment variable. Until the running application transmits the READY=1 state string across this socket, systemd marks the unit in an activating state and delays launching any downstream units configured with After= or Requires= directives.

The systemd-notify CLI tool acts as a convenient, universal client that packages these messages and delivers them across the socket. This enables developers writing Bash scripts or third-party wrappers to leverage advanced systemd coordination without writing low-level C socket bindings.

Supported Notification Directives and Syntax

The systemd-notify utility transmits structured key-value pairs recognized by the init system:

• READY=1: Tells systemd that service initialization is complete and the service is fully operational.
• STATUS=string: Passes an arbitrary single-line status message that appears dynamically in systemctl status.
• WATCHDOG=1: Sends a watchdog keep-alive ping to reset the systemd watchdog timer. If the process freezes and fails to send this heartbeat within the configured interval, systemd terminates and restarts the unit.
• RELOADING=1: Informs systemd that the daemon is reloading configuration files.
• STOPPING=1: Signals that the process has begun graceful termination procedures.
• ERRNO=int: Reports a POSIX error code if startup fails.

Creating a Notification-Aware Bash Worker Script

To demonstrate a realistic production scenario, consider a custom database synchronization script that requires several seconds to verify database connections, download state schemas, and start processing records.

Create the script at /usr/local/bin/sync-worker.sh:

sudo nano /usr/local/bin/sync-worker.sh

Add the following script content:

#!/bin/bash
set -e

# Broadcast initial bootstrap status
systemd-notify --status="Initializing local database connections..."

# Simulate initialization work
sleep 3

# Update status during schema verification
systemd-notify --status="Loading cache tables and validating schemas..."
sleep 2

# Signal complete operational readiness to systemd
systemd-notify --ready --status="Worker active and processing task queue"

# Main processing loop with watchdog heartbeat
while true; do
    # Perform routine task
    sleep 5
    # Send watchdog heartbeat ping
    systemd-notify WATCHDOG=1
done

Make the script executable:

sudo chmod +x /usr/local/bin/sync-worker.sh

Configuring the Type=notify Service Unit

Next, construct the corresponding systemd service unit that expects notification signals from the script. Create a new service file in /etc/systemd/system/:

sudo nano /etc/systemd/system/sync-worker.service

Define the unit configuration:

[Unit]
Description=Custom Sync Worker Service with Notification Support
After=network.target

[Service]
Type=notify
ExecStart=/usr/local/bin/sync-worker.sh
Restart=on-failure
TimeoutStartSec=30
WatchdogSec=15
NotifyAccess=all

[Install]
WantedBy=multi-user.target

Key configuration directives to observe:

• Type=notify: Instructs systemd to withhold the “active (running)” state until READY=1 is received.
• TimeoutStartSec=30: If the script fails to transmit READY=1 within 30 seconds, systemd considers the startup failed and kills the process.
• WatchdogSec=15: Enables hardware/software watchdog monitoring. The service must send WATCHDOG=1 at least once every 15 seconds.
• NotifyAccess=all: Allows child processes or subshells invoked by the script to access the notification socket.

Reloading, Starting, and Verifying the Service

Inform systemd of the new unit file by reloading daemon configurations:

sudo systemctl daemon-reload

Start the service:

sudo systemctl start sync-worker.service

During the initial five seconds of execution, inspect the unit status:

systemctl status sync-worker.service

Observe that the output displays the dynamically updated status line:

● sync-worker.service - Custom Sync Worker Service with Notification Support
     Loaded: loaded (/etc/systemd/system/sync-worker.service; disabled; preset: enabled)
     Active: activating (start) since Thu 2026-11-05 10:15:00 UTC; 2s ago
     Status: "Initializing local database connections..."

Once the script reaches systemd-notify --ready, the status transitions to active (running), and the status line updates to reflect the operational state:

     Active: active (running) since Thu 2026-11-05 10:15:05 UTC; 10s ago
     Status: "Worker active and processing task queue"

Testing Watchdog Failure and Recovery

To verify that systemd enforces health monitoring, simulate a process stall or freeze by sending a SIGSTOP signal to the running script process:

sudo killall -STOP sync-worker.sh

Because the frozen script cannot emit WATCHDOG=1 pings, systemd detects the elapsed WatchdogSec=15 timeout within 15 seconds. Systemd automatically terminates the unresponsive process with SIGABRT, records a watchdog timeout event in the journal, and restarts the service cleanly according to the Restart=on-failure policy.

By integrating systemd-notify into Ubuntu scripts and custom daemons, system administrators eliminate startup race conditions, provide clear diagnostic messages to systemctl, and ensure robust watchdog-driven self-healing across infrastructure components.