How to Automatically Restart Failed Systemd Services in Ubuntu

When running web servers, database backends, Node.js applications, Python automation bots, or background daemons on an Ubuntu server, unexpected process crashes are an inevitable reality of production environments. Memory leaks, unexpected network timeouts, unhandled code exceptions, or temporary database locks can cause critical services to terminate silently. If your system requires manual SSH intervention to run systemctl restart every time a crash occurs, downtime and user frustration mount rapidly.

Fortunately, Linux’s core service manager—systemd—includes robust process monitoring and automated recovery mechanisms. By adding a few simple restart directives to a service unit configuration file, you can instruct Ubuntu to detect crashed processes instantly and relaunch them within seconds. In this guide, you will learn how to configure automated service restarts in Ubuntu using systemd.

Why Automate Service Recovery with Systemd?

Configuring systemd restart policies delivers essential production resilience:

• Zero-touch fault tolerance: Restores failed microservices and web backends immediately without waiting for on-call engineers.
• Prevents crash-loop storms: Built-in rate-limiting mechanisms prevent broken code from endlessly restarting hundreds of times per second and saturating CPU resources.
• Native kernel monitoring: systemd tracks processes using Linux control groups (cgroups), ensuring that spawned subprocesses and worker threads are tracked accurately.

Step 1: Locating or Creating Your Service Unit File

Custom system services in Ubuntu are defined in /etc/systemd/system/. Open an existing service file or create a new one using your preferred terminal text editor:

sudo nano /etc/systemd/system/myapp.service

Step 2: Adding the Restart Directives

Inside your service unit file, locate the [Service] section. Add the Restart= and RestartSec= directives as shown in the following example:

[Unit]
Description=My Custom Production Daemon
After=network.target

[Service]
Type=simple
User=ubuntu
WorkingDirectory=/var/www/myapp
ExecStart=/usr/bin/node /var/www/myapp/server.js
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

Understanding Systemd Restart Policies

The Restart= directive supports several distinct operational behaviours:

• Restart=on-failure (Recommended): Automatically restarts the service only if the process exits with an unclean exit code (non-zero), is killed by an unhandled signal (such as SIGSEGV or SIGKILL), or times out. It will not restart if you explicitly stop the service using systemctl stop or if the program exits cleanly with code 0.
• Restart=always: Restarts the process regardless of how it terminated, including clean exits.
• RestartSec=5s: Introduces a mandatory 5-second buffer before relaunching the process, allowing network sockets and database locks to clear cleanly.

Step 3: Configuring Crash Loop Rate Limiting

To ensure that a fatal bug (such as a missing configuration file) does not cause systemd to spin in an infinite loop, define burst limits in the [Unit] section:

[Unit]
Description=My Custom Production Daemon
StartLimitIntervalSec=60s
StartLimitBurst=5

This setting instructs systemd that if the service crashes and restarts more than 5 times within a 60-second window, automated restarts should halt to protect system stability.

Step 4: Reloading Daemon Configurations and Testing

Whenever you edit a systemd unit file, you must reload the manager configuration before restarting:

sudo systemctl daemon-reload
sudo systemctl restart myapp.service

To verify that auto-recovery works in practice, find the process ID (PID) of your service with systemctl status myapp.service, and forcefully simulate a crash by killing it:

sudo kill -9 [PID]

Check systemctl status myapp.service after 5 seconds; you will see that systemd automatically spawned a new healthy process with a new PID.