In high-security enterprise environments and regulatory compliance frameworks (such as PCI-DSS, SOC 2, and ISO 27001), preserving system log integrity is paramount. Traditionally, if an attacker successfully breached a Linux server and obtained root privileges, one of their primary evasion tactics was to truncate, edit, or selectively delete log records inside /var/log/ to hide reconnaissance, privilege escalation, and lateral movement traces.
To eliminate this threat and make system logs cryptographically tamper-evident, systemd provides Forward Secure Sealing (FSS). Built directly into systemd-journald and managed via journalctl, FSS applies continuous cryptographic seals across local journal files using an advancing mathematical key. Even if an attacker subsequently gains root privileges and modifies existing log files, they cannot forge or rewrite previous seals. Running a cryptographic verification immediately flags the modification, pinpointing the exact microsecond and log entry where tampering occurred. This guide explains how Forward Secure Sealing functions in Ubuntu, how to generate FSS sealing and verification keys, and how to verify journal integrity.
How Forward Secure Sealing (FSS) Operates
Forward Secure Sealing relies on the cryptographic concept of forward secrecy implemented via monotonic epoch-based hash chains:
- Sealing Key and Verification Key Pair: When FSS is initialized, systemd generates a private sealing key and a corresponding public verification key. The sealing key remains on the server and is used by
systemd-journaldto periodically sign and seal new log blocks. The verification key is copied off the server and stored in a secure location (e.g. an offline administrative vault or dedicated SIEM). - Advancing Key Intervals: The sealing key does not remain static. It continuously advances through mathematical one-way cryptographic functions across predefined intervals (defaulting to every 15 minutes). Once a key interval elapses, the old sealing key is permanently destroyed and replaced by the new key. Because the key derivation function is irreversible, possessing the current key does not allow an attacker to reconstruct previous sealing keys.
- Tamper Detection: If an intruder alters or deletes earlier log records, the cryptographic signature of the affected log epoch becomes invalid. When an auditor runs
journalctl --verifyusing the off-site verification key, systemd detects the signature mismatch and reports the exact entry where the seal was broken.
FSS ensures that while an attacker with root access can theoretically stop future logging or corrupt current log files, they cannot rewrite historical logs to conceal their intrusion without instant cryptographic detection.
Prerequisites and Persistent Journal Configuration
Forward Secure Sealing requires persistent systemd journal storage. On Ubuntu Server, verify that systemd-journald stores logs permanently to disk under /var/log/journal/ rather than in volatile RAM (/run/log/journal/).
Inspect the journal storage configuration:
# Check journal storage mode
journalctl --disk-usage
If logs reside in /run/log/journal/, make storage persistent by creating the dedicated journal directory and setting Storage=persistent:
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
# Ensure Storage=persistent in journald.conf
sudo sed -i 's/^#Storage=.*/Storage=persistent/' /etc/systemd/journald.conf
sudo systemctl restart systemd-journald
Generating Forward Secure Sealing (FSS) Keys
To initialize Forward Secure Sealing across your system journal, use the journalctl --setup-keys command as root:
sudo journalctl --setup-keys
The command generates a new FSS key pair and outputs a structured security certificate to the terminal:
Generating seed...
Generating keys...
The sealing key is generated:
Key interval: 15min
Epoch start: Sun 2026-11-08 03:00:00 UTC
The verification key is:
3a8f10b4/2c89e14a/7d4e5f91/0b1c2d3e/4f5a6b7c/8d9e0f1a/2b3c4d5e/6f7a8b9c
Passphrase:
(no passphrase)
Crucial Security Step: Immediately copy and securely store the verification key string (e.g. 3a8f10b4/2c89e14a/...) in an external password manager, offline vault, or administrative management workstation. Once printed to the console, this verification key will never be displayed again on the server. The server retains only the ephemeral sealing key inside /var/log/journal/<machine-id>/fss.
Verifying Journal Log Seals Cryptographically
Periodic integrity audits should be performed during security evaluations, incident response investigations, or automated compliance monitoring.
Validating Journal Integrity with the Verification Key
To audit all local journal files against the trusted verification key, execute journalctl --verify with the --verify-key parameter:
sudo journalctl --verify --verify-key="3a8f10b4/2c89e14a/7d4e5f91/0b1c2d3e/4f5a6b7c/8d9e0f1a/2b3c4d5e/6f7a8b9c"
systemd reads every journal file on disk, validates cryptographic seals epoch-by-epoch, and reports the verification status:
PASS: /var/log/journal/9a1e4c923d1b4fa8b2a049e0c5f1074a/system.journal
PASS: /var/log/journal/9a1e4c923d1b4fa8b2a049e0c5f1074a/user-1000.journal
PASS: /var/log/journal/9a1e4c923d1b4fa8b2a049e0c5f1074a/[email protected]
=> All sealed journal files verified successfully. No tampering detected.
If all files report PASS, the historical integrity of your system logs is mathematically guaranteed.
Simulating and Detecting Log Tampering
To understand how Forward Secure Sealing behaves during an incident, consider what occurs if an unauthorized user attempts to edit a sealed journal file.
If an attacker modifies even a single byte within an existing sealed journal file (e.g. altering an SSH login record or changing an IP address), running journalctl --verify immediately detects the invalid cryptographic digest:
FAIL: /var/log/journal/9a1e4c923d1b4fa8b2a049e0c5f1074a/system.journal
Invalid signature in epoch 14 (Sun 2026-11-08 03:15:00 UTC) at entry 18,402.
Data corrupted or modified after sealing!
The failure output gives forensic investigators immediate actionable data: the exact journal file, the compromised sealing epoch, and the precise log entry index where data was modified.
Automating Journal Verification via cron or systemd Timers
To enforce continuous integrity auditing, schedule an automated periodic verification task. Because the verification key must remain secure, automate audits by pulling journal files to a secure central logging server or executing audits through a read-only bastion host:
# Example daily verification cron script on central log server
0 4 * * * /usr/bin/journalctl --directory=/backup/logs/server01/ --verify --verify-key="3a8f10b4/..." >/var/log/journal_audit.log 2>&1
Combining Forward Secure Sealing with remote log aggregation (via systemd-journal-remote) establishes a multi-layered defence that prevents log tampering and satisfies rigorous compliance baselines across Ubuntu enterprise environments.