Architectural Foundation of chrony, NTP Stratums, and Hardware Timestamping
In distributed infrastructure, financial ledger systems, and high-performance database clusters (such as CockroachDB, Cassandra, or Ceph), clock synchronization is an uncompromising prerequisite. Clock skew between nodes breaks consensus algorithms, invalidates distributed transaction orderings, and induces severe split-brain states. On Ubuntu Server, the modern, high-precision implementation of the Network Time Protocol (NTP) is chrony, which supersedes both legacy ntpd and the basic SNTP client (systemd-timesyncd).
While standard NTP synchronizes system time over UDP port 123 across the public internet with millisecond accuracy, chrony can achieve microsecond and sub-microsecond precision when coupled with Hardware Timestamping and Precision Time Protocol (PTP / IEEE 1588) physical hardware clocks (PHCs). Operating via the chronyd background daemon, chrony continuously calculates system clock drift, evaluating both frequency error (measured in parts per million, ppm) and phase offset. Rather than jumping time abruptly, chrony slews the system clock smoothly, ensuring monotonic time progression essential for database journaling.
| Daemon / Subsystem | Protocol Specification | Clock Disciplining Method | Hardware Timestamping Support | Typical Accuracy Baseline |
|---|---|---|---|---|
systemd-timesyncd |
SNTP (Simple NTP) | Periodic step / crude slew | No (Pure software) | 10–50 milliseconds |
ntpd (Reference NTP) |
NTPv4 (RFC 5905) | Slow gradual slew (Phase Lock Loop) | Limited / Complex | 1–5 milliseconds |
chronyd (Standard NTP) |
NTPv4 / NTS | Rapid adaptive slew + configurable step | Software & OS socket layer | 100–500 microseconds |
chronyd + PTP / PHC |
IEEE 1588 PTP Hardware Clock | Direct hardware counter synchronization | Yes (NIC MAC layer timestamping) | 10–50 nanoseconds |
Disabling systemd-timesyncd and Installing chrony
Ubuntu Server activates systemd-timesyncd by default. Prior to deploying chrony, disable the native SNTP client to prevent dual daemon conflicts on kernel clock discipline:
# Disable and mask systemd-timesyncd
sudo systemctl stop systemd-timesyncd
sudo systemctl disable systemd-timesyncd
sudo systemctl mask systemd-timesyncd
# Install chrony and diagnostic utilities
sudo apt-get update
sudo apt-get install -y chrony
Configuring /etc/chrony/chrony.conf for High Precision
The primary configuration file /etc/chrony/chrony.conf governs upstream time sources, stepping rules, drift recording, and hardware clock bindings.
# Backup default configuration
sudo cp /etc/chrony/chrony.conf /etc/chrony/chrony.conf.orig
# Edit the master configuration file
sudo tee /etc/chrony/chrony.conf > /dev/null << 'EOF'
# ==========================================================
# Production chrony.conf Configuration
# ==========================================================
# 1. Upstream Stratum 1 & 2 NTP Servers (National Physical Laboratory & Pool)
server time.cloudflare.com iburst minpoll 4 maxpoll 8
server time.google.com iburst minpoll 4 maxpoll 8
pool 2.ubuntu.pool.ntp.org iburst maxsources 4
# 2. Record Clock Drift Rate to Non-Volatile Storage
driftfile /var/lib/chrony/chrony.drift
# 3. Clock Step Configuration: Step the clock on initial boot if offset > 1.0s,
# but strictly within the first 3 updates; subsequent adjustments must slew smoothly:
makestep 1.0 3
# 4. Kernel RTC Synchronization (keep real-time hardware clock in sync)
rtcsync
# 5. Network Hardware Timestamping (Enable on physical interface if supported)
# hwtimestamp eth0
# 6. Fallback Stratum Server for Local Subnets (Act as local NTP server)
# Serve time to internal LAN even if upstream internet connectivity is lost:
local stratum 10
allow 192.168.1.0/24
# 7. Security Hardening: Log directory and process permissions
logdir /var/log/chrony
log measurements statistics tracking
EOF
Restart the daemon to commit configuration adjustments:
sudo systemctl restart chrony
sudo systemctl enable chrony
Binding to PTP Hardware Clocks (PHC) for Microsecond Accuracy
On physical enterprise bare-metal servers equipped with modern network interface cards (such as Intel I210, E810, or Mellanox ConnectX), the physical network controller maintains an internal hardware counter driven by an integrated oscillator. If your network switches support PTP, chrony can discipline the kernel directly against the NIC’s hardware clock device (/dev/ptp0).
Step 1: Identify PTP Hardware Capabilities
# Audit network interface timestamping capabilities via ethtool
ethtool -T eth0
Verify that output reports: hardware-transmit, hardware-receive, and PTP Hardware Clock: 0.
Step 2: Bind the PTP Refclock in chrony.conf
Append a reference clock directive pointing directly to the character device node:
# Bind /dev/ptp0 with high sampling frequency
refclock PHC /dev/ptp0 poll 0 dpoll -2 offset 0.0 delay 1e-6 precision 1e-9
Auditing Synchronization Performance via chronyc Commands
The chronyc command-line client communicates with chronyd over a local Unix domain socket, providing real-time telemetry.
1. Tracking Current Offset and Root Delay:
chronyc tracking
Key telemetry indicators:
- Reference ID: The active upstream server currently disciplining the clock.
- Stratum: Number of hops from reference atomic clock (Stratum 1 = direct atomic/GPS; Stratum 2 = synchronized to Stratum 1).
- System time: Current offset relative to reference (e.g.,
0.000014 seconds slow). - RMS offset: Long-term Root-Mean-Square dispersion; optimal production systems report < 0.000100s (100 microseconds).
- Frequency: Clock oscillator drift rate in ppm.
2. Inspecting Upstream Time Sources:
chronyc sources -v
Review the status indicator flags in the left column:
*: Currently selected best reference source.+: Valid candidate source contributing to the weighted average.-: Valid source considered an outlier and excluded by clustering algorithm.?: Source connectivity lost or probe packets unacknowledged.x: False ticker (time source contradicts consensus majority).
3. Analysing Source Statistics and Jitter:
chronyc sourcestats -v
Examine the Est offset and Std Dev columns to determine network jitter across individual upstream NTP peers.
Troubleshooting chrony and Time Synchronization Faults
| Observed Fault | Probable Root Cause | Technical Diagnostic Method | Remediation Protocol |
|---|---|---|---|
Sources Display “?” and Unreachable |
Outbound UDP port 123 blocked by enterprise perimeter firewall | Execute nc -z -v -u time.google.com 123 |
Open UDP port 123 on egress network security groups and local firewall |
506 Cannot talk to daemon |
chronyd service crashed or socket permissions blocked by AppArmor |
Run sudo systemctl status chrony; inspect journalctl |
Ensure daemon is running; verify ownership of /run/chrony/chronyd.sock |
| System Clock Jumping Abruptly | makestep configured without update boundary limits, or VM snapshot restore |
Inspect /var/log/chrony/measurements.log for step events |
Enforce makestep 1.0 3 so steps are restricted strictly to boot initialisation |
No synchronised sources found |
All defined NTP servers reporting excessive root dispersion or untrusted stratum | Run chronyc sources; check stratum ratings |
Verify DNS resolution in /etc/resolv.conf; add geographically local NTP pool servers |
| Virtual Machine Clock Drift Accumulating | Hypervisor stealing CPU cycles (Steal Time) or unsynchronised host clock | Run top and inspect %st (Steal) CPU register |
Install guest integration tools (e.g., open-vm-tools) or sync against host KVM clock source |