How to Monitor Hardware Errors and Memory ECC Events in Linux Using rasdaemon

Modern enterprise servers and mission-critical workstations rely on hardware error correction mechanisms—such as Error-Correcting Code (ECC) system memory, Processor Machine Check Architecture (MCA), and PCI Express Advanced Error Reporting (AER)—to detect and correct transient electrical bit flips before they corrupt active workloads. Historically, Linux administrators relied on legacy daemons like mcelog to log hardware faults. However, modern processors from Intel and AMD, paired with the Linux kernel tracepoints infrastructure, have superseded mcelog with rasdaemon (Reliability, Availability, and Serviceability daemon). Operating directly against kernel trace events via the /sys/kernel/tracing interface, rasdaemon records, categorises, and stores ECC memory anomalies and PCIe errors in an SQLite database, enabling proactive replacement of failing DIMM modules before uncorrectable multi-bit crashes occur.

Architecture of rasdaemon and Kernel RAS Subsystems

Unlike polling-based utilities, rasdaemon acts as an asynchronous event-driven daemon that attaches to kernel tracepoints:

• Memory Controller (EDAC / MC): Listens to ras:mc_event tracepoints, capturing single-bit Correctable Errors (CE) and multi-bit Uncorrectable Errors (UE), mapping the physical memory address to the exact DIMM socket, memory channel, and slot label.
• Machine Check Exceptions (MCE): Listens to ras:aer_event and ras:mce_record, logging internal CPU cache tag faults, translation lookaside buffer (TLB) parity checks, and interconnect fabric anomalies.
• PCIe Advanced Error Reporting (AER): Records non-fatal correctable PCIe bus re-transmissions and fatal uncorrectable bus timeouts directly from device endpoints and root ports.
• Disk Error Monitoring: Intercepts block-layer SATA, SAS, and NVMe controller error events.

Installing and Enabling rasdaemon on Linux

The rasdaemon suite is available across standard distribution repositories, including Ubuntu, Debian, RHEL, and Fedora.

# On Debian and Ubuntu systems
sudo apt update
sudo apt install -y rasdaemon sqlite3

# On Enterprise Linux (RHEL / AlmaLinux / Rocky Linux)
sudo dnf install -y rasdaemon sqlite

# Enable and start the rasdaemon service
sudo systemctl daemon-reload
sudo systemctl enable --now rasdaemon

Verify that the daemon is actively running and bound to the kernel tracepoint filesystem:

sudo systemctl status rasdaemon

Auditing Active Hardware Error Records via ras-mc-ctl

The primary administrative command-line interface for rasdaemon is ras-mc-ctl. To inspect physical memory controllers and DIMM labels detected by the kernel EDAC driver:

# Inspect memory controller layout and DIMM socket mappings
sudo ras-mc-ctl --layout

To query summary error counts across memory controllers, PCIe endpoints, and CPU cores:

sudo ras-mc-ctl --summary

The output categorises detected hardware events into distinct telemetry blocks:

• Memory controller events: Displays total Correctable Errors (CE) and Uncorrectable Errors (UE) per memory channel.
• PCIe AER events: Displays root port transmission anomalies.
• MCE events: Details processor architectural exceptions.

Inspecting Granular Memory ECC Events

To view detailed timestamped records of memory errors—including the offending memory address, grain size, error syndrome, and physical DIMM slot label:

sudo ras-mc-ctl --errors

Examine the granular diagnostic output:

Memory controller events:
1 2026-11-05 14:12:08 +0000 error: Correctable error, memory type: DDR5, err_msg: "CE memory read error", label: "DIMM_A1", MC: 0, top_layer: 0, mid_layer: 0, low_layer: 0, address: 0x3f48a1200, grain: 64, syndrome: 0x0

If a single DIMM module (e.g. DIMM_A1) logs recurring correctable errors within a short timeframe, the physical silicon is deteriorating and approaching imminent uncorrectable failure, warranting immediate scheduled hardware replacement.

Querying the rasdaemon SQLite Database Directly

Because rasdaemon records raw tracepoint payloads into a standard SQLite database (typically stored at /var/lib/rasdaemon/ras-mc_event.db), administrators can execute structured SQL queries for automated monitoring and alerting:

# Inspect database schema and tables
sudo sqlite3 /var/lib/rasdaemon/ras-mc_event.db ".tables"

# Query the most recent 10 memory controller events
sudo sqlite3 -header -column /var/lib/rasdaemon/ras-mc_event.db \
    "SELECT timestamp, err_type, err_msg, label, address FROM mc_event ORDER BY id DESC LIMIT 10;"

# Query PCIe AER events
sudo sqlite3 -header -column /var/lib/rasdaemon/ras-mc_event.db \
    "SELECT timestamp, dev_name, err_type, msg FROM aer_event ORDER BY id DESC LIMIT 10;"

Automating Alerts for Memory Degradation

To integrate rasdaemon into Prometheus node exporters, Zabbix, or Datadog monitoring pipelines, create a lightweight cron script that checks for newly recorded errors:

sudo tee /usr/local/bin/check-ras-errors.sh << 'EOF'
#!/bin/bash
DB="/var/lib/rasdaemon/ras-mc_event.db"
if [ -f "$DB" ]; then
    ERRORS=$(sqlite3 "$DB" "SELECT count(*) FROM mc_event WHERE timestamp >= datetime('now', '-1 hour');")
    if [ "$ERRORS" -gt 0 ]; then
        logger -p user.err "CRITICAL: rasdaemon detected $ERRORS memory ECC event(s) in the past hour."
    fi
fi
EOF
sudo chmod +x /usr/local/bin/check-ras-errors.sh

Deploying rasdaemon provides Linux infrastructure engineers with definitive, hardware-level diagnostic visibility to isolate failing components before catastrophic downtime occurs.