How to Inspect and Debug Hardware PCI Express Advanced Error Reporting (AER) in Linux Using dmesg and setpci

PCI Express (PCIe) devices communicate across high-speed serial interconnects where physical signal degradation, electromagnetic interference, or faulty driver transactions can trigger hardware-level faults. To detect and diagnose these low-level transmission issues before they lead to kernel panics or silent data corruption, the PCI Express specification incorporates Advanced Error Reporting (AER). In Linux environments, the kernel’s PCIe AER service driver intercepts these hardware reports and records correctable and uncorrectable errors directly into system rings. By combining kernel logging utilities with low-level configuration space inspection tools such as setpci and lspci, systems engineers can pinpoint failing peripheral devices, degraded cables, or inadequate motherboard trace routing.

Understanding PCIe Correctable vs Uncorrectable AER Errors

The PCIe AER capability divides transmission and protocol failures into distinct architectural severities:

• Correctable Errors: Minor link or physical layer anomalies—such as receiver framing errors, bad transaction layer packets (TLP), or bad data link layer packets (DLLP)—that hardware automatically recovers through hardware retries. While these errors do not corrupt state, high rates indicate physical physical link degradation or failing transceiver hardware.
• Uncorrectable Non-Fatal Errors: Transactions that could not be completed normally, but the PCIe link remains functional and other devices on the bus continue operating without corruption.
• Uncorrectable Fatal Errors: Critical hardware failures—such as malformed TLPs, unexpected completions, or flow control protocol violations—that render the link unreliable, typically requiring a system reset or secondary bus reset.

Checking Kernel AER Capability and Boot-Time Configuration

Before analysing runtime errors, confirm that the Linux kernel was compiled with PCIe AER support and verify whether AER handling is globally enabled or suppressed in boot flags.

Execute the following command to verify kernel configuration flags:

zgrep -E "CONFIG_PCIEAER|CONFIG_PCIE_ECRC" /proc/config.gz 2>/dev/null || grep -E "CONFIG_PCIEAER|CONFIG_PCIE_ECRC" /boot/config-$(uname -r)

Ensure that CONFIG_PCIEAER=y is enabled. Next, check if any command-line parameters in GRUB suppress AER events:

cat /proc/cmdline | grep -o -E "pci=noaer|pci=nomsi|pcie_ports=[a-z]+" || echo "Standard PCIe AER handling active"

Monitoring AER Events via dmesg and journalctl

When an AER-capable endpoint or root port detects a protocol violation, the kernel PCIe AER service logs structured diagnostic data. You can filter the kernel ring buffer specifically for AER reports:

sudo dmesg -T | grep -i -E "aer|pcie.*error"

A typical uncorrectable or correctable AER message appears with the PCI domain, bus, device, and function (BDF) identifier along with bit flags indicating the specific layer violation:

pcieport 0000:00:1c.0: AER: Corrected error received: 0000:01:00.0
0000:01:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver Error)
0000:01:00.0:   device [10de:2204] error status/mask=00000001/0000e000
0000:01:00.0:    [ 0] RxErr

To inspect real-time AER reporting across systemd journals across reboots, execute:

sudo journalctl -k -b 0 --grep="AER" --no-pager

Inspecting AER Capability Registers with lspci

To view the full register dump of a PCIe device’s Advanced Error Reporting structure, use lspci with verbose flags pointing to the target BDF address:

# Identify target device address
lspci | grep -i -E "nvme|vga|ethernet"

# Inspect detailed AER capability block
sudo lspci -s 01:00.0 -vvv | grep -A 25 -i "Advanced Error Reporting"

The output enumerates whether Uncorrectable Error Status (UESta), Uncorrectable Error Mask (UEMsk), and Correctable Error Status (CESta) bits are set or masked.

Reading and Clearing AER Status Registers Using setpci

The setpci utility allows direct querying and manipulation of PCI configuration space registers. Because AER capability addresses can vary depending on where the capability pointer resides within extended configuration space, first locate the AER capability offset.

To locate the exact hexadecimal offset of the AER extended capability (Capability ID 0001h), query lspci:

sudo lspci -s 01:00.0 -xxx | head -n 8

Alternatively, locate the offset programmatically:

AER_OFFSET=$(sudo lspci -s 01:00.0 -vvv | awk '/Advanced Error Reporting/ {print $NF}' | tr -d '[]')
echo "AER Capability Base Offset: 0x$AER_OFFSET"

Under PCIe specifications, standard AER registers are positioned relative to the base offset:

• +0x04: Uncorrectable Error Status Register (32-bit)
• +0x08: Uncorrectable Error Mask Register (32-bit)
• +0x10: Correctable Error Status Register (32-bit)
• +0x14: Correctable Error Mask Register (32-bit)

To read the 32-bit Uncorrectable Error Status register directly via setpci:

sudo setpci -s 01:00.0 0x104.L

To read the Correctable Error Status register:

sudo setpci -s 01:00.0 0x110.L

PCIe error status registers are Write-1-to-Clear (W1C). If you are troubleshooting an active link and want to reset the logged error flags to observe whether errors recur under synthetic load, write back the current bitmask:

# Clear Correctable Error Status register (writing 1s clears the flagged bits)
STATUS=$(sudo setpci -s 01:00.0 0x110.L)
echo "Current CE Status: $STATUS"
sudo setpci -s 01:00.0 0x110.L=0x$STATUS

Masking Spurious AER Correctable Errors

Certain high-throughput cards or active optical cables produce minor correctable receiver errors that flood dmesg and consume CPU cycles. While physical link inspection is the primary remedy, you can temporarily mask specific correctable error reporting at the hardware level using setpci.

The Correctable Error Mask register uses bit 0 for Receiver Error (RxErr) and bit 6 for Bad TLP. To mask Receiver Errors (setting bit 0 to 1), calculate the desired mask and write it to the mask register:

# Read current mask
CURRENT_MASK=$(sudo setpci -s 01:00.0 0x114.L)
echo "Current CE Mask: 0x$CURRENT_MASK"

# Set bit 0 to 1 (masking RxErr)
sudo setpci -s 01:00.0 0x114.L=0x00000001

Verifying and managing PCIe AER registers directly from the Linux command line provides deep hardware visibility, allowing you to distinguish benign link retries from catastrophic bus failures before hardware permanently drops offline.