How to Configure Kernel Crash Dumps on Ubuntu Server Using kdump-tools

When a production Linux server encounters a fatal kernel panic or critical hardware deadlock, the system halts instantly without writing error diagnostics to standard log files in /var/log. Because the crashed kernel cannot execute user-space processes safely, the root cause of the crash—whether triggered by a faulty kernel module, a hardware memory parity fault, or a storage driver deadlock—is lost upon system reboot.

The Linux Kernel Crash Dump mechanism (kdump), powered by kexec, ensures full diagnostic recovery during unexpected panics. When a fatal kernel error occurs, kexec boots directly into a lightweight secondary recovery kernel (the “capture kernel”) residing in reserved memory, bypassing the standard BIOS/UEFI boot cycle. The capture kernel then writes a complete snapshot of the crashed kernel memory space (vmcore) to disk for post-mortem analysis.

Installing kdump-tools on Ubuntu Server

Ubuntu provides user-space management utilities for kdump in its standard repositories:

1. Update local package indexes and install the kdump packages:

sudo apt update && sudo apt install -y kdump-tools crash kexec-tools linux-crashdump

2. During installation, an interactive prompt will ask whether to enable kdump by default. Select Yes.

Configuring Reserved Memory in the GRUB Bootloader

For the capture kernel to boot reliably when the primary kernel panics, dedicated system RAM must be reserved in advance so the crashed kernel cannot corrupt it:

1. Open the GRUB default configuration file:

sudo nano /etc/default/grub.d/kdump-tools.cfg

2. Verify or adjust the crashkernel memory allocation:

# Allocate memory dynamically based on total system RAM
GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT crashkernel=512M"

For servers with more than 16GB of RAM, allocating 512M to 1G ensures sufficient workspace for large core dumps.

3. Update the GRUB configuration to write the parameters into boot records:

sudo update-grub

4. Reboot the server once to apply the hardware memory reservation:

sudo reboot

Verifying kdump Status and Crash Kernel Reservation

Once the server reboots, verify that the capture kernel is loaded into reserved memory:

1. Inspect the status of kdump-tools:

sudo kdump-config status

The output must confirm:

current state   : ready to kdump
kexec_load      : loaded
kdump kernel    : /boot/vmlinuz-...

2. Verify reserved memory boundaries via kernel memory maps:

cat /sys/kernel/kexec_crash_loaded
dmesg | grep -i crashkernel

A return value of 1 confirms the recovery kernel is primed in active memory.

Configuring Dump Destinations in kdump-tools.conf

By default, crash dumps are saved locally to /var/crash. You can customize storage policies to manage disk consumption:

1. Open the kdump configuration file:

sudo nano /etc/default/kdump-tools

2. Adjust core dump parameters:

# Maximum number of crash dump archives to retain
NUM_DUMPS=3

# Directory where vmcore images will be saved
KDUMP_DIR=/var/crash

# Filter level for makedumpfile (compresses data and omits zero pages)
MAKEDUMP_ARGS="-c -d 31"

Analysing Crash Dumps Using the crash Utility

When a kernel panic occurs, a timestamped subfolder containing the compressed core file (dump.YYYYMMDDHHMM) is generated in /var/crash:

1. Launch the crash debugger against the active debug kernel and dump image:

sudo crash /usr/lib/debug/boot/vmlinux-$(uname -r) /var/crash/202611070245/dump.202611070245

2. Inside the crash environment, inspect the stack trace at the exact moment of failure:

bt          # Print kernel backtrace
log         # View dmesg buffer captured prior to panic
sys         # Display system architecture and panic string

Configuring kdump guarantees that mission-critical Ubuntu servers provide full diagnostic forensic trails whenever an unexpected kernel panic occurs.