Configure RHEL 9 Kdump and Triage a Kernel Crash

Kdump boots a reserved capture kernel after a kernel panic and saves the failed kernel’s memory as a vmcore. Preparing it before an incident turns an unexplained reboot into evidence that can be analyzed. This guide covers RHEL 9 on x86_64 with a conventional supported boot configuration, a local dump target, and basic offline crash triage.

Check the platform and plan a reboot

Use root access and an approved maintenance window for package, boot-argument, and service changes. Unified Kernel Images, other CPU architectures, remote targets, and encrypted storage need their specific Red Hat procedure. This is a kernel-dump guide; user-process core dumps use a different capture and analysis path.

The optional crash test stops the entire host

Step 5 deliberately panics the kernel. Run it only on a disposable lab host or during an explicitly approved outage with console access, saved data, and a recovery plan. Kdump cannot guarantee capture after every hardware failure or power loss.

01

Record the Kernel and Existing Capture Configuration

Baseline

Identify the OS release, architecture, running kernel, installed tools, and existing crashkernel command-line value. Record the original boot and kdump configuration before changing it. Keep the failed kernel’s exact release when investigating a later dump: the current kernel after an update may differ. A present configuration file is not evidence that a capture kernel is loaded.

cat /etc/redhat-release
uname -r
uname -m
rpm -q kexec-tools crash
cat /proc/cmdline
sudo grubby --info=ALL
sudo cat /etc/kdump.conf

❯ What to Verify

Normal kernel -> panic -> reserved capture kernel -> dump target/vmcore
                                                        |
                                                        v
                                  matching vmlinux symbols + crash analysis

The exact kernel and pre-change boot/capture configuration are recorded.

02

Install the Tools and Reserve Suitable Crash Memory

Planned Change

Install kexec-tools if missing. Use the packaged kdumpctl estimate and supported default reservation procedure rather than copying a RAM value from another server. The estimate helps evaluate device drivers, initramfs, and encrypted-target requirements. Check the installed tool’s help because options depend on the RHEL minor release. A boot-argument change needs an approved reboot before the memory is reserved, and the reservation reduces memory available to normal workloads.

# Approved package/boot change window:
sudo dnf install kexec-tools
sudo kdumpctl estimate
kdumpctl --help
# On a release supporting this option, apply the packaged default reservation:
sudo kdumpctl reset-crashkernel --kernel=ALL
sudo grubby --info=ALL
# Perform the scheduled reboot through your maintenance procedure.
# After reboot:
cat /proc/cmdline

❯ What to Verify

The booted kernel has the intended crashkernel reservation.
The reserved memory is suitable for this host and its planned target.

RHEL 9 does not support crashkernel=auto

Use the current packaged defaults or a reviewed explicit reservation. Red Hat documents crashkernel=auto as unsupported on RHEL 9 and later. A successful memory estimate does not replace an end-to-end dump test.

03

Prepare a Dump Target with Capacity and Access Controls

Capture Storage

For this local-target example, verify that /var/crash is on a supported filesystem, check its owner and mode, and confirm adequate space for the planned capture and retention. Do not assume the compressed dump will have a fixed size. Back up the configuration, then review the target, collector, failure_action, and final_action directives in place. Keep the distribution’s supported collector defaults unless a specific analysis requirement justifies another filter. Remote or encrypted targets must work from the capture kernel’s reduced environment, not just the running OS.

sudo test -f /etc/kdump.conf || { echo 'Expected kdump.conf is missing; stop.' >&2; exit 1; }
sudo test ! -e /etc/kdump.conf.incident01-before || { echo 'Backup path already exists; choose a new incident-specific name.' >&2; exit 1; }
sudo test ! -L /etc/kdump.conf.incident01-before || { echo 'Backup path is a symlink; inspect it before continuing.' >&2; exit 1; }
sudo cp -a /etc/kdump.conf /etc/kdump.conf.incident01-before || exit 1
sudo test ! -L /var/crash || { echo '/var/crash is a symlink; inspect it before continuing.' >&2; exit 1; }
if sudo test ! -e /var/crash; then
  sudo install -o root -g root -m 0700 -d /var/crash
fi
sudo stat -c '%A %U:%G %n' /var/crash
findmnt --target /var/crash
df -h /var/crash
sudoedit /etc/kdump.conf
# Review active directives; edit existing lines instead of appending duplicates.
sudo awk '!/^[[:space:]]*(#|$)/ {print}' /etc/kdump.conf

❯ What to Verify

The chosen target is supported, capacity/retention are planned,
and active path, collector, and completion/failure actions are understood.

04

Load the Capture Kernel and Check Readiness

Operational Check

Enable and start kdump after the approved configuration and reboot. Read service diagnostics, the kdumpctl status, and the loaded-crash-kernel flag. On a successful oneshot service, active (exited) can be normal. A flag of 1 indicates a capture kernel is loaded, but it does not prove that the target will accept the dump under panic conditions. Diagnose memory, driver, initramfs, or target-access errors before attempting the crash test.

sudo systemctl enable --now kdump
sudo kdumpctl status
cat /sys/kernel/kexec_crash_loaded
systemctl status kdump --no-pager
sudo journalctl -u kdump -b --no-pager
# After a later approved target/configuration edit, rebuild/reload via:
# sudo kdumpctl restart

❯ What to Verify

kexec_crash_loaded: 1
Kdump reports operational readiness with no unresolved target/load error.
This is readiness evidence, not proof of a saved vmcore.

05

Test Capture During an Approved Lab Outage

Optional Destructive Test

Use a lab host matching the intended drivers and dump target, with working console access and no unsaved work. Stop the application through its maintenance procedure and confirm kdump readiness. The final command below immediately requests a kernel panic; it is not a service restart. Observe the capture kernel at the console, wait for dump completion, and follow the configured final action. Do not run this step from an unattended fleet job.

# LAB / EXPLICITLY APPROVED OUTAGE ONLY.
sudo kdumpctl status
cat /sys/kernel/kexec_crash_loaded
sync
# This next command deliberately crashes the host immediately:
sudo sh -c 'echo c > /proc/sysrq-trigger'

❯ What to Verify

Console evidence shows the capture kernel saving the dump to the chosen target.
After recovery, a new vmcore and diagnostic files exist for this test.

Memory dumps remain sensitive after filtering

A vmcore can reveal kernel memory, identifiers, credentials, and application-related data. Store it as restricted incident evidence and share it only through an approved support channel. Page filtering or compression does not guarantee removal of sensitive material.

06

Locate the Dump and Preserve Its Provenance

Evidence

Find the new dump directory and compare its timestamp with the incident or lab test. Retain vmcore, available kernel logs, the host/kernel identity, capture configuration, and console errors. Compute a hash before transferring the file. Analyze a protected copy and keep the original intact. If no vmcore exists, return to the console/target error rather than assuming that an automatic reboot was a successful capture.

sudo find /var/crash -maxdepth 2 -type f \
  \( -name vmcore -o -name 'vmcore-dmesg*' \) -ls
# Set the real path selected from the listing, in this Bash session:
vmcore='/var/crash/REPLACE_WITH_REAL_DUMP_DIRECTORY/vmcore'
sudo test -f "$vmcore" || exit 1
sudo sha256sum "$vmcore"
sudo journalctl -b -1 -k --no-pager

❯ What to Verify

The dump is matched to the failed host/kernel and incident timestamp.
A hash and capture details accompany the protected analysis copy.

07

Install Matching Symbols on the Analysis Host

Offline Analysis

Use an authorized analysis host with enough space and the crash package. Obtain the exact kernel-debuginfo package for the kernel that generated the dump through your supported debug repositories or vendor support. Do not select symbols using the analysis host’s uname output. The uncompressed vmlinux must match the dumped kernel’s release and architecture. Verify the path and dump identity before interpreting output; symbol mismatch can prevent analysis or produce unreliable conclusions.

# On the analysis host, through approved repositories:
sudo dnf install crash
# Install the exact failed-kernel debuginfo package using your supported repository.
# Set this from incident/dump metadata, not the analysis host's uname -r:
crashed_kernel='REPLACE_WITH_EXACT_FAILED_KERNEL_RELEASE'
symbols="/usr/lib/debug/lib/modules/$crashed_kernel/vmlinux"
test -f "$symbols" || exit 1
# Set the exact path to the protected analysis copy on this host.
vmcore='/secure/incidents/incident01/vmcore'
test -f "$vmcore" || exit 1
sudo crash "$symbols" "$vmcore"

❯ What to Verify

crash opens the saved dump with matching symbols and identifies the failed kernel.
Any mismatch or unsupported dump-format error is resolved before analysis.

08

Inspect the Panic, Backtrace, Tasks, and Modules

Triage

At the crash prompt, inspect the system summary and kernel log first, then the panic task’s backtrace, process list, memory summary, and loaded modules. A function or module appearing in a stack is a lead, not proof that it caused the failure. Distinguish a deliberately triggered lab panic from the production crash signature. Record the panic text, taint/module information, and relevant stacks for the kernel or hardware support team.

crash> sys
crash> log
crash> bt
crash> ps
crash> kmem -i
crash> mod
crash> help bt
crash> quit

❯ What to Verify

The failed kernel, panic message, panic-task stack, and relevant module/task
state are retained, with uncertainty and further analysis ownership recorded.

09

Maintain Capture Readiness After Kernel and Storage Changes

Operations

Recheck kdump after a kernel update, driver change, target migration, or reservation change. Verify readiness after each planned reboot and repeat the destructive end-to-end test only in an appropriate lab or approved window. If a change breaks capture, restore the reviewed previous configuration and boot reservation through the maintenance plan. Monitor dump-target capacity and retention so the next incident can still produce usable evidence.

uname -r
cat /proc/cmdline
sudo kdumpctl status
cat /sys/kernel/kexec_crash_loaded
df -h /var/crash
sudo journalctl -u kdump -b --no-pager

❯ What to Verify

The current kernel has a ready capture environment and a maintained target.
The last end-to-end test date and any untested changes are documented.

Primary references

Use the kernel kdump architecture guide, kernel SysRq reference, Red Hat’s RHEL 9 kdump configuration procedure, and RHEL 9 core-dump analysis guide. Consult local kdump.conf(5), kdumpctl(8), and crash(8) manuals.