Kdump Configuration

Configure kdump to analyse Linux kernel crash and kernel panics.

Linux kernel crash and panics and the reason behind it is analyzed with the help of kdump utility.
Kdump is a crash dumping mechanism and it uses the context of another kernel at boot time to capture the crash and core dump. The context of the kernel reserves a small amount of memory, and its only purpose is to capture the core dump of the crashed kernel.

Following are the steps to configure kdump from command prompt.

Login as a user root and edit /boot/grub/grub.conf file, and add the crashkernel=M parameter to the list of kernel options. After editing grub.conf file looks like as follows:

# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/sda3
# initrd /initrd-version.img
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Red Hat Enterprise Linux Server (2.6.18-194.8.1.el5)
root (hd0,0)
kernel /vmlinuz-2.6.18-194.8.1.el5 ro root=/dev/sda3 crashkernel=128M
initrd /initrd-2.6.18-194.8.1.el5.img

Now configure the target location in /etc/kdump.conf file. The default file location to store dump file i.e. vmcore files in the /var/crash/ directory of the local system.
It is also possible to save the file on remote location using NFS and SCP, but these techniques are not written here.

Edit /etc/kdump.conf file:
To change the local directory in which the core dump is to be saved, remove the hash sign (“#”) from the beginning of the #path /var/crash line, and replace the value with a desired directory path.

ext3 /dev/sda4
path /usr/local/cores

To write the dump directly to a device, remove the hash sign (“#”) from the beginning of the #raw /dev/sdc5 line, and replace the value with a desired device name. For example:
raw /dev/sdb1

Also possible to configure dump core using core collector. To reduce the size of the vmcore we can use makedumpfile utility.
To enable core collector search for core_collector directive in /etc/kdump.conf file and uncomment it if it is not.

core_collector makedumpfile –c

To remove the unwanted pages from the dump file we can pass –d option to core_collector.

core_collector makedumpfile -d –c
where Numeric value is a sum of values of pages we want to omit.

Option Page type to omit
1 Zero Pages
2 Cache Pages
4 Cache Private
8 User Pages
16 Blank Pages

Now all we need is to start the kdump service on boot time.

#chkconfig kdump on
Start the kdump service.

# service kdump start
No kdump initial ramdisk found. [WARNING]
Rebuilding /boot/initrd-2.6.18-194.8.1.el5kdump.img
Starting kdump: [ OK ]

Test the kdump configuration.
To test the configuration, reboot the system with kdump enabled, and make sure that the service is running:

# service kdump status
Kdump is operational
Then type the following commands at a shell prompt:
# echo 1 > /proc/sys/kernel/sysrq
# echo c > /proc/sysrq-trigger

The above command makes Linux kernel to crash, and the YYYY-MM-DD-HH:MM/vmcore file will be copied to the location we have selected in the configuration.

Analyze the core dump:
To analyze the crash kernel we need to packages and they are crash and kernel-debuginfo.

Now all we need to start the crash utility.

crash /var/crash/timestamp/vmcore /usr/lib/debug/lib/modules/kernel/vmlinux.

At the crash prompt we can run several commands like
Crash> log ## Display the mesg buffer
Crash>bt ## backtrace
Crash>[ps | vm | files] ## Refer man page for more options.