4.16.  Kernel dumps

[Note]Note
This feature is only available since openSUSE 11.1 and SLES 11. It is not available on the zSeries (s390x) architecture!

With kdump the system is able to create crashdump files if the whole system (i.e., the kernel) crashes. That crash dump files contain the memory contents while the system crashed. Such core files can be analyzed later by the support or a (kernel) developer to find the reason why the system crashed. It's mostly useful for servers where you cannot easily reproduce such crashes on another system and where it's important that the problem gets fixed.

The only downside of enabling kdump is that this costs you between 64 MiB and 128 MiB of System RAM (on "normal" sized systems) that needs to be reserved to be used by kdump in the case the system crashes and the dump needs to be generated.

This section only describes how to setup kdump with AutoYaST. It does not describe how kdump works in general and also does not describe each small detail. Refer to the kdump(7) manual page which is in the package named kdump or to the openSUSE Kdump documentation.

At first, let's present an overall example for the kdump configuration:

Example 4.40.  Kdump configuration

<kdump>
  <!-- memory reservation -->
  <add_crash_kernel config:type="boolean">true</add_crash_kernel>
  <crash_kernel>256M-:64M</crash_kernel>

  <general>
    <!-- dump target settings -->
    <KDUMP_SAVEDIR>ftp://stravinsky.suse.de/incoming/dumps</KDUMP_SAVEDIR>
    <KDUMP_COPY_KERNEL>true</KDUMP_COPY_KERNEL>
    <KDUMP_FREE_DISK_SIZE>64</KDUMP_FREE_DISK_SIZE>
    <KDUMP_KEEP_OLD_DUMPS>5</KDUMP_KEEP_OLD_DUMPS>

    <!-- filtering and compression -->
    <KDUMP_DUMPFORMAT>compressed</KDUMP_DUMPFORMAT>
    <KDUMP_DUMPLEVEL>1</KDUMP_DUMPLEVEL>

    <!-- notification -->
    <KDUMP_NOTIFICATION_TO>bwalle@suse.de</KDUMP_NOTIFICATION_TO>
    <KDUMP_NOTIFICATION_CC></KDUMP_NOTIFICATION_CC>
    <KDUMP_SMTP_SERVER>mail.suse.de</KDUMP_SMTP_SERVER>
    <KDUMP_SMTP_USER></KDUMP_SMTP_USER>
    <KDUMP_SMTP_PASSWORD></KDUMP_SMTP_PASSWORD>

    <!-- kdump kernel -->
    <KDUMP_KERNELVER></KDUMP_KERNELVER>
    <KDUMP_COMMANDLINE></KDUMP_COMMANDLINE>
    <KDUMP_COMMANDLINE_APPEND></KDUMP_COMMANDLINE_APPEND>

    <!-- expert settings -->
    <KDUMP_IMMEDIATE_REBOOT>yes</KDUMP_IMMEDIATE_REBOOT>
    <KDUMP_VERBOSE>15</KDUMP_VERBOSE>
    <KEXEC_OPTIONS></KEXEC_OPTIONS>
  </general>
</kdump> 

4.16.1. Memory Reservation

As already mentioned above, it's necessary to reserve some memory at bootup for kdump. Because that memory must be reserved very early during the boot process, the configuration is done via a kernel command line parameter called crashkernel. That memory will be used to load a second kernel in that memory that will be executed without rebooting if the first kernel crashes. That kernel also has a special initrd which contains all programs that are necessary to save the dump to network or disk, send the notification email and finally reboot.

You can enable or disable that the crashkernel parameter is written for the default boot kernel with the add_crash_kernel tag and you can specify the value of the crashkernel parameter using the crash_kernel tag.

For the memory reservation there are two things to specify: The amount of reserved memory (such as 64M to reserve 64 MiB of memory from the RAM) and the offset. The syntax is crashkernel=AMOUNT@OFFSET. Luckily the kernel is able to auto-detect the right offset for you nowadays (with the exception of the Xen hypervisor where you have to specify 16M as offset), so you don't have to worry about that too much. You can just specify <crash_kernel>crashkernel=64M</crash_kernel> and the right thing will happen.

For the amount of memory, following values are recommended:

Table 4.10. Recommended values for the reserved memory amount

PlatformRecommended values
i386 and x86-64 64M for small machines (about 2 GiB of RAM, 4 cores) and 128M for larger machines
PPC64 128M for small machines and 256M for larger machines
IA64 256M for small machines, 512M for medium machines and 1G and more for really large machines (mostly SGI Altix systems)


To make things even more complicated, there's a so-called extended command line syntax where you can specify the amount of reserved memory dependent of the System RAM. That is good if you share one AutoYaST profile for multiple installations or when you often remove or install memory on one machine. The syntax is:

BEGIN_RANGE_1-END_RANGE_1:AMOUNT_1,BEGIN_RANGE_2-END_RANGE_2:AMOUNT_2@OFFSET

In that syntax BEGIN_RANGE_1 is the start of the first memory range (for example: 0M) and END_RANGE_1 is the end of the first memory range (and can be empty in case "infinity" should be assumed) and so on. So for example 256M-2G:64M,2G-:128M means to reserve 64 MiB of crashkernel memory when the system has between 256 MiB and 2 GiB RAM and to reserve 128 MiB of crashkernel memory when the system has above 2 GiB RAM.

The following table contains the settings that are necessary for the memory reservation:

Table 4.11. XML representation of the memory reservation settings

ElementDescriptionComment
add_crash_kernelIf the memory should be reserved, that basically enables or disables kdump.
<add_crash_kernel config:type="boolean">true</add_crash_kernel>
required
crash_kernelThe syntax of the crashkernel command line as discussed above.
<crash_kernel>256M:64M</crash_kernel>
required


4.16.2.  Dump Saving

4.16.2.1.  Target

The element KDUMP_SAVEDIR holds an URL to where the dump is saved. Following methods are possible:

  • file to save to the local disk

  • ftp to save to an FTP server (without encryption)

  • sftp to save to an SSH2 SFTP server

  • nfs to save to a NFS location and

  • cifs to save the dump to a CIFS/SMP export from Samba or Microsoft Windows.

For details see the kdump(5) manual page. Two examples are: file:///var/crash (which is the default location according to FHS) and ftp://user:password@host:port/incoming/dumps. Below that directory, a directory name that contains a time stamp will be created in which the dumps are saved.

When the dump is saved to the local disk, KDUMP_KEEP_OLD_DUMPS can be used to delete old dumps automatically. This setting takes a number that specifies how much old dumps should be kept. If the partition has less than KDUMP_FREE_DISK_SIZE megabytes free disk space after saving the dump, the dump is not copied at all.

If you would not like only to save the dump but also the whole kernel and (if installed) the debug information of the kernel to that directory to have everything you need (except all kernel modules and the debugging information of all kernel modules) to analyze the dump in one directory, you can set KDUMP_COPY_KERNEL to true.

4.16.2.2.  Filtering and Compression

The size of kernel dumps is uncompressed and unfiltered as large as your system has RAM. To get smaller files (for example, to send it to support), you can compress the whole dump file afterwards. However, the drawback is that the dump has to be uncompressed afterwards before opening, so the disk space needs to be there in any case.

To use page compression which compresses every page and allows dynamic decompression with the crash(8) debugging tool, set KDUMP_DUMPFORMAT to compressed (which is actually the default).

To filter the dump, you have to set the KDUMP_DUMPLEVEL. Then not all memory is saved to disk but only memory that does not fulfill some criteria. I.e. you may want to leave out pages that are completely filled by zeroes as they don't contain any useful information. 0 produces a full dump and 31 is the smallest dump. The manual page kdump(5) and makedumpfile(8) contain a table that lists for each value which pages will be saved.

4.16.2.3.  Summary

Table 4.12. XML representation of the dump target settings

ElementDescriptionComment
KDUMP_SAVEDIRAn URL that specifies the target to which the dump and related files will be saved.
<KDUMP_SAVEDRIR>file:///var/crash/</KDUMP_SAVEDIR>
required
KDUMP_COPY_KERNELIf not only the dump itself should be saved to KDUMP_SAVEDIR but also the kernel and its debugging information (if installed).
<KDUMP_COPY_KERNEL>false</KDUMP_COPY_KERNEL>
optional
KDUMP_FREE_DISK_SIZE The number of megabytes that should always be free after saving the dump. If that space would be below that value, the dump will not be copied.
<KDUMP_FREE_DISK_SIZE>64</KDUMP_FREE_DISK_SIZE>
optional
KDUMP_KEEP_OLD_DUMPS The number of dumps that are kept (i.e., not deleted) if KDUMP_SAVEDIR points to a local directory. Specify 0 if you don't want to delete dumps at all and specify -1 if all dumps (except the one that is just saved) should be deleted.
<KDUMP_KEEP_OLD_DUMPS>4</KDUMP_KEEP_OLD_DUMPS>
optional


4.16.3.  Email Notification

It's useful to get notified via email that a machine has crashed and a dump has been saved. That way you can for example setup a dump server in a company and trigger some actions by that email automatically like calling the administrator from home to check if everything runs again.

Because the dump is saved in a special initrd environment, we cannot use a local mail server just to send that notification email. However, it's better to send that email in the initrd just because it's more likely that we have a working network connection here (which we need in the netdump case to save the dump away anyway) compared that the server comes up again and everything is working.

You have to provide at least exactly one address in KDUMP_NOTIFICATION_TO and zero, one or more addresses in KDUMP_NOTIFICATION_CC. Please note that you can only specify the address here, not a real name or some other fancy stuff.

To actually send the email, we need KDUMP_SMTP_SERVER and (if the server needs authentication) KDUMP_SMTP_USER and KDUMP_SMTP_PASSWORD. Please note that TSL or SSL are not supported. That may be added in future.

Table 4.13. XML representation of the email notification settings

ElementDescriptionComment
KDUMP_NOTIFICATION_TOExactly one email address (and only an address) to which the mail should be sent. Additional recipients can be specified in KDUMP_NOTIFICATION_CC.
<KDUMP_NOTIFICATION_TO>bwalle@suse.de</KDUMP_NOTIFICATION_TO>
optional (email notification is disabled if empty)
KDUMP_NOTIFICATION_CCZero, one or more recipients that are in the Cc line of the notification mail.
<KDUMP_NOTIFICATION_CC>spam@suse.de devnull@suse.de</KDUMP_NOTIFICATION_CC>
optional
KDUMP_SMTP_SERVER Host name of the SMTP server that will be used for the mail delivery. Please note that the SMTP authentication is supported (see KDUMP_SMTP_USER and KDUMP_SMTP_PASSWORD) but TSL and SSL are not supported.
<KDUMP_SMTP_SERVER>email.suse.de</KDUMP_SMTP_SERVER>
optional (email notification is disabled if empty)
KDUMP_SMTP_USER User name that is used together with KDUMP_SMTP_PASSWORD for SMTP authentication.
<KDUMP_SMTP_USER>bwalle</KDUMP_SMTP_USER>
optional
KDUMP_SMTP_PASSWORD Password that is used together with KDUMP_SMTP_USER for SMTP authentication.
<KDUMP_SMTP_PASSWORD>geheim</KDUMP_SMTP_PASSWORD>
optional


4.16.4.  Kdump kernel settings

As already mentioned, a special kernel is booted to save the dump. If you don't want to use the auto-detection mechanism to find out which kernel is used (see the kdump(5) manual page that describes the algorithm which is used to find the kernel), you can specify the version of a custom kernel in KDUMP_KERNELVER. If you set that to foo, then the kernel located in /boot/vmlinuz-foo or /boot/vmlinux-foo (in that order on platforms that have a vmlinuz file) will be used.

You can even specify the command line which will be used to boot the kdump kernel. Normally the boot command line is used minus some settings that hurt in the kdump case (like the crashkernel parameter itself) plus some settings that are needed in the kdump case (see the manual page kdump(5)). If you just want some additional parameters like a overwritten console setting then use KDUMP_COMMANDLINE_APPEND. If you know what you're doing and you want to specify the whole command line, set KDUMP_COMMANDLINE.

Table 4.14. XML representation of the kernel settings

ElementDescriptionComment
KDUMP_KERNELVERVersion string for the kernel that will be used for kdump. Leave it empty to use the auto-detection mechanism (strongly recommended).
<KDUMP_KERNELVER>2.6.27-default</KDUMP_KERNELVER>
optional (auto-detection if empty)
KDUMP_COMMANDLINE_APPENDAdditional command line parameters for the kdump kernel.
<KDUMP_COMMANDLINE_APPEND>console=ttyS0,57600</KDUMP_COMMANDLINE_APPEND>
optional
KDUMP_COMMANDLINE Overwrite the automatically generated kdump command line. Use with care. Normally KDUMP_COMMANDLINE_APPEND is the setting you're looking for.
<KDUMP_COMMANDLINE_APPEND>root=/dev/sda5 maxcpus=1 irqpoll</KDUMP_COMMANDLINE>
optional (email notification is disabled if empty)


4.16.5.  Expert settings

Table 4.15. XML representation of the expert settings

ElementDescriptionComment
KDUMP_IMMEDIATE_REBOOTtrue if the system should be rebooted automatically after the dump has been saved, false otherwise. The default is to reboot the system automatically.
<KDUMP_IMMEDIATE_REBOOT>true</KDUMP_IMMEDIATE_REBOOT>
optional
KDUMP_VERBOSEBitmask that specifies how to verbose the kdump process should be. Read kdump(5) for details.
<KDUMP_VERBOSE>3</KDUMP_VERBOSE>
optional
KEXEC_OPTIONSAdditional options that are passed to kexec when loading the kdump kernel. Normally empty.
<KEXEC_OPTIONS>--noio</KEXEC_OPTIONS>
optional