Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
System Analysis and Tuning Guide / Handling system dumps / Using systemd-coredump to debug application crashes
Applies to openSUSE Leap 15.4

19 Using systemd-coredump to debug application crashes

  

systemd-coredump collects and displays core dumps, for analyzing application crashes. The core dump contains an image of the process's memory at the time of termination. When a process crashes (or all processes belonging to an application), its default is to log the core dump to the systemd journal, including a backtrace if possible, and to store the core dump in a file in /var/lib/systemd/coredump. You also have the option to examine the dump file with other tools such as gdb or crash (see Section 18.8, “Analyzing the crash dump”).

Core dumps stored in /var/lib/systemd/coredump are deleted after three days (see the d /var/lib/systemd/coredump line in /usr/lib/tmpfiles.d/systemd.conf).

There is an option to not store core dumps, but to log only to the journal, which may be useful to minimize the collection and storage of sensitive information.

19.1 Use and configuration

  

systemd-coredump is enabled and ready to run by default. The default configuration is in /etc/systemd/coredump.conf: 

[Coredump]
#Storage=external
#Compress=yes
#ProcessSizeMax=2G
#ExternalSizeMax=2G
#JournalSizeMax=767M
#MaxUse=
#KeepFree=

Size units are B, K, M, G, T, P, and E. ExternalSizeMax also supports a value of infinity.

The following example shows how to use Vim for simple testing, by creating a segfault to generate journal entries and a core dump.

Procedure 19.1: Creating a core dump with Vim
  

  1. Enable the debuginfo-pool and debuginfo-update repositories

  2. Install vim-debuginfo

  3. Launch vim testfile and type a few characters

  4. Get the PID and generate a segfault: 

    > ps ax | grep vim
2345 pts/3    S+     0:00 vim testfile

# kill -s SIGSEGV 2345

    Vim will emit error messages: 

    Vim: Caught deadly signal SEGV
Vim: Finished.
Segmentation fault (core dumped)

  5. List your core dumps, then examine them: 

    # coredumpctl
TIME                        PID  UID  GID SIG PRESENT EXE
Wed 2019-11-12 11:56:47 PST 2345 1000 100 11  *       /bin/vim

# coredumpctl info
PID: 2345 (vim)
UID: 0 (root)
GID: 0 (root)
Signal: 11 (SEGV)
Timestamp: Wed 2019-11-12 11:58:05 PST
Command Line: vim testfile
Executable: /bin/vim
Control Group: /user.slice/user-1000.slice/session-1.scope
    Unit: session-1.scope
    Slice: user-1000.slice
    Session: 1
    Owner UID: 1000 (tux)
    Boot ID: b5c251b86ab34674a2222cef102c0c88
    Machine ID: b43c44a64696799b985cafd95dc1b698
    Hostname: linux-uoch
    Coredump: /var/lib/systemd/coredump/core.vim.0.b5c251b86ab34674a2222cef102
    Message: Process 2345 (vim) of user 0 dumped core.

         Stack trace of thread 2345:
         #0  0x00007f21dd87e2a7 kill (libc.so.6)
         #1  0x000000000050cb35 may_core_dump (vim)
         #2  0x00007f21ddbfec70 __restore_rt (libpthread.so.0)
         #3  0x00007f21dd92ea33 __select (libc.so.6)
         #4  0x000000000050b4e3 RealWaitForChar (vim)
         #5  0x000000000050b86b mch_inchar (vim)
[...]

When you have multiple core dumps, coredumpctl info displays all of them. Filter them by PID, COMM (command), or EXE (full path to the executable), for example, all core dumps for Vim: 

# coredumpctl info /bin/vim

See a single core dump by PID: 

# coredumpctl info 2345

Output the selected core to gdb: 

# coredumpctl gdb 2345

The asterisk in the PRESENT column indicates that a stored core dump is present. If the field is empty there is no stored core dump, and coredumpctl retrieves crash information from the journal. You can control this behavior in /etc/systemd/coredump.conf with the Storage option:

  • Storage=none—core dumps are logged in the journal, but not stored. This is useful to minimize collecting and storing sensitive information, for example for General Data Protection Regulation (GDPR) compliance.

  • Storage=external—cores are stored in /var/lib/systemd/coredump

  • Storage=journal—cores are stored in the systemd journal

A new instance of systemd-coredump is invoked for every core dump, so configuration changes are applied with the next core dump, and there is no need to restart any services.

Core dumps are not preserved after a system restart. You may save them permanently with coredumpctl. The following example filters by the PID and stores the core in vim.dump: 

# coredumpctl -o vim.dump dump 2345

See man systemd-coredump, man coredumpctl, man core, and man coredump.conf for complete command and option listings.