Chapter 4. Analyzing and Managing System Log Files

Contents

4.1. System Log Files in /var/log/
4.2. Viewing and Parsing Log Files
4.3. Managing Log Files with logrotate
4.4. Monitoring Log Files with logwatch
4.5. Using logger to Make System Log Entries

System log file analysis is one of the most important tasks when analyzing the system. In fact, looking at the system log files should be the first thing to do when maintaining or troubleshooting a system. SUSE Linux Enterprise Server automatically logs almost everything that happens on the system in detail. Normally, system log files are written in plain text and therefore, can be easily read using an editor or pager. They are also parsable by scripts, allowing you to easily filter their content.

4.1. System Log Files in /var/log/

System log files are always located under the /var/log directory. The following list presents an overview of all system log files from SUSE Linux Enterprise Server present after a default installation. Depending on your installation scope, /var/log also contains log files from other services and applications not listed here. Some files and directories described below are placeholders and are only used, when the corresponding application is installed. Most log files are only visible for the user root.

acpid

Log of the advanced configuration and power interface event daemon (acpid), a daemon to notify user-space programs of ACPI events. acpid will log all of its activities, as well as the STDOUT and STDERR of any actions to syslog.

apparmor

AppArmor log files. See Part “Confining Privileges with Novell AppArmor” (↑Security Guide) for details of AppArmor.

audit

Logs from the audit framework. See Part “The Linux Audit Framework” (↑Security Guide) for details.

boot.msg

Log of the system init process—this file contains all boot messages from the Kernel, the boot scripts and the services started during the boot sequence.

Check this file to find out whether your hardware has been correctly initialized or all services have been started successfully.

boot.omsg

Log of the system shutdown process - this file contains all messages issued on the last shutdown or reboot.

ConsoleKit/*

Logs of the ConsoleKit daemon (daemon for tracking what users are logged in and how they interact with the computer).

cups/

Access and error logs of the Common UNIX Printing System (cups).

faillog

Database file that contains all login failures. Use the faillog command to view. See man 8 faillog for more information.

firewall

Firewall logs.

gdm/*

Log files from the GNOME display manager.

krb5

Log files from the Kerberos network authentication system.

lastlog

The lastlog file is a database which contains info on the last login of each user. Use the command lastlog to view. See man 8 lastlog for more information.

localmessages

Log messages of some boot scripts, for example the log of the DHCP client.

mail*

Mail server (postfix, sendmail) logs.

messages

This is the default place where all Kernel and system log messages go and should be the first place (along with /var/log/warn) to look at in case of problems.

NetworkManager

NetworkManager log files

news/*

Log messages from a news server.

ntp

Logs from the Network Time Protocol daemon (ntpd).

pk_backend_zypp

PackageKit (with libzypp backend) log files.

puppet/*

Log files from the data center automation tool puppet.

samba/*

Log files from samba, the Windows SMB/CIFS file server.

SaX.log

Logs from SaX2, the SUSE advanced X11 configuration tool.

scpm

Logs from the system configuration profile management (scpm).

warn

Log of all system warnings and errors. This should be the first place (along with /var/log/messages) to look at in case of problems.

wtmp

Database of all login/logout activities, runlevel changes and remote connections. Use the command last to view. See man 1 last for more information.

xinetd.log

Log files from the extended Internet services daemon (xinetd).

Xorg.0.log

X startup log file. Refer to this in case you have problems starting X. Copies from previous X starts are numbered Xorg.?.log.

YaST2/*

All YaST log files.

zypp/*

libzypp log files. Refer to these files for the package installation history.

zypper.log

Logs from the command line installer zypper.

4.2. Viewing and Parsing Log Files

To view log files, you can use your favorite text editor. There is also a simple YaST module for viewing /var/log/messages, available in the YaST Control Center under Miscellaneous+System Log.

For viewing log files in a text console, use the commands less or more. Use head and tail to view the beginning or end of a log file. To view entries appended to a log file in real-time use tail -f. For information about how to use these tools, see their man pages.

To search for strings or regular expressions in log files use grep. awk is useful for parsing and rewriting log files.

4.3. Managing Log Files with logrotate

Log files under /var/log grow on a daily basis and quickly become very big. logrotate is a tool for large amounts of log files and helps you to manage these files and to control their growth. It allows automatic rotation, removal, compression, and mailing of log files. Log files can be handled periodically (daily, weekly, or monthly) or when exceeding a particular size.

logrotate is usually run as a daily cron job. It does not modify any log files more than once a day unless the log is to be modified because of its size, because logrotate is being run multiple times a day, or the --force option is used.

The main configuration file of logrotate is /etc/logrotate.conf. System packages as well as programs that produce log files (for example, apache2) put their own configuration files in the /etc/logrotate.d/ directory. The content of /etc/logrotate.d/ is included via /etc/logrotate.conf.

Example 4.1. Example for /etc/logrotate.conf

# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
dateext

# uncomment this if you want your log files compressed
#compress

# comment these to switch compression to use gzip or another
# compression scheme
compresscmd /usr/bin/bzip2
uncompresscmd /usr/bin/bunzip2

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

[Important]

The create option pays heed to the modes and ownerships of files specified in /etc/permissions*. If you modify these settings, make sure no conflicts arise.

logrotate is controlled through cron and is called daily by /etc/cron.daily/logrotate. Use /var/lib/logrotate.status to find out when a particular file has been rotated lastly.

4.4. Monitoring Log Files with logwatch

logwatch is a customizable, pluggable log-monitoring script. It parses system logs, extracts the important information and presents them in a human readable manner. To use logwatch, install the logwatch package.

logwatch can either be used at the command-line to generate on-the-fly reports, or via cron to regularly create custom reports. Reports can either be printed on the screen, saved to a file, or be mailed to a specified address. The latter is especially useful when automatically generating reports via cron.

The command-line syntax is easy. You basically tell logwatch for which service, time span and to which detail level to generate a report:

# Detailed report on all kernel messages from yesterday
logwatch --service kernel --detail High --range Yesterday --print

# Low detail report on all sshd events recorded (incl. archived logs)
logwatch --service sshd --detail Low --range All --archives --print

# Mail a report on all smartd messages from May 5th to May 7th to root@localhost
logwatch --service smartd --range 'between 5/5/2005 and 5/7/2005' \
--mailto root@localhost --print

The --range option has got a complex syntax—see logwatch --range help for details. A list of all services that can be queried is available with the following command:

ls /usr/share/logwatch/default.conf/services/ | sed 's/\.conf//g'

logwatch can be customized to great detail. However, the default configuration should be sufficient in most cases. The default configuration files are located under /usr/share/logwatch/default.conf/. Never change them because they would get overwritten again with the next update. Rather place custom configuration in /etc/logwatch/conf/ (you may use the default configuration file as a template, though). A detailed HOWTO on customizing logwatch is available at /usr/share/doc/packages/logwatch/HOWTO-Customize-LogWatch. The following config files exist:

logwatch.conf

The main configuration file. The default version is extensively commented. Each configuration option can be overwritten on the command line.

ignore.conf

Filter for all lines that should globally be ignored by logwatch.

services/*.conf

The service directory holds configuration files for each service you can generate a report for.

logfiles/*.conf

Specifications on which log files should be parsed for each service.

4.5. Using logger to Make System Log Entries

logger is a tool for making entries in the system log. It provides a shell command interface to the syslog(3) system log module. For example, the following line outputs its message in /var/log/messages:

logger -t Test "This messages comes from $USER"

Depending on the current user and hostname, /var/log/messages contains a line similar to this:

Sep 28 13:09:31 venus Test: This messages comes from tux