vmstat collects information about processes, memory, I/O, interrupts and CPU. If called without a sampling rate, it displays average values since the last reboot. When called with a sampling rate, it displays actual samples:

tux@mercury:~> vmstat -a 2
procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu-------
 r  b   swpd   free  inact active   si   so    bi    bo   in   cs us sy  id wa st
 0  0      0 750992 570648 548848    0    0     0     1    8    9  0  0 100  0  0
 0  0      0 750984 570648 548912    0    0     0     0   63   48  1  0 99   0  0
 0  0      0 751000 570648 548912    0    0     0     0   55   47  0  0 100  0  0
 0  0      0 751000 570648 548912    0    0     0     0   56   50  0  0 100  0  0
 0  0      0 751016 570648 548944    0    0     0     0   57   50  0  0 100  0  0

tux@mercury:~> vmstat 2
procs -----------memory----------- ---swap-- -----io---- -system-- -----cpu------
 r  b   swpd   free   buff   cache   si   so    bi    bo   in   cs us sy id wa st
32  1  26236 459640 110240 6312648    0    0  9944     2 4552 6597 95  5  0  0  0
23  1  26236 396728 110336 6136224    0    0  9588     0 4468 6273 94  6  0  0  0
35  0  26236 554920 110508 6166508    0    0  7684 27992 4474 4700 95  5  0  0  0
28  0  26236 518184 110516 6039996    0    0 10830     4 4446 4670 94  6  0  0  0
21  5  26236 716468 110684 6074872    0    0  8734 20534 4512 4061 96  4  0  0  0

The first line of the vmstat output always displays average values since the last reboot.

The columns show the following:

See vmstat --help for more options.

sar can generate extensive reports on almost all important system activities, among them CPU, memory, IRQ usage, IO, or networking. It can either generate reports on the fly or query existing reports gathered by the system activity data collector (sadc). sar and sadc both gather all their data from the /proc file system.

	sysstat Package
sar and sadc are part of sysstat package. You need to install the package either with YaST, or with zypper in sysstat.

sar 2 10                         # on-the-fly report, 10 times every 2 seconds
sar -f ~/reports/sar_2010_05_03  # queries file sar_2010_05_03
sar                              # queries file from today in /var/log/sa/
cd /var/log/sa &&\
sar -f sa01 -f sa02              # queries files /var/log/sa/0[12]

mercury:~ # sar 10 5
Linux 2.6.31.12-0.2-default (mercury) 03/05/10   _x86_64_   (2 CPU)

14:15:43   CPU    %user   %nice   %system   %iowait    %steal     %idle
14:15:53   all    38.55    0.00      6.10      0.10      0.00     55.25
14:16:03   all    12.59    0.00      4.90      0.33      0.00     82.18
14:16:13   all    56.59    0.00      8.16      0.44      0.00     34.81
14:16:23   all    58.45    0.00      3.00      0.00      0.00     38.55
14:16:33   all    86.46    0.00      4.70      0.00      0.00      8.85
Average:   all    49.94    0.00      5.38      0.18      0.00     44.50

mercury:~ # sar -r 10 5
Linux 2.6.31.12-0.2-default (mercury) 03/05/10   _x86_64_   (2 CPU)

16:12:12 kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit
16:12:22    548188   1507488    73.33     20524    64204  2338284   65.10
16:12:32    259320   1796356    87.39     20808    72660  2229080   62.06
16:12:42    381096   1674580    81.46     21084    75460  2328192   64.82
16:12:52    642668   1413008    68.74     21392    81212  1938820   53.98
16:13:02    311984   1743692    84.82     21712    84040  2212024   61.58
Average:    428651   1627025    79.15     21104    75515  2209280   61.51

mercury:~ # sar -B 10 5 
Linux 2.6.31.12-0.2-default (mercury) 03/05/10   _x86_64_   (2 CPU)

16:11:43 pgpgin/s pgpgout/s   fault/s majflt/s  pgfree/s pgscank/s pgscand/s pgsteal/s  %vmeff
16:11:53   225.20    104.00  91993.90     0.00  87572.60      0.00      0.00      0.00    0.00
16:12:03   718.32    601.00  82612.01     2.20  99785.69    560.56    839.24   1132.23   80.89
16:12:13  1222.00   1672.40 103126.00     1.70 106529.00   1136.00    982.40   1172.20   55.33
16:12:23   112.18     77.84 113406.59     0.10  97581.24     35.13    127.74    159.38   97.86
16:12:33   817.22     81.28 121312.91     9.41 111442.44      0.00      0.00      0.00    0.00
Average:   618.72    507.20 102494.86     2.68 100578.98    346.24    389.76    492.60   66.93

mercury:~ # sar -d -p 10 5
Linux 2.6.31.12-0.2-default (neo) 	03/05/10 	_x86_64_	(2 CPU)

16:28:31  DEV    tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz  await  svctm  %util
16:28:41  sdc  11.51     98.50    653.45     65.32      0.10   8.83   4.87   5.61
16:28:41 scd0   0.00      0.00      0.00      0.00      0.00   0.00   0.00   0.00

16:28:41  DEV    tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz  await  svctm  %util
16:28:51  sdc  15.38    329.27    465.93     51.69      0.10   6.39   4.70   7.23
16:28:51 scd0   0.00      0.00      0.00      0.00      0.00   0.00   0.00   0.00

16:28:51  DEV    tps rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz  await  svctm  %util
16:29:01  sdc  32.47   876.72    647.35     46.94      0.33  10.20   3.67  11.91
16:29:01 scd0   0.00     0.00      0.00      0.00      0.00   0.00   0.00   0.00

16:29:01  DEV    tps rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz  await  svctm  %util
16:29:11  sdc  48.75  2852.45    366.77     66.04      0.82  16.93   4.91  23.94
16:29:11 scd0   0.00     0.00      0.00      0.00      0.00   0.00   0.00   0.00

16:29:11  DEV    tps rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz  await  svctm  %util
16:29:21  sdc  13.20   362.40    412.00     58.67      0.16  12.03   6.09   8.04
16:29:21 scd0   0.00     0.00      0.00      0.00      0.00   0.00   0.00   0.00

Average:  DEV    tps rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz  await  svctm  %util
Average:  sdc  24.26   903.52    509.12     58.23      0.30  12.49   4.68  11.34
Average: scd0   0.00     0.00      0.00      0.00      0.00   0.00   0.00   0.00

iostat monitors the system device loading. It generates reports that can be useful for better balancing the load between physical disks attached to your system.

The first iostat report shows statistics collected since the system was booted. Subsequent reports cover the time since the previous report.

tux@mercury:~> iostat
Linux 2.6.32.7-0.2-default (geeko@buildhost) 	02/24/10 	_x86_64_

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0,49    0,01    0,10    0,31    0,00   99,09

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               1,34         5,59        25,37    1459766    6629160
sda1              0,00         0,01         0,00       1519          0
sda2              0,87         5,11        17,83    1335365    4658152
sda3              0,47         0,47         7,54     122578    1971008

When invoked with the -n option, iostat adds statistics of network file systems (NFS) load. The option -x shows extended statistics information.

You can also specify which device should be monitored at what time intervals. For example, iostat -p sda 3 5 will display five reports at three second intervals for device sda.

	sysstat Package
iostat is part of sysstat package. To use it, install the package with zypper in sysstat

The utility mpstat examines activities of each available processor. If your system has one processor only, the global average statistics will be reported.

With the -P option, you can specify the number of processors to be reported (note that 0 is the first processor). The timing arguments work the same way as with the iostat command. Entering mpstat -P 1 2 5 prints five reports for the second processor (number 1) at 2 second intervals.

tux@mercury:~> mpstat -P 1 2 5
Linux 2.6.32.7-0.2-default (geeko@buildhost) 	02/24/10 	_x86_64_

08:57:10  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  \
 %guest   %idle
08:57:12    1    4.46    0.00    5.94    0.50    0.00    0.00    0.00  \
 0.00   89.11
08:57:14    1    1.98    0.00    2.97    0.99    0.00    0.99    0.00  \
 0.00   93.07
08:57:16    1    2.50    0.00    3.00    0.00    0.00    1.00    0.00  \
 0.00   93.50
08:57:18    1   14.36    0.00    1.98    0.00    0.00    0.50    0.00  \
 0.00   83.17
08:57:20    1    2.51    0.00    4.02    0.00    0.00    2.01    0.00  \
 0.00   91.46
Average:    1    5.17    0.00    3.58    0.30    0.00    0.90    0.00  \
 0.00   90.05

If you need to see what load a particular task applies to your system, use pidstat command. It prints activity of every selected task or all tasks managed by Linux kernel if no task is specified. You can also set the number of reports to be displayed and the time interval between them.

For example, pidstat -C top 2 3 prints the load statistic for tasks whose command name includes the string “top”. There will be three reports printed at two second intervals.

tux@mercury:~> pidstat -C top 2 3
Linux 2.6.27.19-5-default (geeko@buildhost) 	03/23/2009 	_x86_64_

09:25:42 AM       PID    %usr %system  %guest    %CPU   CPU  Command
09:25:44 AM     23576   37.62   61.39    0.00   99.01     1  top

09:25:44 AM       PID    %usr %system  %guest    %CPU   CPU  Command
09:25:46 AM     23576   37.00   62.00    0.00   99.00     1  top

09:25:46 AM       PID    %usr %system  %guest    %CPU   CPU  Command
09:25:48 AM     23576   38.00   61.00    0.00   99.00     1  top

Average:          PID    %usr %system  %guest    %CPU   CPU  Command
Average:        23576   37.54   61.46    0.00   99.00     -  top

The Linux kernel keeps certain messages in a ring buffer. To view these messages, enter the command dmesg:

tux@mercury:~> dmesg
[...]
end_request: I/O error, dev fd0, sector 0
subfs: unsuccessful attempt to mount media (256)
e100: eth0: e100_watchdog: link up, 100Mbps, half-duplex
NET: Registered protocol family 17
IA-32 Microcode Update Driver: v1.14 <tigran@veritas.com>
microcode: CPU0 updated from revision 0xe to 0x2e, date = 08112004
IA-32 Microcode Update Driver v1.14 unregistered
bootsplash: status on console 0 changed to on
NET: Registered protocol family 10
Disabled Privacy Extensions on device c0326ea0(lo)
IPv6 over IPv4 tunneling driver
powernow: This module only works with AMD K7 CPUs
bootsplash: status on console 0 changed to on

Older events are logged in the files /var/log/messages and /var/log/warn.

To view a list of all the files open for the process with process ID PID, use -p. For example, to view all the files used by the current shell, enter:

tux@mercury:~> lsof -p $$
COMMAND  PID   USER   FD   TYPE DEVICE  SIZE/OFF NODE NAME
bash    5552 tux  cwd    DIR    3,3    1512 117619 /home/tux
bash    5552 tux  rtd    DIR    3,3     584      2 /
bash    5552 tux  txt    REG    3,3  498816  13047 /bin/bash
bash    5552 tux  mem    REG    0,0              0 [heap] (stat: No such
bash    5552 tux  mem    REG    3,3  217016 115687 /var/run/nscd/passwd
bash    5552 tux  mem    REG    3,3  208464  11867 /usr/lib/locale/en_GB.
[...]
bash    5552 tux  mem    REG    3,3     366   9720 /usr/lib/locale/en_GB.
bash    5552 tux  mem    REG    3,3   97165   8828 /lib/ld-2.3.6.so
bash    5552 tux    0u   CHR  136,5              7 /dev/pts/5
bash    5552 tux    1u   CHR  136,5              7 /dev/pts/5
bash    5552 tux    2u   CHR  136,5              7 /dev/pts/5
bash    5552 tux  255u   CHR  136,5              7 /dev/pts/5

The special shell variable $$, whose value is the process ID of the shell, has been used.

The command lsof lists all the files currently open when used without any parameters. There are often thousands of open files, therefore, listing all of them is rarely useful. However, the list of all files can be combined with search functions to generate useful lists. For example, list all used character devices:

tux@mercury:~> lsof | grep CHR
bash      3838     tux    0u      CHR  136,0                 2 /dev/pts/0
bash      3838     tux    1u      CHR  136,0                 2 /dev/pts/0
bash      3838     tux    2u      CHR  136,0                 2 /dev/pts/0
bash      3838     tux  255u      CHR  136,0                 2 /dev/pts/0
bash      5552     tux    0u      CHR  136,5                 7 /dev/pts/5
bash      5552     tux    1u      CHR  136,5                 7 /dev/pts/5
bash      5552     tux    2u      CHR  136,5                 7 /dev/pts/5
bash      5552     tux  255u      CHR  136,5                 7 /dev/pts/5
X         5646       root  mem       CHR    1,1              1006 /dev/mem
lsof      5673     tux    0u      CHR  136,5                 7 /dev/pts/5
lsof      5673     tux    2u      CHR  136,5                 7 /dev/pts/5
grep      5674     tux    1u      CHR  136,5                 7 /dev/pts/5
grep      5674     tux    2u      CHR  136,5                 7 /dev/pts/5

When used with -i, lsof lists currently open Internet files as well:

tux@mercury:~> lsof -i
[...]
pidgin     4349 tux   17r  IPv4  15194      0t0  TCP \
 jupiter.example.com:58542->www.example.net:https (ESTABLISHED)
pidgin     4349 tux   21u  IPv4  15583      0t0  TCP \
 jupiter.example.com:37051->aol.example.org:aol (ESTABLISHED)
evolution  4578 tux   38u  IPv4  16102      0t0  TCP \
 jupiter.example.com:57419->imap.example.com:imaps (ESTABLISHED)
npviewer.  9425 tux   40u  IPv4  24769      0t0  TCP \
 jupiter.example.com:51416->www.example.com:http (CLOSE_WAIT)
npviewer.  9425 tux   49u  IPv4  24814      0t0  TCP \
 jupiter.example.com:43964->www.example.org:http (CLOSE_WAIT)
ssh       17394 tux    3u  IPv4  40654      0t0  TCP \
 jupiter.example.com:35454->saturn.example.com:ssh (ESTABLISHED)

udevadm monitor listens to the kernel uevents and events sent out by a udev rule and prints the device path (DEVPATH) of the event to the console. This is a sequence of events while connecting a USB memory stick:

	Monitoring udev Events
Only root user is allowed to monitor udev events by running the udevadm command.

UEVENT[1138806687] add@/devices/pci0000:00/0000:00:1d.7/usb4/4-2/4-2.2
UEVENT[1138806687] add@/devices/pci0000:00/0000:00:1d.7/usb4/4-2/4-2.2/4-2.2
UEVENT[1138806687] add@/class/scsi_host/host4
UEVENT[1138806687] add@/class/usb_device/usbdev4.10
UDEV  [1138806687] add@/devices/pci0000:00/0000:00:1d.7/usb4/4-2/4-2.2
UDEV  [1138806687] add@/devices/pci0000:00/0000:00:1d.7/usb4/4-2/4-2.2/4-2.2
UDEV  [1138806687] add@/class/scsi_host/host4
UDEV  [1138806687] add@/class/usb_device/usbdev4.10
UEVENT[1138806692] add@/devices/pci0000:00/0000:00:1d.7/usb4/4-2/4-2.2/4-2.2
UEVENT[1138806692] add@/block/sdb
UEVENT[1138806692] add@/class/scsi_generic/sg1
UEVENT[1138806692] add@/class/scsi_device/4:0:0:0
UDEV  [1138806693] add@/devices/pci0000:00/0000:00:1d.7/usb4/4-2/4-2.2/4-2.2
UDEV  [1138806693] add@/class/scsi_generic/sg1
UDEV  [1138806693] add@/class/scsi_device/4:0:0:0
UDEV  [1138806693] add@/block/sdb
UEVENT[1138806694] add@/block/sdb/sdb1
UDEV  [1138806694] add@/block/sdb/sdb1
UEVENT[1138806694] mount@/block/sdb/sdb1
UEVENT[1138806697] umount@/block/sdb/sdb1

The Linux audit framework is a complex auditing system that collects detailed information about all security related events. These records can be consequently analyzed to discover if, for example, a violation of security policies occurred.

xrestop provides statistics for each connected X11 client's server-side resource. The output is very similar to Section 2.3.4, “Table of Processes: top”.

xrestop - Display: localhost:0
          Monitoring 40 clients. XErrors: 0
          Pixmaps:   42013K total, Other:     206K total, All:   42219K total

res-base Wins  GCs Fnts Pxms Misc   Pxm mem  Other   Total   PID Identifier
3e00000   385   36    1  751  107    18161K     13K  18175K   ?   NOVELL: SU
4600000   391  122    1 1182  889     4566K     33K   4600K   ?   amaroK - S
1600000    35   11    0   76  142     3811K      4K   3816K   ?   KDE Deskto
3400000    52   31    1   69   74     2816K      4K   2820K   ?   Linux Shel
2c00000    50   25    1   43   50     2374K      3K   2378K   ?   Linux Shel
2e00000    50   10    1   36   42     2341K      3K   2344K   ?   Linux Shel
2600000    37   24    1   34   50     1772K      3K   1775K   ?   Root - Kon
4800000    37   24    1   34   49     1772K      3K   1775K   ?   Root - Kon
2a00000   209   33    1  323  238     1111K     12K   1123K   ?   Trekstor25
1800000   182   32    1  302  285     1039K     12K   1052K   ?   kicker
1400000   157  121    1  231  477      777K     18K    796K   ?   kwin
3c00000   175   36    1  248  168      510K      9K    520K   ?   de.comp.la
3a00000   326   42    1  579  444      486K     20K    506K   ?   [opensuse-
0a00000    85   38    1  317  224      102K      9K    111K   ?   Kopete
4e00000    25   17    1   60   66       63K      3K     66K   ?   YaST Contr
2400000    11   10    0   56   51       53K      1K     55K 22061 suseplugge
0e00000    20   12    1   50   92       50K      3K     54K 22016 kded
3200000     6   41    5   72   84       40K      8K     48K   ?   EMACS
2200000    54    9    1   30   31       42K      3K     45K   ?   SUSEWatche
4400000     2   11    1   30   34       34K      2K     36K 16489 kdesu
1a00000   255    7    0   42   11       19K      6K     26K   ?   KMix
3800000     2   14    1   34   37       21K      2K     24K 22242 knotify
1e00000    10    7    0   42    9       15K    624B     15K   ?   KPowersave
3600000   106    6    1   30    9        7K      3K     11K 22236 konqueror
2000000    10    5    0   21   34        9K      1K     10K   ?   klipper
3000000    21    7    0   11    9        7K    888B      8K   ?   KDE Wallet

The command ipcs produces a list of the IPC resources currently in use:

------ Shared Memory Segments --------
key        shmid      owner     perms      bytes      nattch     status
0x00000000 58261504   tux    600        393216     2          dest
0x00000000 58294273   tux    600        196608     2          dest
0x00000000 83886083   tux    666        43264      2
0x00000000 83951622   tux    666        192000     2
0x00000000 83984391   tux    666        282464     2
0x00000000 84738056   root      644        151552     2          dest

------ Semaphore Arrays --------
key        semid      owner     perms      nsems
0x4d038abf 0          tux    600        8

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages

The command ps produces a list of processes. Most parameters must be written without a minus sign. Refer to ps --help for a brief help or to the man page for extensive help.

To list all processes with user and command line information, use ps axu:

tux@mercury:~> ps axu
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0    696   272 ?        S    12:59   0:01 init [5]
root         2  0.0  0.0      0     0 ?        SN   12:59   0:00 [ksoftirqd
root         3  0.0  0.0      0     0 ?        S<   12:59   0:00 [events
[...]
tux    4047  0.0  6.0 158548 31400 ?        Ssl  13:02   0:06 mono-best
tux    4057  0.0  0.7   9036  3684 ?        Sl   13:02   0:00 /opt/gnome
tux    4067  0.0  0.1   2204   636 ?        S    13:02   0:00 /opt/gnome
tux    4072  0.0  1.0  15996  5160 ?        Ss   13:02   0:00 gnome-scre
tux    4114  0.0  3.7 130988 19172 ?        SLl  13:06   0:04 sound-juic
tux    4818  0.0  0.3   4192  1812 pts/0    Ss   15:59   0:00 -bash
tux    4959  0.0  0.1   2324   816 pts/0    R+   16:17   0:00 ps axu

To check how many sshd processes are running, use the option -p together with the command pidof, which lists the process IDs of the given processes.

tux@mercury:~> ps -p $(pidof sshd)
  PID TTY      STAT   TIME COMMAND
 3524 ?        Ss     0:00 /usr/sbin/sshd -o PidFile=/var/run/sshd.init.pid
 4813 ?        Ss     0:00 sshd: tux [priv]
 4817 ?        R      0:00 sshd: tux@pts/0

The process list can be formatted according to your needs. The option -L returns a list of all keywords. Enter the following command to issue a list of all processes sorted by memory usage:

tux@mercury:~> ps ax --format pid,rss,cmd --sort rss
  PID   RSS CMD
    2     0 [ksoftirqd/0]
    3     0 [events/0]
    4     0 [khelper]
    5     0 [kthread]
   11     0 [kblockd/0]
   12     0 [kacpid]
  472     0 [pdflush]
  473     0 [pdflush]
[...]
 4028 17556 nautilus --no-default-window --sm-client-id default2
 4118 17800 ksnapshot
 4114 19172 sound-juicer
 4023 25144 gnome-panel --sm-client-id default1
 4047 31400 mono-best --debug /usr/lib/beagle/Best.exe --autostarted
 3973 31520 mono-beagled --debug /usr/lib/beagle/BeagleDaemon.exe --bg --aut

The command pstree produces a list of processes in the form of a tree:

tux@mercury:~> pstree
init-+-NetworkManagerD
     |-acpid
     |-3*[automount]
     |-cron
     |-cupsd
     |-2*[dbus-daemon]
     |-dbus-launch
     |-dcopserver
     |-dhcpcd
     |-events/0
     |-gpg-agent
     |-hald-+-hald-addon-acpi
     |      `-hald-addon-stor
     |-kded
     |-kdeinit-+-kdesu---su---kdesu_stub---yast2---y2controlcenter
     |         |-kio_file
     |         |-klauncher
     |         |-konqueror
     |         |-konsole-+-bash---su---bash
     |         |         `-bash
     |         `-kwin
     |-kdesktop---kdesktop_lock---xmatrix
     |-kdesud
     |-kdm-+-X
     |     `-kdm---startkde---kwrapper
[...]

The parameter -p adds the process ID to a given name. To have the command lines displayed as well, use the -a parameter:

The command top, which stands for table of processes, displays a list of processes that is refreshed every two seconds. To terminate the program, press Q. The parameter -n 1 terminates the program after a single display of the process list. The following is an example output of the command top -n 1:

tux@mercury:~> top -n 1
top - 17:06:28 up  2:10,  5 users,  load average: 0.00, 0.00, 0.00
Tasks:  85 total,   1 running,  83 sleeping,   1 stopped,   0 zombie
Cpu(s):  5.5% us,  0.8% sy,  0.8% ni, 91.9% id,  1.0% wa,  0.0% hi,  0.0% si
Mem:    515584k total,   506468k used,     9116k free,    66324k buffers
Swap:   658656k total,        0k used,   658656k free,   353328k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
    1 root      16   0   700  272  236 S  0.0  0.1   0:01.33 init
    2 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/0
    3 root      10  -5     0    0    0 S  0.0  0.0   0:00.27 events/0
    4 root      10  -5     0    0    0 S  0.0  0.0   0:00.01 khelper
    5 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kthread
   11 root      10  -5     0    0    0 S  0.0  0.0   0:00.05 kblockd/0
   12 root      20  -5     0    0    0 S  0.0  0.0   0:00.00 kacpid
  472 root      20   0     0    0    0 S  0.0  0.0   0:00.00 pdflush
  473 root      15   0     0    0    0 S  0.0  0.0   0:00.06 pdflush
  475 root      11  -5     0    0    0 S  0.0  0.0   0:00.00 aio/0
  474 root      15   0     0    0    0 S  0.0  0.0   0:00.07 kswapd0
  681 root      10  -5     0    0    0 S  0.0  0.0   0:00.01 kseriod
  839 root      10  -5     0    0    0 S  0.0  0.0   0:00.02 reiserfs/0
  923 root      13  -4  1712  552  344 S  0.0  0.1   0:00.67 udevd
 1343 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 khubd
 1587 root      20   0     0    0    0 S  0.0  0.0   0:00.00 shpchpd_event
 1746 root      15   0     0    0    0 S  0.0  0.0   0:00.00 w1_control
 1752 root      15   0     0    0    0 S  0.0  0.0   0:00.00 w1_bus_master1
 2151 root      16   0  1464  496  416 S  0.0  0.1   0:00.00 acpid
 2165 messageb  16   0  3340 1048  792 S  0.0  0.2   0:00.64 dbus-daemon
 2166 root      15   0  1840  752  556 S  0.0  0.1   0:00.01 syslog-ng
 2171 root      16   0  1600  516  320 S  0.0  0.1   0:00.00 klogd
 2235 root      15   0  1736  800  652 S  0.0  0.2   0:00.10 resmgrd
 2289 root      16   0  4192 2852 1444 S  0.0  0.6   0:02.05 hald
 2403 root      23   0  1756  600  524 S  0.0  0.1   0:00.00 hald-addon-acpi
 2709 root      19   0  2668 1076  944 S  0.0  0.2   0:00.00 NetworkManagerD
 2714 root      16   0  1756  648  564 S  0.0  0.1   0:00.56 hald-addon-stor

By default the output is sorted by CPU usage (column %CPU, shortcut Shift+P). Use following shortcuts to change the sort field:

To use any other field for sorting, press F and select a field from the list. To toggle the sort order, Use Shift+R.

The parameter -U UID monitors only the processes associated with a particular user. Replace UID with the user ID of the user. Use top -U $(id -u) to show processes of the current user

The kernel determines which processes require more CPU time than others by the process' nice level, also called niceness. The higher the “nice” level of a process is, the less CPU time it will take from other processes. Nice levels range from -20 (the least “nice” level) to 19. Negative values can only be set by root.

Adjusting the niceness level is useful when running a non time-critical process that lasts long and uses large amounts of CPU time, such as compiling a kernel on a system that also performs other tasks. Making such a process “nicer”, ensures that the other tasks, for example a Web server, will have a higher priority.

Calling nice without any parameters prints the current niceness:

tux@mercury:~> nice
0

Running nice command increments the current nice level for the given command by 10. Using nice -n level command lets you specify a new niceness relative to the current one.

To change the niceness of a running process, use renice priority -p process id, for example:

renice +5 3266

To renice all processes owned by a specific user, use the option -u user. Process groups are reniced by the option -g process group id.

The utility free examines RAM and swap usage. Details of both free and used memory and swap areas are shown:

tux@mercury:~> free
            total       used       free     shared    buffers     cached
Mem:       2062844    2047444      15400          0     129580     921936
-/+ buffers/cache:     995928    1066916
Swap:      2104472          0    2104472

The options -b, -k, -m, -g show the output in bytes, KB, MB, or GB, respectively. The parameter -d delay ensures that the display is refreshed every delay seconds. For example, free -d 1.5 produces an update every 1.5 seconds.

Use /proc/meminfo to get more detailed information on memory usage than with free. Actually free uses some of the data from this file. See an example output from a 64bit system below. Note that it slightly differs on 32bit systems due to different memory management):

tux@mercury:~> cat /proc/meminfo
MemTotal:        8182956 kB
MemFree:         1045744 kB
Buffers:          364364 kB
Cached:          5601388 kB
SwapCached:         1936 kB
Active:          4048268 kB
Inactive:        2674796 kB
Active(anon):     663088 kB
Inactive(anon):   107108 kB
Active(file):    3385180 kB
Inactive(file):  2567688 kB
Unevictable:           4 kB
Mlocked:               4 kB
SwapTotal:       2096440 kB
SwapFree:        2076692 kB
Dirty:                44 kB
Writeback:             0 kB
AnonPages:        756108 kB
Mapped:           147320 kB
Slab:             329216 kB
SReclaimable:     300220 kB
SUnreclaim:        28996 kB
PageTables:        21092 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     6187916 kB
Committed_AS:    1388160 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      133384 kB
VmallocChunk:   34359570939 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:     2689024 kB
DirectMap2M:     5691392 kB

The most important entries are:

Exactly determining how much memory a certain process is consuming is not possible with standard tools like top or ps. Use the smaps subsystem, introduced in Kernel 2.6.14, if you need exact data. It can be found at /proc/pid/smaps and shows you the number of clean and dirty memory pages the process with the ID PID is using at that time. It differentiates between shared and private memory, so you are able to see how much memory the process is using without including memory shared with other processes.

netstat shows network connections, routing tables (-r), interfaces (-i), masquerade connections (-M), multicast memberships (-g), and statistics (-s).

tux@mercury:~> netstat -r
Kernel IP routing table
Destination     Gateway         Genmask        Flags  MSS Window  irtt Iface
192.168.2.0     *               255.255.254.0  U        0 0          0 eth0
link-local      *               255.255.0.0    U        0 0          0 eth0
loopback        *               255.0.0.0      U        0 0          0 lo
default         192.168.2.254   0.0.0.0        UG       0 0          0 eth0

tux@mercury:~> netstat -i
Kernel Interface table
Iface   MTU Met   RX-OK RX-ERR RX-DRP RX-OVR  TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0   1500   0 1624507 129056      0      0   7055      0      0      0 BMNRU
lo    16436   0   23728      0      0      0  23728      0      0      0 LRU

When displaying network connections or statistics, you can specify the socket type to display: TCP (-t), UDP (-u), or raw (-r). The -p option shows the PID and name of the program to which each socket belongs.

The following example lists all TCP connections and the programs using these connections.

mercury:~ # netstat -t -p
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address  Foreign Address       State       PID/Pro
[...]
tcp      0      0 mercury:33513    www.novell.com:www-http ESTABLISHED 6862/fi
tcp      0    352 mercury:ssh      mercury2.:trc-netpoll     ESTABLISHED 19422/s
tcp      0      0 localhost:ssh  localhost:17828         ESTABLISHED -

In the following, statistics for the TCP protocol are displayed:

tux@mercury:~> netstat -s -t
Tcp:
    2427 active connections openings
    2374 passive connection openings
    0 failed connection attempts
    0 connection resets received
    1 connections established
    27476 segments received
    26786 segments send out
    54 segments retransmited
    0 bad segments received.
    6 resets sent
[...]
    TCPAbortOnLinger: 0
    TCPAbortFailed: 0
    TCPMemoryPressures: 0

The iptraf utility is a menu based Local Area Network (LAN) monitor. It generates network statistics, including TCP and UDP counts, Ethernet load information, IP checksum errors and others.

If you enter the command without any option, it runs in an interactive mode. You can navigate through graphical menus and choose the statistics that you want iptraf to report. You can also specify which network interface to examine.

Figure 2.1. iptraf Running in Interactive Mode¶

The command iptraf understands several options and can be run in a batch mode as well. The following example will collect statistics for network interface eth0 (-i) for 1 minute (-t). It will be run in the background (-B) and the statistics will be written to the iptraf.log file in your home directory (-L).

tux@mercury:~> iptraf -i eth0 -t 1 -B -L ~/iptraf.log

You can examine the log file with the more command:

tux@mercury:~> more ~/iptraf.log
Mon Mar 23 10:08:02 2010; ******** IP traffic monitor started ********
Mon Mar 23 10:08:02 2010; UDP; eth0; 107 bytes; from 192.168.1.192:33157 to \
 239.255.255.253:427
Mon Mar 23 10:08:02 2010; VRRP; eth0; 46 bytes; from 192.168.1.252 to \
 224.0.0.18
Mon Mar 23 10:08:03 2010; VRRP; eth0; 46 bytes; from 192.168.1.252 to \
 224.0.0.18
Mon Mar 23 10:08:03 2010; VRRP; eth0; 46 bytes; from 192.168.1.252 to \
 224.0.0.18
[...]
Mon Mar 23 10:08:06 2010; UDP; eth0; 132 bytes; from 192.168.1.54:54395 to \
 10.20.7.255:111
Mon Mar 23 10:08:06 2010; UDP; eth0; 46 bytes; from 192.168.1.92:27258 to \
 10.20.7.255:8765
Mon Mar 23 10:08:06 2010; UDP; eth0; 124 bytes; from 192.168.1.139:43464 to \
 10.20.7.255:111
Mon Mar 23 10:08:06 2010; VRRP; eth0; 46 bytes; from 192.168.1.252 to \
 224.0.0.18
--More--(7%)

Important information from the /proc file system is summarized by the command procinfo:

tux@mercury:~> procinfo
Linux 2.6.32.7-0.2-default (geeko@buildhost) (gcc 4.3.4) #1 2CPU

Memory:      Total        Used        Free      Shared     Buffers
Mem:       2060604     2011264       49340           0      200664
Swap:      2104472         112     2104360

Bootup: Wed Feb 17 03:39:33 2010    Load average: 0.86 1.10 1.11 3/118 21547

user  :       2:43:13.78   0.8%  page in :   71099181  disk 1:  2827023r 968
nice  :   1d 22:21:27.87  14.7%  page out:  690734737
system:      13:39:57.57   4.3%  page act:  138388345
IOwait:      18:02:18.59   5.7%  page dea:   29639529
hw irq:       0:03:39.44   0.0%  page flt: 9539791626
sw irq:       1:15:35.25   0.4%  swap in :         69
idle  :   9d 16:07:56.79  73.8%  swap out:        209
uptime:   6d 13:07:11.14         context :  542720687

irq  0: 141399308 timer          irq 14:   5074312 ide0
irq  1:     73784 i8042          irq 50:   1938076 uhci_hcd:usb1, ehci_
irq  4:         2                irq 58:         0 uhci_hcd:usb2
irq  6:         5 floppy [2]     irq 66:    872711 uhci_hcd:usb3, HDA I
irq  7:         2                irq 74:        15 uhci_hcd:usb4
irq  8:         0 rtc            irq 82: 178717720 0         PCI-MSI  e
irq  9:         0 acpi           irq169:  44352794 nvidia
irq 12:         3                irq233:   8209068 0         PCI-MSI  l

To see all the information, use the parameter -a. The parameter -nN produces updates of the information every N seconds. In this case, terminate the program by pressing q.

By default, the cumulative values are displayed. The parameter -d produces the differential values. procinfo -dn5 displays the values that have changed in the last five seconds:

	Accessing PCI configuration.
Most operating systems require root user privileges to grant access to the computer's PCI configuration.

The command lspci lists the PCI resources:

mercury:~ # lspci
00:00.0 Host bridge: Intel Corporation 82845G/GL[Brookdale-G]/GE/PE \
    DRAM Controller/Host-Hub Interface (rev 01)
00:01.0 PCI bridge: Intel Corporation 82845G/GL[Brookdale-G]/GE/PE \
    Host-to-AGP Bridge (rev 01)
00:1d.0 USB Controller: Intel Corporation 82801DB/DBL/DBM \
    (ICH4/ICH4-L/ICH4-M) USB UHCI Controller #1 (rev 01)
00:1d.1 USB Controller: Intel Corporation 82801DB/DBL/DBM \
    (ICH4/ICH4-L/ICH4-M) USB UHCI Controller #2 (rev 01)
00:1d.2 USB Controller: Intel Corporation 82801DB/DBL/DBM \
    (ICH4/ICH4-L/ICH4-M) USB UHCI Controller #3 (rev 01)
00:1d.7 USB Controller: Intel Corporation 82801DB/DBM \
    (ICH4/ICH4-M) USB2 EHCI Controller (rev 01)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 81)
00:1f.0 ISA bridge: Intel Corporation 82801DB/DBL (ICH4/ICH4-L) \
    LPC Interface Bridge (rev 01)
00:1f.1 IDE interface: Intel Corporation 82801DB (ICH4) IDE \
    Controller (rev 01)
00:1f.3 SMBus: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) \
    SMBus Controller (rev 01)
00:1f.5 Multimedia audio controller: Intel Corporation 82801DB/DBL/DBM \
    (ICH4/ICH4-L/ICH4-M) AC'97 Audio Controller (rev 01)
01:00.0 VGA compatible controller: Matrox Graphics, Inc. G400/G450 (rev 85)
02:08.0 Ethernet controller: Intel Corporation 82801DB PRO/100 VE (LOM) \
    Ethernet Controller (rev 81)

Using -v results in a more detailed listing:

mercury:~ # lspci -v
[...]
00:03.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet \
Controller (rev 02)
  Subsystem: Intel Corporation PRO/1000 MT Desktop Adapter
  Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 19
  Memory at f0000000 (32-bit, non-prefetchable) [size=128K]
  I/O ports at d010 [size=8]
  Capabilities: [dc] Power Management version 2
  Capabilities: [e4] PCI-X non-bridge device
  Kernel driver in use: e1000
  Kernel modules: e1000

Information about device name resolution is obtained from the file /usr/share/pci.ids. PCI IDs not listed in this file are marked “Unknown device.”

The parameter -vv produces all the information that could be queried by the program. To view the pure numeric values, use the parameter -n.

The command lsusb lists all USB devices. With the option -v, print a more detailed list. The detailed information is read from the directory /proc/bus/usb/. The following is the output of lsusb with these USB devices attached: hub, memory stick, hard disk and mouse.

mercury:/ # lsusb
Bus 004 Device 007: ID 0ea0:2168 Ours Technology, Inc. Transcend JetFlash \
    2.0 / Astone USB Drive
Bus 004 Device 006: ID 04b4:6830 Cypress Semiconductor Corp. USB-2.0 IDE \
    Adapter
Bus 004 Device 005: ID 05e3:0605 Genesys Logic, Inc.
Bus 004 Device 001: ID 0000:0000
Bus 003 Device 001: ID 0000:0000
Bus 002 Device 001: ID 0000:0000
Bus 001 Device 005: ID 046d:c012 Logitech, Inc. Optical Mouse
Bus 001 Device 001: ID 0000:0000

The command file determines the type of a file or a list of files by checking /usr/share/misc/magic.

tux@mercury:~> file /usr/bin/file
/usr/bin/file: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), \
    for GNU/Linux 2.6.4, dynamically linked (uses shared libs), stripped

The parameter -f list specifies a file with a list of filenames to examine. The -z allows file to look inside compressed files:

tux@mercury:~> file /usr/share/man/man1/file.1.gz
/usr/share/man/man1/file.1.gz: gzip compressed data, from Unix, max compression
tux@mercury:~> file -z /usr/share/man/man1/file.1.gz
/usr/share/man/man1/file.1.gz: troff or preprocessor input text \
    (gzip compressed data, from Unix, max compression)

The parameter -i outputs a mime type string rather than the traditional description.

tux@mercury:~> file -i /usr/share/misc/magic
/usr/share/misc/magic: text/plain charset=utf-8

The command mount shows which file system (device and type) is mounted at which mount point:

tux@mercury:~> mount
/dev/sda2 on / type ext4 (rw,acl,user_xattr)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
debugfs on /sys/kernel/debug type debugfs (rw)
devtmpfs on /dev type devtmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,mode=1777)
devpts on /dev/pts type devpts (rw,mode=0620,gid=5)
/dev/sda3 on /home type ext3 (rw)
securityfs on /sys/kernel/security type securityfs (rw)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
gvfs-fuse-daemon on /home/tux/.gvfs type fuse.gvfs-fuse-daemon \
(rw,nosuid,nodev,user=tux)

Obtain information about total usage of the file systems with the command df. The parameter -h (or --human-readable) transforms the output into a form understandable for common users.

tux@mercury:~> df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2              20G  5,9G   13G  32% /
devtmpfs              1,6G  236K  1,6G   1% /dev
tmpfs                 1,6G  668K  1,6G   1% /dev/shm
/dev/sda3             208G   40G  159G  20% /home

Display the total size of all the files in a given directory and its subdirectories with the command du. The parameter -s suppresses the output of detailed information and gives only a total for each argument. -h again transforms the output into a human-readable form:

tux@mercury:~> du -sh /opt
192M    /opt

Read the content of binaries with the readelf utility. This even works with ELF files that were built for other hardware architectures:

tux@mercury:~> readelf --file-header /bin/ls
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF64
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           Advanced Micro Devices X86-64
  Version:                           0x1
  Entry point address:               0x402540
  Start of program headers:          64 (bytes into file)
  Start of section headers:          95720 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         9
  Size of section headers:           64 (bytes)
  Number of section headers:         32
  Section header string table index: 31

The command stat displays file properties:

tux@mercury:~> stat /etc/profile
  File: `/etc/profile'
  Size: 9662      	Blocks: 24         IO Block: 4096   regular file
Device: 802h/2050d	Inode: 132349      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2009-03-20 07:51:17.000000000 +0100
Modify: 2009-01-08 19:21:14.000000000 +0100
Change: 2009-03-18 12:55:31.000000000 +0100

The parameter --file-system produces details of the properties of the file system in which the specified file is located:

tux@mercury:~> stat /etc/profile --file-system
  File: "/etc/profile"
    ID: d4fb76e70b4d1746 Namelen: 255     Type: ext2/ext3
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 2581445    Free: 1717327    Available: 1586197
Inodes: Total: 655776     Free: 490312

It can be useful to determine what processes or users are currently accessing certain files. Suppose, for example, you want to unmount a file system mounted at /mnt. umount returns "device is busy." The command fuser can then be used to determine what processes are accessing the device:

tux@mercury:~> fuser -v /mnt/*

                     USER        PID ACCESS COMMAND
/mnt/notes.txt       tux    26597 f....  less

Following termination of the less process, which was running on another terminal, the file system can successfully be unmounted. When used with -k option, fuser will kill processes accessing the file as well.

With the command w, find out who is logged onto the system and what each user is doing. For example:

tux@mercury:~> w
 14:58:43 up 1 day,  1:21,  2 users,  load average: 0.00, 0.00, 0.00
USER     TTY        LOGIN@   IDLE   JCPU   PCPU WHAT
tux      :0        12:25   ?xdm?   1:23   0.12s /bin/sh /usr/bin/startkde
root     pts/4     14:13    0.00s  0.06s  0.00s w

If any users of other systems have logged in remotely, the parameter -f shows the computers from which they have established the connection.

Determine the time spent by commands with the time utility. This utility is available in two versions: as a shell built-in and as a program (/usr/bin/time).

tux@mercury:~> time find . > /dev/null

real    0m4.051s
user    0m0.042s
sys     0m0.205s

RRDtool is a shortcut of Round Robin Database tool. Round Robin is a method for manipulating with a constant amount of data. It uses the principle of a circular buffer, where there is no end nor beginning to the data row which is being read. RRDtool uses Round Robin Databases to store and read its data.

As mentioned above, RRDtool is designed to work with data that change in time. The ideal case is a sensor which repeatedly reads measured data (like temperature, speed etc.) in constant periods of time, and then exports them in a given format. Such data are perfectly ready for RRDtool, and it is easy to process them and create the desired output.

Sometimes it is not possible to obtain the data automatically and regularly. Their format needs to be pre-processed before it is supplied to RRDtool, and often you need to manipulate RRDtool even manually.

The following is a simple example of basic RRDtool usage. It illustrates all three important phases of the usual RRDtool workflow: creating a database, updating measured values, and viewing the output.

Suppose we want to collect and view information about the memory usage in the Linux system as it changes in time. To make the example more vivid, we measure the currently free memory for the period of 40 seconds in 4-second intervals. During the measuring, the three hungry applications that usually consume a lot of system memory have been started and closed: the Firefox Web browser, the Evolution e-mail client, and the Eclipse development framework.

tux@mercury:~> cat free_mem.sh
INTERVAL=4
for steps in {1..10}
do
    DATE=`date +%s`
    FREEMEM=`free -b | grep "Mem" | awk '{ print $4 }'`
    sleep $INTERVAL
    echo "rrdtool update free_mem.rrd $DATE:$FREEMEM"
done

tux@mercury:~> sh free_mem.sh
rrdtool update free_mem.rrd 1272974835:1182994432
rrdtool update free_mem.rrd 1272974839:1162817536
rrdtool update free_mem.rrd 1272974843:1096269824
rrdtool update free_mem.rrd 1272974847:1034219520
rrdtool update free_mem.rrd 1272974851:909438976
rrdtool update free_mem.rrd 1272974855:832454656
rrdtool update free_mem.rrd 1272974859:829120512
rrdtool update free_mem.rrd 1272974863:1180377088
rrdtool update free_mem.rrd 1272974867:1179369472
rrdtool update free_mem.rrd 1272974871:1181806592

rrdtool create free_mem.rrd --start 1272974834 --step=4 \
DS:memory:GAUGE:600:U:U RRA:AVERAGE:0.5:1:24

tux@mercury:~> ls -l free_mem.rrd
-rw-r--r-- 1 tux users 776 May  5 12:50 free_mem.rrd

tux@mercury:~> sh free_mem_updates.log; ls -l free_mem.rrd
-rw-r--r--  1 tux users  776 May  5 13:29 free_mem.rrd

2.11.2.4. Viewing Measured Values¶

We have already measured the values, created the database, and stored the measured value in it. Now we can play with the database, and retrieve or view its values.

To retrieve all the values from our database, enter the following on the command line:

tux@mercury:~> rrdtool fetch free_mem.rrd AVERAGE --start 1272974830 \
--end 1272974871
          memory
1272974832: nan
1272974836: 1.1729059840e+09
1272974840: 1.1461806080e+09
1272974844: 1.0807572480e+09
1272974848: 1.0030243840e+09
1272974852: 8.9019289600e+08
1272974856: 8.3162112000e+08
1272974860: 9.1693465600e+08
1272974864: 1.1801251840e+09
1272974868: 1.1799787520e+09
1272974872: nan

Points to Notice

• AVERAGE will fetch average value points from the database, because only one data source is defined (Section 2.11.2.2, “Creating Database”) with AVERAGE processing and no other function is available.

• The first line of the output prints the name of the data source as defined in Section 2.11.2.2, “Creating Database”.

• The left results column represents individual points in time, while the right one represents corresponding measured average values in scientific notation.

• The nan in the last line stands for “not a number”.

Now a graph representing representing the values stored in the database is drawn:

tux@mercury:~> rrdtool graph free_mem.png \
--start 1272974830 \
--end 1272974871 \
--step=4 \
DEF:free_memory=free_mem.rrd:memory:AVERAGE \
LINE2:free_memory#FF0000 \
--vertical-label "GB" \
--title "Free System Memory in Time" \
--zoom 1.5 \
--x-grid SECOND:1:SECOND:4:SECOND:10:0:%X

Points to Notice

• free_mem.png is the file name of the graph to be created.

• --start and --end limit the time range within which the graph will be drawn.

• --step specifies the time resolution (in seconds) of the graph.

• The DEF:... part is a data definition called free_memory. Its data are read from the free_mem.rrd database and its data source called memory. The average value points are calculated, because no others were defined in Section 2.11.2.2, “Creating Database”.

• The LINE... part specifies properties of the line to be drawn into the graph. It is 2 pixels wide, its data come from the free_memory definition, and its color is red.

• --vertical-label sets the label to be printed along the y axis, and --title sets the main label for the whole graph.

• --zoom specifies the zoom factor for the graph. This value must be greater than zero.

• --x-grid specifies how to draw grid lines and their labels into the graph. Our example places them every second, while major grid lines are placed every 4 seconds. Labels are placed every 10 seconds under the major grid lines.

Figure 2.2. Example Graph Created with RRDtool¶

RRDtool is a very complex tool with a lot of sub-commands and command line options. Some of them are easy to understand, but you have to really study RRDtool to make it produce the results you want and fine-tune them according to your liking.

Apart form RRDtool's man page (man 1 rrdtool) which gives you only basic information, you should have a look at the RRDtool homepage. There is a detailed documentation of the rrdtool command and all its sub-commands. There are also several tutorials to help you understand the common RRDtool workflow.

If you are interested in monitoring network traffic, have a look at MRTG. It stands for Multi Router Traffic Grapher and can graph the activity of all sorts of network devices. It can easily make use of RRDtool.

In addition to those configuration files Nagios comes with very flexible and highly customizable configuration files called Object Definition configuration files. Those configuration files are very important since they define the following objects:

The flexibility lies in the fact that objects are easily enhanceable. Imagine you are responsible for a host with only one service running. However, you want to install another service on the same host machine and you want to monitor that service as well. It is possible to add another service object and assign it to the host object without huge efforts.

Right after the installation, Nagios offers default templates for object definition configuration files. They can be found at /etc/nagios/objects. In the following see a description on how hosts, services and contacts are added:

define host {
 name                   SRV1             
 host_name              SRV1
 address                192.168.0.1
 use                    generic-host     
 check_period           24x7            
 check_interval         5           
 retry_interval         1              
 max_check_attempts     10            
 notification_period    workhours     
 notification_interval  120
 notification_options   d,u,r
}

The host_name option defines a name to identify the host that has to be monitored. address is the IP address of this host. The use statement tells Nagios to inherit other configuration values from the generic-host template. check_period defines whether the machine has to be monitored 24x7. check_interval makes Nagios checking the service every 5 minutes and retry_interval tells Nagios to schedule host check retries at 1 minute intervals. Nagios tries to execute the checks multiple times when they do not pass. You can define how many attempts Nagios should do with the max_check_attempts directive. All configuration flags beginning with notification handle how Nagios should behave when a failure of a monitored service occurs. In the host definition above, Nagios notifies the administrators only on working hours. However, this can be adjusted with notification_period. According to notification_interval notifications will be resend every two hours. notification_options contains four different flags: d, u, r and n. They control in which state Nagios should notify the administrator. d stands for a down state, u for unreachable and r for recoveries. n does not send any notifications anymore.

define service {
 use                    generic-service
 host_name              SRV1
 service_description    PING
 contact_groups         router-admins
 check_command          check_ping!100.0,20%!500.0,60%
} 

The first configuration directive use tells Nagios to inherit from the generic-service template. host_name is the name that assigns the service to the host object. The host itself is defined in the host object definition. A description can be set with service_description. In the example above the description is just PING. Within the contact_groups option it is possible to refer to a group of people who will be contacted on a failure of the service. This group and its members are later defined in a contact group object definition. check_command sets the program that checks whether the service is available, or not.

define contact {
 contact_name           admins                
 use                    generic-contact        
 alias                  Nagios Admin
 email                  nagios@localhost        
}
                
define contactgroup {
 contactgroup_name      router-admins
 alias                  Administrators
 members                admins
}

The example listing above shows the direct contact definition and its proper contactgroup. The contact definition contains the e-mail address and the name of the person who is contacted on a failure of a service. Usually this is the responsible administrator. use inherits configuration values from the generic-contact definition.

An overview of all Nagios objects and further information about them can be found at: http://nagios.sourceforge.net/docs/3_0/objectdefinitions.html.

This section explains how to monitor remote services with Nagios. Proceed as follows to monitor a remote service:

If you need to monitor a different remote service, it is possible to adjust check_command in step Step 5. A full list of all available check programs can be obtained by executing ls /usr/lib/nagios/plugins/check_*

See Section 3.5, “Troubleshooting” if an error occurred.

This section explains how to monitor remote host resources with Nagios.

Proceed as follows on the Nagios server:

Proceed as follows on the client you want to monitor:

SystemTap usage is based on SystemTap scripts (*.stp). They tell SystemTap which type of information to collect, and what to do once that information is collected. The scripts are written in the SystemTap scripting language that is similar to AWK and C. For the language definition, see http://sourceware.org/systemtap/langref/.

The essential idea behind a SystemTap script is to name events, and to give them handlers. When SystemTap runs the script, it monitors for certain events. When an event occurs, the Linux kernel runs the handler as a sub-routine, then resumes. Thus, events serve as the triggers for handlers to run. Handlers can record specified data and print it in a certain manner.

The SystemTap language only uses a few data types (integers, strings, and associative arrays of these), and full control structures (blocks, conditionals, loops, functions). It has a lightweight punctuation (semicolons are optional) and does not need detailed declarations (types are inferred and checked automatically).

For more information about SystemTap scripts and their syntax, refer to Section 5.3, “Script Syntax” and to the stapprobes and stapfuncs man pages, that are available with the systemtap-doc package.

Tapsets are a library of pre-written probes and functions that can be used in SystemTap scripts. When a user runs a SystemTap script, SystemTap checks the script's probe events and handlers against the tapset library. SystemTap then loads the corresponding probes and functions before translating the script to C. Like SystemTap scripts themselves, tapsets use the filename extension *.stp.

However, unlike SystemTap scripts, tapsets are not meant for direct execution—they constitute the library from which other scripts can pull definitions. Thus, the tapset library is an abstraction layer designed to make it easier for users to define events and functions. Tapsets provide useful aliases for functions that users may want to specify as an event (knowing the proper alias is mostly easier than remembering specific kernel functions that might vary between kernel versions).

The main commands associated with SystemTap are stap and staprun. To execute them, you either need root privileges or must be a member of the stapdev or stapusr group.

For a list of options for each command, use --help. For details, refer to the stap and the staprun man pages.

Apart from the commands above which are used in a setup where you build the kernel modules on the same machine that you want to probe, there is also a specific set of commands for a client-server setup: systemtap-client and systemtap-server, the latter containing a number of subcommands. This set of commands allows you to compile a SystemTap module on a machine other than the one on which it will be run. For more information about this specific setup and the commands involved, refer to Section 5.2, “Installation and Setup” and to the stap-server and stap-client man pages.

To avoid giving root access to users just for running SystemTap, you can make use of the following SystemTap groups. They are not available by default on SUSE Linux Enterprise, but you can create the groups and modify the access rights accordingly.

The following list gives an overview of the SystemTap main files and directories.

If you use the SystemTap client-server setup, the following directories are also important:

For this setup, install the following packages (using either YaST or zypper).

To check if all packages are correctly installed on the machine and if SystemTap is ready to use, execute the following command as root.

stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'

It probes the currently used kernel by running a script and returning an output. If the output is similar to the following, SystemTap is successfully deployed and ready to use:

Pass : parsed user script and 59 library script(s) in 80usr/0sys/214real ms.
Pass : analyzed script: 1 probe(s), 11 function(s), 2 embed(s), 1 global(s) in 
 140usr/20sys/412real ms. 
Pass : translated to C into
 "/tmp/stapDwEk76/stap_1856e21ea1c246da85ad8c66b4338349_4970.c" in 160usr/0sys/408real ms. 
Pass : compiled C into "stap_1856e21ea1c246da85ad8c66b4338349_4970.ko" in 
 2030usr/360sys/10182real ms. 
Pass : starting run. 
 read performed
Pass : run completed in 10usr/20sys/257real ms. 

In case any error messages appear during the test, check the output for hints about any missing packages and make sure they are installed correctly. Rebooting and loading the appropriate kernel may also be needed.

A SystemTap compile server listens for connections from SystemTap clients on a secure SSL network port and accepts requests to run the SystemTap front-end. The server advertises its presence and configuration on the local network using avahi (a free Zeroconf implementation that allows programs to publish and discover services and hosts in a local network without any specific configuration). The compile server broadcasts its IP address, port, and details about the Linux kernel it runs. Thus, the SystemTap client can automatically detect a compile server on the network that is compatible to the client's kernel version.

As SystemTap exposes kernel internal data structures and potentially private user information, it provides several layers of security:

5.2.2.2. Setting Up the Server¶

You have two choices for setting up the SystemTap compile server: you can run it as root or as non-root user. This has implications on the certificate management on server- and client-side and on the process of establishing a given compile server as trusted by a given client. For the SSL connection between the compile server and the SystemTap client, you need to create a certificate for authentication. Depending on how the SystemTap compile server is set up (as root or as non-root), the location of the server certificate differs. When set up as root user, the certificate is stored in a database at /etc/systemtap/ssl/server. However, when the compile server is set up as non-root (usually by the user stap-server), the server certificate is stored in a database in the systemtap-server user's home directory: ~/.systemtap/ssl/server.

Procedure 5.1. Running the Compile Server as Non-root User¶

For this setup, it is advisable to create a dedicated system group and user for the compile server.

1. Log in as root.

2. Create a home directory for the compile server user, for example:

   mkdir /var/lib/stapserver

3. Add a system group for the operation of the compile server. In the following example, the group is named stap-server and the group ID (GID) is 155, but you can also specify a different group name or GID:

   groupadd -g 155 -r stap-server

4. Add a user belonging to the group you created before and specify the user's home directory:

   useradd -c "SystemTap Compile Server" -u 155 -g stap-server -d \ /var/lib/stapserver -m -r -s /sbin/nologin stap-server

   The command above will create a user named stap-server with the user ID 155. The user's finger information is specified with -c and the options -g and -d specify the user's main group (stap-server) and his home directory you created in Step 2, respectively. The user account will be a system account (specified with -r) and the user will not be able to log in, as his login shell is set to /sbin/nologin with the -s option.

5. Change the owner and the group for the home directory to use:

   chown -R stap-server.stap-server /var/lib/stapserver/

6. Run a shell as user stap-server and pass the stap-gen-cert command to generate a SystemTap certificate:

   su -s /bin/sh - stap-server -c  /usr/bin/stap-gen-cert

   You are prompted to set a password for the SystemTap server certificate and key database.

7. Enter a password for the SystemTap server certificate and confirm it.

   This generates a certificate (stap.cert) that is stored in the systemtap-server user's home directory—in this case: /var/lib/stapserver/.systemtap/ssl/server.

8. Start the compile server with:

   su -s /bin/sh - stap-server -c /usr/bin/stap-start-server

   Upon first start of the compile server, this creates a client-side certificate database in the systemtap-server user's home directory (~/.systemtap/ssl/client) to which the server's certificate has now automatically been added. Thus, a server started by the user stap-server is automatically trusted by clients run by that user.

Procedure 5.2. Running the Compile Server as root User¶

Compared to Procedure 5.1, “Running the Compile Server as Non-root User”, this setup is much simpler but it has security implications.

	Security Risk
In the following setup, the compile server certificate is stored in /etc/systemtap/ssl/server, together with the client-side database located at /etc/systemtap/ssl/client. As these files are accessible for anyone, anyone can run the stap-client command, thus potentially exposing kernel internal data structures and private user information.

1. Log in as root.

2. Create a SystemTap certificate by executing the following command:

   /usr/bin/stap-gen-cert

   You are prompted to set a password for the SystemTap server certificate and key database.

3. Enter a password and confirm it.

   The certificate (stap.cert) is generated. In contrast to the setup as non-root, it is stored in a database located at /etc/systemtap/ssl/server.

4. Start a SystemTap server on the local host by using the following command:

   /usr/bin/stap-start-server

   At the same time, a client-side certificate database is created at /etc/systemtap/ssl/client. The server certificate is automatically added to the client-side certificate database.

The client-side certificate database created for root is also the global client-side database for all clients on the host. Thus, a server started by root is automatically trusted by clients run by any user on that host: Any user can now compile kernel modules on the compile server using the stap-client command. For more information about the security implications, see the Safety and Security section of the stap-server man page.

5.2.2.4. Using the Client¶

After you have set up the SystemTap compile server and client as described in the previous sections, you can make use of the stap-client program. It is analogous to the stap front-end, except that it tries to find a compatible SystemTap compile server on the local network. It then uses this server for compiling the SystemTap script into a module, loading the kernel module and enabling the probes (passes 1-4 of a SystemTap session). If requested, pass 5 actions are performed on the localhost using staprun. For more information about a SystemTap session and the individual passes, see Section 5.2.1, “Classic Setup and Initial Test”.

	Executing stap-client
You can run stap-client either as root or as non-root. If run as non-root, the underlying staprun command needs to be suid and the user executing stap-client must be a member of the stapdev group. For more information, refer to Section 5.1.3, “Commands and Privileges”.

Usually, a running SystemTap compile server on the local network advertises its presence using avahi and is automatically detected by the SystemTap client. The following procedure illustrates how to make use of the SystemTap client-server setup and covers the most common commands and options needed for that.

1. To make sure that a compatible SystemTap server is running on your local network, execute the following command on the SystemTap client:

   stap-find-servers

   This invokes avahi-browse to find servers. The details of any servers found are echoed to standard output. If this command does not return anything, no compatible SystemTap server can be found on your network.

2. In this case, log in to the compile server and run

   stap-start-server

   This starts avahi-publish-service in the background. The server listens for connections on a random port and advertises its presence on the local network using the avahi daemon. If the server is started successfully, the process ID of the server is echoed to standard output.

   Note that stap-start-server does not work for the initial setup as described in Procedure 5.2, “Running the Compile Server as root User”, where /usr/bin/stap-serverd is used instead. stap-start-server puts the server in the background—thus, you would not see the prompt asking for the server certificate password.

   ps -ef | grep avahi

   should now return an output similar to the following:

   avahi 3300     1  0 15:14 ?      00:00:00 avahi-daemon: running [linux-48zp.local]
   root  4687  4655  0 18:03 ttyS0  00:00:00 avahi-publish-service Systemtap Compile Se
   root  4700  4160  0 18:05 ttyS0  00:00:00 grep avahi

3. To run a simple test, execute the following command on the SystemTap client:

   stap-client -e 'probe begin { printf("Hello"); exit(); }'

   This compiles and executes a simple example on any compatible SystemTap server found on the local network. If the test is successful, it prints “Hello” to standard output.

Instead of using any compatible server found on the network, you can also determine which SystemTap server to contact and use. To do so, run the stap-client command with the --server option. It lets you specify the hostname or IP address of the SystemTap server, optionally also a port (which is useful fore connecting to non-local servers). For more information and details about the other available commands and options, refer to the stap-server and stap-client man pages.

A SystemTap script can have multiple probes. They must be written in the following format:

probe event {statements}

Each probe has a corresponding statement block. This statement block must be enclosed in { } and contains the statements to be executed per event.

probe begin
{
   printf ("hello world\n")
   exit ()
}

If your statement block holds several statements, SystemTap executes these statements in sequence—you do not need to insert special separators or terminators between multiple statements. A statement block can also be nested within another statement blocks. Generally, statement blocks in SystemTap scripts use the same syntax and semantics as in the C programming language.

SystemTap supports a number of built-in events.

The general event syntax is a dotted-symbol sequence. This allows a breakdown of the event namespace into parts. Each component identifier may be parametrized by a string or number literal, with a syntax like a function call. A component may include a * character, to expand to other matching probe points. A probe point may be followed by a ? character, to indicate that it is optional, and that no error should result if it fails to expand. Alternately, a probe point may be followed by a ! character to indicate that it is both optional and sufficient.

SystemTap supports multiple events per probe—they need to be separated by a comma (,). If multiple events are specified in a single probe, SystemTap will execute the handler when any of the specified events occur.

In general, events can be classified into the following categories:

probe timer.s(4)
{
   printf("hello world\n")
}

For detailed information about supported events, refer to the stapprobes man page. The See Also section of the man page also contains links to other man pages that discuss supported events for specific subsystems and components.

Each SystemTap event is accompanied by a corresponding handler defined for that event, consisting of a statement block.

function function_name(arguments) {statements}
probe event {function_name(arguments)}

printf ("%s(%d) open\n", execname(), pid())

[...]
vmware-guestd(2206) open
hald(2360) open
[...]
   

foo = gettimeofday( )

global count_jiffies, count_ms
probe timer.jiffies(100) { count_jiffies ++ }
probe timer.ms(100) { count_ms ++ }
probe timer.ms(12345)
{
  hz=(1000*count_jiffies) / count_ms
  printf ("jiffies:ms ratio %d:%d => CONFIG_HZ=%d\n",
    count_jiffies, count_ms, hz)
  exit ()
  }

Kprobe can be attached to any instruction in the Linux kernel. When it is registered, it inserts a break-point at the first bytes of the probed instruction. When the processor hits this break-point, the processor registers are saved, and the processing passes to kprobes. First, a pre-handler is executed, then the probed instruction is stepped, and, finally a post-handler is executed. The control is then passed to the instruction following the probe point.

Jprobe is implemented through the kprobe mechanism. It is inserted on a function's entry point and allows direct access to the arguments of the function which is being probed. Its handler routine must have the same argument list and return value as the probed function. It also has to end by calling the jprobe_return() function.

When jprobe is hit, the processor registers are saved, and the instruction pointer is directed to the jprobe handler routine. The control then passes to the handler with the same register contents as the function being probed. Finally, the handler calls the jprobe_return() function, and switches the control back to the control function.

In general, you can insert multiple probes on one function. Jprobe is, however, limited to only one instance per function.

Return probes are also implemented through kprobes. When the register_kretprobe() function is called, a kprobe is attached to the entry of the probed function. After hitting the probe, the Kernel probes mechanism saves the probed function return address and calls a user-defined return handler. The control is then passed back to the probed function.

Before you call register_kretprobe(), you need to set a maxactive argument, which specifies how many instances of the function can be probed at the same time. If set too low, you will miss a certain number of probes.

The list of all currently registered kprobes is in the /sys/kernel/debug/kprobes/list file.

saturn.example.com:~ # cat /sys/kernel/debug/kprobes/list
c015d71a  k  vfs_read+0x0   [DISABLED]
c011a316  j  do_fork+0x0
c03dedc5  r  tcp_v4_rcv+0x0

The first column lists the address in the kernel where the probe is inserted. The second column prints the type of the probe: k for kprobe, j for jprobe, and r for return probe. The third column specifies the symbol, offset and optional module name of the probe. The following optional columns include the status information of the probe. If the probe is inserted on a virtual address which is not valid anymore, it is marked with [GONE]. If the probe is temporarily disabled, it is marked with [DISABLED].

The /sys/kernel/debug/kprobes/enabled file represents a switch with which you can globally and forcibly turn on or off all the registered kernel probes. To turn them off, simply enter

echo "0" > /sys/kernel/debug/kprobes/enabled

on the command line as root. To turn them on again, enter

echo "1" > /sys/kernel/debug/kprobes/enabled

Note that this way you do not change the status of the probes. If a probe is temporarily disabled, it will not be enabled automatically but will remain in the [DISABLED] state after entering the latter command.

Performance monitoring is “the action of collecting information related to how an application or system performs”. The information can be obtained from the code or the CPU/chipset.

Modern processors contain a performance monitoring unit (PMU). The design and functionality of a PMU is CPU specific: for example, the number of registers, counters and features supported will vary by CPU implementation.

The Perfmon interface is designed to be generic, flexible and extensible. It can monitor at the program (thread) or system levels. In either mode, it is possible to count or sample your profile information. This uniformity makes it easier to write portable tools. Figure 7.1, “Architecture of perfmon2” gives an overview.

Figure 7.1. Architecture of perfmon2¶

Each PMU model consists of a set of registers: the performance monitor configuration (PMC) and the performance monitor data (PMD). Only PMCs is writeable, but both can be read. These registers store configuration information and data.

Perfmon2 supports two modes where you can run your profiling: sampling or couting.

Sampling is usually expressed by an interval of time (time-based) or an occurance of a definied number of events (event-based). Perfmon indirectly supports time-based sampling by using an event-based sample with constant correlation to time (for example, unhalted_reference_cycles.)

In contrast, Counting is expressed in terms of a number of occurances of an event.

Both methods store their information into a sample. This sample contains information about, for example, where a thread was or instruction pointers.

The following example demonstrates the counting of the CPU_OP_CYCLES event and the sampling of this event, generating a sample per 100000 occurances of the event:

pfmon --no-cmd-output -e CPU_OP_CYCLES_ALL /bin/ls
1306604 CPU_OP_CYCLES_ALL

The following command gives the count of a specific function and the procentual amount of the total cycles:

pfmon --no-cmd-output --short-smpl-periods=100000   -e CPU_OP_CYCLES_ALL /bin/ls
# results for [28119:28119<-[28102]] (/bin/ls)
# total samples          : 12
# total buffer overflows : 0
#
#                   event00 
#     counts   %self    %cum          code addr 
           1   8.33%   8.33% 0x2000000000007180 
           1   8.33%  16.67% 0x20000000000195a0 
           1   8.33%  25.00% 0x2000000000019260 
           1   8.33%  33.33% 0x2000000000014e60 
           1   8.33%  41.67% 0x20000000001f38c0 
           1   8.33%  50.00% 0x20000000001ea481 
           1   8.33%  58.33% 0x200000000020b260 
           1   8.33%  66.67% 0x2000000000203490 
           1   8.33%  75.00% 0x2000000000203360 
           1   8.33%  83.33% 0x2000000000203440 
           1   8.33%  91.67% 0x4000000000002690 
           1   8.33% 100.00% 0x20000000001cfdf1

To get a list of supported events, use the option -l from pfmon to list them. Keep in mind, this list depends on the host PMU:

pfmon -l
ALAT_CAPACITY_MISS_ALL
ALAT_CAPACITY_MISS_FP
ALAT_CAPACITY_MISS_INT
BACK_END_BUBBLE_ALL
BACK_END_BUBBLE_FE
BACK_END_BUBBLE_L1D_FPU_RSE
...
CPU_CPL_CHANGES_ALL
CPU_CPL_CHANGES_LVL0
CPU_CPL_CHANGES_LVL1
CPU_CPL_CHANGES_LVL2
CPU_CPL_CHANGES_LVL3
CPU_OP_CYCLES_ALL
CPU_OP_CYCLES_QUAL
CPU_OP_CYCLES_HALTED
DATA_DEBUG_REGISTER_FAULT
DATA_DEBUG_REGISTER_MATCHES
DATA_EAR_ALAT
...

Get an explanation of these entries with the option -i and the event name:

pfmon -i CPU_OP_CYCLES_ALL
Name     : CPU_OP_CYCLES_ALL
Code     : 0x12
Counters : [ 4 5 6 7 8 9 10 11 12 13 14 15 ]
Desc     : CPU Operating Cycles -- All CPU cycles counted
Umask    : 0x0
EAR      : None
ETB      : No
MaxIncr  : 1  (Threshold 0)
Qual     : None
Type     : Causal
Set      : None

Use the --system-wide option to enable monitoring all processes that execute on a specific CPU or sets of CPUs. You do not have to be root to do so; per default, user level is turned on for all events (option -u).

It is possible that one system wide session can run concurrently with another system wide sessions as long as they do not monitor the same set of CPUs. However, you can not run a system wide session with any per-thread sessions together.

The following examples are taken from a Itanium IA64 Montecito processor. To execute a system-wide session, perform the following procedure:

Perfmon can also monitor an existing thread. This is useful for monitoring system daemons or programs which take a long time to start. First determine the process ID you wish to monitor:

ps ax | grep foo
10027  pts/1    R     2:23  foo

Use the found PID for the --attach-task option of pfmon:

pfmon --attach-task=10027
3682190 CPU_OP_CYCLES_ALL

In its simplest form, start the daemon, collect data, stop the daemon, and create your report. This method is described in detail in the following procedure:

The general procedure for event configuration is as follows:

If you need to profile certain events, first check the available events supported by your processor with the ophelp command (example output generated from Intel Core i5 CPU):

ophelp 
oprofile: available events for CPU type "Intel Architectural Perfmon"

See Intel 64 and IA-32 Architectures Software Developer's Manual
Volume 3B (Document 253669) Chapter 18 for architectural perfmon events
This is a limited set of fallback events because oprofile doesn't know your CPU
CPU_CLK_UNHALTED: (counter: all))
        Clock cycles when not halted (min count: 6000) 
INST_RETIRED: (counter: all))
        number of instructions retired (min count: 6000) 
LLC_MISSES: (counter: all))
        Last level cache demand requests from this core that missed the LLC (min count: 6000) 
        Unit masks (default 0x41)
        ----------
        0x41: No unit mask 
LLC_REFS: (counter: all))
        Last level cache demand requests from this core (min count: 6000) 
        Unit masks (default 0x4f)
        ----------
        0x4f: No unit mask 
BR_MISS_PRED_RETIRED: (counter: all))
        number of mispredicted branches retired (precise) (min count: 500)

You can get the same output from opcontrol --list-events.

Specify the performance counter events with the option --event. Multiple options are possible. This option needs an event name (from ophelp) and a sample rate, for example:

opcontrol --event=CPU_CLK_UNHALTED:100000

	Be Careful with Low Sampling Rates with CPU_CLK_UNHALTED
Setting sampling rates is dangerous as small rates cause the system to overload and freeze.

Depending on the server's range of application and the hardware layout the partitioning scheme can influence the machine's performance (although to a lesser extend only). It is beyond the scope of this manual to suggest different partion schemes for particular workloads, however, the following rules will positively affect performance. Of course they do not apply when using an external storage system.

Actually, the installation scope has no direct influence on the machine's performance, but a carefully chosen scope of packages nevertheless has got advantages. It is recommended to install the minimum of packages needed to run the server. A system with a minimum set of packages is easier to mainatain and has got less potential security issues. Furthermore, a tailor made installation scope also ensures no unnecessary services are started by default.

openSUSE lets you customize the installation scope on the Installation Summary screen. By default, you can select or remove preconfigured patterns for specific tasks, but it is also possible to start the YaST Software Manager for a fine-grained package based selection.

One or more of the following patterns selected for installation by default may not be needed in all cases:

A running X Window system eats up many ressources and is seldomly needed on a server. It is strongly recommended to start the system in runlevel 3 (Full multi-user with network, no X). You will still be able to start graphical applications from remote or use the startx command to start a local graphical desktop.

openSUSE ships with a number of different file systems, including Ext3, Ext2, ReiserFS, and XFS. Each file system has its own advantages and disadvantages.

Whenever a file is read on a Linux file system, its access time (atime) is updated. As a result, each read-only file access in fact causes a write. On a journaling file system it is even two write operations since the journal will be updated, too. It is recommended to turn this feature off when you do not need to keep track of access times. This is possibly true for file and Web servers as well as for a netwok storage.

To turn off access time updates, mount the file system with the noatime option. To do so, either edit /etc/fstab directly, or use the Fstab Options dialog when editing or adding a partition with the YaST Partitioner.

The ionice command lets you prioritize disk access for single processes. This enables you to give less I/O priority to non time-critical background processes with heavy disk access such as backup jobs. On the other hand ionice lets you raise I/O priority for a specific process to make sure this process has always immediate access to the disk. You may set the following three scheduling classes:

For more details and the exact command syntax refer to the ionice(1) man page for ionice.

To use cgroups, install the following additional packages:

The kernel shipped with openSUSE supports cgroups. There is no need to apply additional patches. Execute lxc-checkconfig to see a cgroups environment similar to the following output:

--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled
Multiple /dev/pts instances: enabled

--- Control groups ---
Cgroup: enabled
Cgroup namespace: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled
Macvlan: enabled
Vlan: enabled
File capabilities: enabled

To find out which subsystems are available, proceed as follows:

mkdir /cgroups
mount -t cgroup none /cgroups
grep cgroup /proc/mounts

The following subsystems are available: rw, freezer, devices, cpuacct, cpu, ns, cpuset, memory. Disk and network subsystem controllers may become available during SUSE Linux Enterprise Server 11 lifetime.

With the command line proceed as follows:

For background information and additional configuration flags, see /usr/src/linux/Documentation/cgroups/cpusets.txt.

With the cset tool, proceed as follows:

# Determine the number of CPUs and memory nodes
cset set --list
# Creating the cpuset hierarchy
cset set --cpu=2-3 --mem=1 --set=Charlie
# Starting processes in a cpuset
cset proc --set Charlie --exec -- stress -c 1 &
# Moving existing processes to a cpuset
cset proc --move --pid PID --toset=Charlie
# List task in a cpuset
cset proc --list --set Charlie
# Removing a cpuset
cset set --destroy Charlie
 

Using shell commands, proceed as follows:

Modern processors have several power saving modes called C-states. They reflect the capability of an idle processor to turn off unused components in order to save power. Whereas C-states have been available for laptops for some time, they are a rather recent trend in the server market (for example, with Intel* processors, C-modes are only available since Nehalem).

When a processor runs in the C0 state, it is executing instructions. A processor running in any other C-state is idle. The higher the C number, the deeper the CPU sleep mode: more components are shut down to save power. Deeper sleep states save more power, but the downside is that they have higher latency (the time the CPU needs to go back to C0).

Some states also have submodes with different power saving latency levels. Which C-states and submodes are supported depends on the respective processor. However, C1 is always available.

Table 11.1, “C-States” gives an overview of the most common C-states.

While a processor operates (in C0 state), it can be in one of several CPU performance states (P-states). Whereas C-states are idle states (all but C0), P-states are operational states that relate to CPU frequency and voltage.

The higher the P-state, the lower the frequency and voltage at which the processor runs. The number of P-states is processor-specific and the implementation differs across the various types. However, P0 is always the highest-performance state. Higher P-state numbers represent slower processor speeds and lower power consumption. For example, a processor in P3 state runs more slowly and uses less power than a processor running at P1 state. To operate at any P-state, the processor must be in the C0 state where the processor is working and not idling. The CPU P-states are also defined in the Advanced Configuration and Power Interface (ACPI) specification, see http://www.acpi.info/spec.htm.

C-states and P-states can vary independently of one another.

T-states refer to throttling the processor clock to lower frequencies in order to reduce thermal effects. This means that the CPU is forced to be idle a fixed percentage of its cycles per second. Throttling states range from T1 (the CPU has no forced idle cycles) to Tn, with the percentage of idle cycles increasing the greater n is.

This differs from changing the frequency (which makes the CPU have fewer cycles per second), and from running in a C-state other than C1. Note that throttling does not reduce voltage and since the CPU is forced to idle part of the time, processes will take longer to finish and will consume more power instead of saving any power.

T-states are a concept from the times when dynamic frequency scaling and C-states did not exist. With the implementation of the latter, T-states are only useful if reducing thermal effects is the primary goal. Since T-states can interfere with C-states (preventing the CPU from reaching higher C-states), they can even increase power consumption in a modern CPU capable of C-states.

You can think of the in-kernel governors as a sort of pre-configured power schemes for the CPU. The CPUfreq governors use P-states to change frequencies and lower power consumption. The dynamic governors can switch between CPU frequencies, based on CPU utilization to allow for power savings while not sacrificing performance. These governors also allow for some tuning so you can customize and change the frequency scaling.

The following governors are available with the CPUfreq subsystem:

If the CPUfreq subsystem in enabled on your system (which it is by default with SUSE Linux Enterprise Server), you can find the relevant files and directories under /sys/devices/system/cpu/. If you list the contents of this directory, you will find a cpu{0..x} subdirectory for each processor, and several other files and directories. You will find a cpufreq subdirectory in each processor directory, holding a number of files and directories that define the parameters for CPUfreq. Some of them are writable (for root), some of them are read-only. If your system currently uses the on-demand or conservative governor, you will see a separate subdirectory for those governors in cpufreq, containing the parameters for the governors.

	Different Processor Settings
The settings under the cpufreq directory can be different for each processor. If you want to use the same policies across all processors, you need to adjust the parameters for each processor.

The cpufreq-info command helps you to retrieve CPUfreq kernel information. Run without any options, it collects the information available for your system and shows an output similar to the following:

 

cpufrequtils 004: cpufreq-info (C) Dominik Brodowski 2004-2006
Report errors and bugs to http://bugs.opensuse.org, please.    
analyzing CPU 0:
  driver: acpi-cpufreq
  CPUs which need to switch frequency at the same time: 0
  hardware limits: 2.80 GHz - 3.40 GHz
  available frequency steps: 3.40 GHz, 2.80 GHz
  available cpufreq governors: conservative, userspace, powersave, ondemand, performance
  current policy: frequency should be within 2.80 GHz and 3.40 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 3.40 GHz.
analyzing CPU 1:
  driver: acpi-cpufreq
  CPUs which need to switch frequency at the same time: 1
  hardware limits: 2.80 GHz - 3.40 GHz
  available frequency steps: 3.40 GHz, 2.80 GHz
  available cpufreq governors: conservative, userspace, powersave, ondemand, performance
  current policy: frequency should be within 2.80 GHz and 3.40 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 3.40 GHz.
    

Using the appropriate options, you can view the current CPU frequency, the minimum and maximum CPU frequency allowed, show the currently used CPUfreq policy, the available CPUfreq governors, or determine the CPUfreq kernel driver used. For more details and the available options, refer to the cpufreq-info man page or run cpufreq-info --help.

To modify CPUfreq settings, use the cpufreq-set command as root. It allows you set values for the minimum or maximum CPU frequency the governor may select or to create a new governor. With the -c option, you can also specify for which of the processors the settings should be modified. That makes it easy to use a consistent policy across all processors without adjusting the settings for each processor individually. For more details and the available options, refer to the cpufreq-set man page or run cpufreq-set --help.

You can switch to another governor at runtime with the -g option. For example, the following command will activate the on-demand governor:

cpufreq-set -g ondemand

If you want the change in the governor to persist after a reboot or shutdown, use the pm-profiler as described in Section 11.5, “Creating and Using Power Management Profiles”.

Apart from the governor settings that can be influenced with cpufreq-set (like minimum or maximum CPU frequency to be used), you can also tune further governor parameters manually, for example, Ignoring Nice Values in Processor Utilization.

Another parameter that significantly impacts the performance loss caused by dynamic frequency scaling is the sampling rate (rate at which the governor checks the current CPU load and adjusts the processor's frequency accordingly). Its default value depends on a BIOS value and it should be as low as possible. However, in modern systems, an appropriate sampling rate is set by default and does not need manual intervention.

When setting the ignore_nice_load value for cpu0, the same value is automatically used for all cores. In this case, you do not need to repeat the steps above for each of the processors where you want to modify this governor parameter.

This is the default option. Fairness-oriented I/O scheduler. The algorithm assigns each thread a time slice in which it is allowed to submit I/O to disk. This way each thread gets a fair share of I/O throughput. This I/O scheduler also allows assigning tasks I/O priorities which are taken into account during scheduling decisions (see man 1 ionice). The CFQ scheduler has the following parameters:

A trivial scheduler that just passes down the I/O that comes to it. Useful for checking whether complex I/O scheduling decisions of other schedulers are not causing I/O performance regressions.

In some cases it can be helpful for devices that do I/O scheduling themselves, as intelligent storage, or devices that do not depend on mechanical movement, like SSDs. Usually, the DEADLINE I/O scheduler is a better choice for these devices. It is a rather lightweight I/O scheduler but already does some useful work. However, NOOP may produce better performance on certain workloads.

Latency-oriented I/O scheduler. The algorithm preferably serves reads before writes. /sys/block/<device>/queue/iosched/writes_starved controls how many reads can be sent to disk before it is possible to send writes and tries to observe given deadlines /sys/block/<device>/queue/iosched/read_expire for reads and /sys/block/<device>/queue/iosched/write_expire for writes after which I/O must be submitted to disk. This I/O scheduler can provide a superior throughput over the CFQ I/O scheduler in cases where several threads read and write and fairness is not an issue. For example, for several parallel readers from a SAN or some database-like loads.

The theory behind task scheduling is very simple. If there are runnable processes in a system, at least one process must always be running. If there are more runnable processes than processors in a system, not all the processes can be running all the time.

Therefore, some processes need to be stopped temporarily, or suspended, so that others can be running again. The scheduler decides what process in the queue will run next.

As already mentioned, Linux, like all other Unix variants, is a multitasking operating system. That means that several tasks can be running at the same time. Linux provides a so called preemptive multitasking, where the scheduler decides when a process is suspended. This forced suspension is called preemption. All Unix flavors have been providing preemptive multitasking since the beginning.

The time period for which a process will be running before it is preempted is defined in advance. It is called a process' timeslice and represents the amount of processor time that is provided to each process. By assigning timeslices, the scheduler makes global decisions for the running system, and prevents individual processes from dominating over the processor resources.

The scheduler evaluates processes based on their priority. To calculate the current priority of a process, the task scheduler uses complex algorithms. As a result, each process is given a value according to which it is “allowed” to run on a processor.

CFS tries to guarantee a fair approach to each runnable task. To find the most balanced way of task scheduling, it uses the concept of red-black tree. A red-black tree is a type of self-balancing data search tree which provides inserting and removing entries in a reasonable way so that it remains well balanced. For more information, see the wiki pages of Red-black tree.

When a task enters into the run queue (a planned time line of processes to be executed next), the scheduler records the current time. While the process waits for processor time, its “wait” value gets incremented by an amount derived from the total number of tasks currently in the run queue and the process priority. As soon as the processor runs the task, its “wait” value gets decremented. If the value drops below a certain level, the task is preempted by the scheduler and other tasks get closer to the processor. By this algorithm, CFS tries to reach the ideal state where the “wait” value is always zero.

Since the Linux kernel version 2.6.24, CFS can be tuned to be fair to users or groups rather than to tasks only. Runnable tasks are then grouped to form entities, and CFS tries to be fair to these entities instead of individual runnable tasks. The scheduler also tries to be fair to individual tasks within these entities.

Tasks can be grouped in two mutually exclusive ways:

The way the kernel scheduler lets you group the runnable tasks depends on setting the kernel compile-time options CONFIG_FAIR_USER_SCHED and CONFIG_FAIR_CGROUP_SCHED. The default setting in openSUSE® 11.4 is to use control groups, which lets you create groups as needed. For more information, see Chapter 10, Kernel Control Groups.

Basic aspects of the task scheduler behavior can be set through the kernel configuration options. Setting these options is part of the kernel compilation process. Because kernel compilation process is a complex task and out of this document's scope, refer to relevant source of information (for example http://en.opensuse.org/Configure,_Build_and_Install_a_Custom_Linux_Kernel).

	Kernel Compilation
If you run openSUSE on a kernel that was not shipped with it, for example on a self-compiled kernel, you loose the entire support entitlement.

Documents regarding task scheduling policy often use several technical terms which you need to know to understand the information correctly. Here are some of them:

The sysctl interface for examining and changing kernel parameters at runtime introduces important variables by means of which you can change the default behavior of the task scheduler. The syntax of the sysctl is simple, and all the following commands must be entered on the command line as root.

To read a value from a kernel variable, enter

sysctl variable

To assign a value, enter

sysctl variable=value

To get a list of all scheduler related sysctl variables, enter

sysctl -A | grep "sched" | grep -v "domain"

saturn.example.com:~ # sysctl -A | grep "sched" | grep -v "domain"
kernel.sched_child_runs_first = 0
kernel.sched_min_granularity_ns = 1000000
kernel.sched_latency_ns = 5000000
kernel.sched_wakeup_granularity_ns = 1000000
kernel.sched_shares_ratelimit = 250000
kernel.sched_tunable_scaling = 1
kernel.sched_shares_thresh = 4
kernel.sched_features = 15834238
kernel.sched_migration_cost = 500000
kernel.sched_nr_migrate = 32
kernel.sched_time_avg = 1000
kernel.sched_rt_period_us = 1000000
kernel.sched_rt_runtime_us = 950000
kernel.sched_compat_yield = 0

Note that variables ending with “_ns” and “_us” accept values in nanoseconds and microseconds, respectively.

A list of the most important task scheduler sysctl tuning variables (located at /proc/sys/kernel/) with a short description follows: