systemd
Daemonjournalctl
: Query the systemd
Journaludev
systemd
Daemon
The program systemd
is the process with process ID 1. It is
responsible for initializing the system in the required way. systemd
is started directly by the Kernel and resists signal 9, which
normally terminates processes. All other programs are either started
directly by systemd or by one of its child processes.
Starting with openSUSE Leap 12 systemd is a replacement for the popular
System V init daemon. systemd
is fully compatible with System V init
(by supporting init scripts). One of the main advantages of systemd is
that it considerably speeds up boot time by aggressively paralleling
service starts. Furthermore, systemd only starts a service when it is
really needed. Daemons are not started unconditionally at boot time, but
rather when being required for the first time. systemd also supports
Kernel Control Groups (cgroups), snapshotting and restoring the system
state and more. See
http://www.freedesktop.org/wiki/Software/systemd/ for
details.
This section will go into detail about the concept behind systemd.
systemd is a system and session manager for Linux, compatible with System V and LSB init scripts. The main features are:
provides aggressive parallelization capabilities
uses socket and D-Bus activation for starting services
offers on-demand starting of daemons
keeps track of processes using Linux cgroups
supports snapshotting and restoring of the system state
maintains mount and automount points
implements an elaborate transactional dependency-based service control logic
A unit configuration file encodes information about a service, a socket, a device, a mount point, an automount point, a swap file or partition, a start-up target, a watched file system path, a timer controlled and supervised by systemd, a temporary system state snapshot, a resource management slice or a group of externally created processes. “Unit file” is a generic term used by systemd for the following:
Service. Information about a process (for example running a daemon); file ends with .service
Targets. Used for grouping units and as synchronization points during start-up; file ends with .target
Sockets.
Information about an IPC or network socket or a file system FIFO, for
socket-based activation (like
inetd
); file ends with
.socket
Path. Used to trigger other units (for example running a service when files change); file ends with .path
Timer. Information about a timer controlled, for timer-based activation; file ends with .timer
Mount point. Usually auto-generated by the fstab generator; file ends with .mount
Automount point. Information about a file system automount point; file ends with .automount
Swap. Information about a swap device or file for memory paging; file ends with .swap
Device. Information about a device unit as exposed in the sysfs/udev(7) device tree; file ends with .device
Scope / Slice. A concept for hierarchically managing resources of a group of processes; file ends with .scope/.slice
For more information about systemd.unit see http://www.freedesktop.org/software/systemd/man/systemd.unit.html
The System V init system uses several different commands to handle
services—the init scripts, insserv
,
telinit
and others. systemd makes it easier to manage
services, since there is only one command to memorize for the majority of
service-handling tasks: systemctl
. It uses the
“command plus subcommand” notation like
git
or zypper
:
systemctl [general OPTIONS] subcommand [subcommand OPTIONS]
See man 1 systemctl
for a complete manual.
If the output goes to a terminal (and not to a pipe or a file, for
example) systemd commands send long output to a pager by default. Use
the --no-pager
option to turn off paging mode.
systemd also supports bash-completion, allowing you to enter the first
letters of a subcommand and then press →| to
automatically complete it. This feature is only available in the
bash
shell and requires the installation of the
package bash-completion
.
Subcommands for managing services are the same as for managing a service
with System V init (start
, stop
,
...). The general syntax for service management commands is as follows:
systemctl reload|restart|start|status|stop|... <my_service(s)>
rc<my_service(s)> reload|restart|start|status|stop|...
systemd allows you to manage several services in one go. Instead of executing init scripts one after the other as with System V init, execute a command like the following:
systemctl start <my_1st_service> <my_2nd_service>
If you want to list all services available on the system:
systemctl list-unit-files --type=service
The following table lists the most important service management commands for systemd and System V init:
Task |
systemd Command |
System V init Command |
---|---|---|
Starting. |
start |
start |
Stopping. |
stop |
stop |
Restarting. Shuts down services and starts them afterwards. If a service is not yet running it will be started. |
restart |
restart |
Restarting conditionally. Restarts services if they are currently running. Does nothing for services that are not running. |
try-restart |
try-restart |
Reloading.
Tells services to reload their configuration files without
interrupting operation. Use case: Tell Apache to reload a modified
|
reload |
reload |
Reloading or restarting. Reloads services if reloading is supported, otherwise restarts them. If a service is not yet running it will be started. |
reload-or-restart |
n/a |
Reloading or restarting conditionally. Reloads services if reloading is supported, otherwise restarts them if currently running. Does nothing for services that are not running. |
reload-or-try-restart |
n/a |
Getting detailed status information.
Lists information about the status of services. The |
status |
status |
Getting short status information. Shows whether services are active or not. |
is-active |
status |
The service management commands mentioned in the previous section let you manipulate services for the current session. systemd also lets you permanently enable or disable services, so they are automatically started when requested or are always unavailable. You can either do this by using YaST, or on the command line.
The following table lists enabling and disabling commands for systemd and System V init:
When enabling a service on the command line, it is not started
automatically. It is scheduled to be started with the next system
start-up or runlevel/target change. To immediately start a service
after having enabled it, explicitly run systemctl
start
<my_service>
or
rc <my_service>
start
.
Task |
|
System V init Command |
---|---|---|
Enabling. |
|
|
Disabling. |
|
|
Checking. Shows whether a service is enabled or not. |
|
n/a |
Re-enabling. Similar to restarting a service, this commands first disables and then enables a service. Useful to re-enable a service with its defaults. |
|
n/a |
Masking. After “disabling” a service, it can still be started manually. To completely disable a service, you need to mask it. Use with care. |
|
n/a |
Unmasking. A service that has been masked can only be used again after it has been unmasked. |
|
n/a |
The entire process of starting the system and shutting it down is maintained by systemd. From this point of view, the Kernel can be considered a background process to maintain all other processes and adjust CPU time and hardware access according to requests from other programs.
With System V init the system was booted into a so-called
“Runlevel”. A runlevel defines how the system is started
and what services are available in the running system. Runlevels are
numbered; the most commonly known ones are 0
(shutting down the system), 3
(multiuser with
network) and 5
(multiuser with network and display
manager).
systemd introduces a new concept by using so-called “target
units”. However, it remains fully compatible with the runlevel
concept. Target units are named rather than numbered and serve specific
purposes. For example, the targets
local-fs.target
and
swap.target
mount local file systems and swap
spaces.
The target graphical.target
provides a
multiuser system with network and display manager capabilities and is
equivalent to runlevel 5. Complex targets, such as
graphical.target
act as “meta”
targets by combining a subset of other targets. Since systemd makes it
easy to create custom targets by combining existing targets, it offers
great flexibility.
The following list shows the most important systemd target units. For a
full list refer to man 7 systemd.special
.
default.target
The target that is booted by default. Not a “real”
target, but rather a symbolic link to another target like
graphic.target
. Can be permanently changed
via YaST (see Section 10.4, “Managing Services with YaST”). To
change it for a session, use the Kernel command line option
systemd.unit=<my_target>.target
at the boot prompt.
emergency.target
Starts an emergency shell on the console. Only use it at the boot
prompt as systemd.unit=emergency.target
.
graphical.target
Starts a system with network, multiuser support and a display manager.
halt.target
Shuts down the system.
mail-transfer-agent.target
Starts all services necessary for sending and receiving mails.
multi-user.target
Starts a multiuser system with network.
reboot.target
Reboots the system.
rescue.target
Starts a single-user system without network.
To remain compatible with the System V init runlevel system, systemd
provides special targets named
runlevelX.target
mapping
the corresponding runlevels numbered X.
If you want to know the current target, use the command:
systemctl get-default
systemd
Target Units #
System V runlevel |
|
Purpose |
---|---|---|
0 |
|
System shutdown |
1, S |
|
Single-user mode |
2 |
|
Local multiuser without remote network |
3 |
|
Full multiuser with network |
4 |
|
Unused/User-defined |
5 |
|
Full multiuser with network and display manager |
6 |
|
System reboot |
/etc/inittab
The runlevels in a System V init system are configured in
/etc/inittab
. systemd does
not use this configuration. Refer to
Section 10.5.3, “Creating Custom Targets” for instructions on
how to create your own bootable target.
Use the following commands to operate with target units:
Task |
systemd Command |
System V init Command |
---|---|---|
Change the current target/runlevel |
|
|
Change to the default target/runlevel |
|
n/a |
Get the current target/runlevel |
With systemd there is usually more than one active target. The command lists all currently active targets. |
or
|
persistently change the default runlevel |
Use the Services Manager or run the following command:
|
Use the Services Manager or change the line
in |
Change the default runlevel for the current boot process |
Enter the following option at the boot prompt
|
Enter the desired runlevel number at the boot prompt. |
Show a target's/runlevel's dependencies |
“Requires” lists the hard dependencies (the ones that must be resolved), whereas “Wants” lists the soft dependencies (the ones that get resolved if possible). |
n/a |
systemd offers the means to analyze the system start-up process. You can
conveniently review the list of all services and their status (rather
than having to parse /varlog/
). systemd also allows
you to scan the start-up procedure to find out how much time each
service start-up consumes.
To review the complete list of services that have been started since
booting the system, enter the command systemctl
. It
lists all active services like shown below (shortened). To get more
information on a specific service, use systemctl
status <my_service>
.
root #
systemctl
UNIT LOAD ACTIVE SUB JOB DESCRIPTION
[...]
iscsi.service loaded active exited Login and scanning of iSC+
kmod-static-nodes.service loaded active exited Create list of required s+
libvirtd.service loaded active running Virtualization daemon
nscd.service loaded active running Name Service Cache Daemon
ntpd.service loaded active running NTP Server Daemon
polkit.service loaded active running Authorization Manager
postfix.service loaded active running Postfix Mail Transport Ag+
rc-local.service loaded active exited /etc/init.d/boot.local Co+
rsyslog.service loaded active running System Logging Service
[...]
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
161 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
To restrict the output to services that failed to start, use the
--failed
option:
root #
systemctl --failed
UNIT LOAD ACTIVE SUB JOB DESCRIPTION
apache2.service loaded failed failed apache
NetworkManager.service loaded failed failed Network Manager
plymouth-start.service loaded failed failed Show Plymouth Boot Screen
[...]
To debug system start-up time, systemd offers the
systemd-analyze
command. It shows the total start-up
time, a list of services ordered by start-up time and can also generate
an SVG graphic showing the time services took to start in relation to
the other services.
root #
systemd-analyze
Startup finished in 2666ms (kernel) + 21961ms (userspace) = 24628ms
root #
systemd-analyze blame
6472ms systemd-modules-load.service
5833ms remount-rootfs.service
4597ms network.service
4254ms systemd-vconsole-setup.service
4096ms postfix.service
2998ms xdm.service
2483ms localnet.service
2470ms SuSEfirewall2_init.service
2189ms avahi-daemon.service
2120ms systemd-logind.service
1210ms xinetd.service
1080ms ntp.service
[...]
75ms fbset.service
72ms purge-kernels.service
47ms dev-vda1.swap
38ms bluez-coldplug.service
35ms splash_early.service
root #
systemd-analyze plot > jupiter.example.com-startup.svg
The above-mentioned commands let you review the services that started
and the time it took to start them. If you need to know more details,
you can tell systemd
to verbosely log the complete start-up
procedure by entering the following parameters at the boot prompt:
systemd.log_level=debug systemd.log_target=kmsg
Now systemd
writes its log messages into the kernel ring buffer.
View that buffer with dmesg
:
dmesg -T | less
Systemd is compatible with System V, allowing you to still use existing
System V init scripts. However, there is at least one known issue where
a System V init script does not work with Systemd out of the box:
starting a service as a different user via su
or
sudo
in init scripts will result in a failure of the
script, producing an “Access denied” error.
When changing the user with su
or
sudo
, a PAM session is started. This session will be
terminated after the init script is finished. As a consequence, the
service that has been started by the init script will also be
terminated. To work around this error, proceed as follows:
Create a service file wrapper with the same name as the init script
plus the file name extension .service
:
[Unit] Description=DESCRIPTION After=network.target [Service] User=USER Type=forking1 PIDFile=PATH TO PID FILE1 ExecStart=PATH TO INIT SCRIPT start ExecStop=PATH TO INIT SCRIPT stop ExecStopPost=/usr/bin/rm -f PATH TO PID FILE1 [Install] WantedBy=multi-user.target2
Replace all values written in UPPERCASE LETTERS with appropriate values.
Start the daemon with systemctl start
APPLICATION
.
Basic service management can also be done with the YaST Services Manager module. It supports starting, stopping, enabling and disabling services. It also lets you show a service's status and change the default target. Start the YaST module with
› › .To change the target the system boots into, choose a target from the
drop-down box. The most often used targets are (starting a graphical login screen) and (starting the system in command line mode).Select a service from the table. The
column shows whether it is currently running ( ) or not ( ). Toggle its status by choosing .Starting or stopping a service changes its status for the currently running session. To change its status throughout a reboot, you need to enable or disable it.
Select a service from the table. The
column shows whether it is currently or . Toggle its status by choosing .By enabling or disabling a service you configure whether it is started during booting (
) or not ( ). This setting will not affect the current session. To change its status in the current session, you need to start or stop it.
To view the status message of a service, select it from the list and
choose systemctl
-l
status
<my_service>.
Faulty runlevel settings may make your system unusable. Before applying your changes, make absolutely sure that you know their consequences.
systemd
#
The following sections contain some examples for
systemd
customizations.
Always do systemd customization in /etc/systemd/
,
never in /usr/lib/systemd/
.
Otherwise your changes will be overwritten by the next update of
systemd.
The systemd service files are located in
/usr/lib/systemd/system
. If you want to customize
them, proceed as follows:
Copy the files you want to modify from
/usr/lib/systemd/system
to
/etc/systemd/system
. Keep the file names
identical to the original ones.
Modify the copies in /etc/systemd/system
according to your needs.
For an overview of your configuration changes, use the
systemd-delta
command. It can compare and identify
configuration files that override other configuration files. For
details, refer to the systemd-delta
man page.
The modified files in /etc/systemd
will take
precedence over the original files in
/usr/lib/systemd/system
, provided that their file
name is the same.
If you only want to add a few lines to a configuration file or modify a small part of it, you can use so-called “drop-in” files. Drop-in files let you extend the configuration of unit files without having to edit or override the unit files themselves.
For example, to change one value for the foobar
service located in
/usr/lib/systemd/system/foobar.service
,
proceed as follows:
Create a directory called
/etc/systemd/system/<my_service>.service.d/
.
Note the .d
suffix. The directory must otherwise be
named like the service that you want to patch with the drop-in file.
In that directory, create a file
whatevermodification.conf
.
Make sure it only contains the line with the value that you want to modify.
Save your changes to the file. It will be used as an extension of the original file.
On System V init SUSE systems, runlevel 4 is unused to allow
administrators to create their own runlevel configuration. systemd
allows you to create any number of custom targets. It is suggested to
start by adapting an existing target such as
graphical.target
.
Copy the configuration file
/usr/lib/systemd/system/graphical.target
to
/etc/systemd/system/<my_target>.target
and adjust it according to your needs.
The configuration file copied in the previous step already covers the
required (“hard”) dependencies for the target. To also
cover the wanted (“soft”) dependencies, create a
directory
/etc/systemd/system/<my_target>.target.wants
.
For each wanted service, create a symbolic link from
/usr/lib/systemd/system
into
/etc/systemd/system/<my_target>.target.wants
.
Once you have finished setting up the target, reload the systemd configuration to make the new target available:
systemctl daemon-reload
The following sections cover advanced topics for system administrators. For even more advanced systemd documentation, refer to Lennart Pöttering's series about systemd for administrators at http://0pointer.de/blog/projects.
systemd
supports cleaning temporary directories regularly. The
configuration from the previous system version is automatically migrated
and active. tmpfiles.d
—which is responsible
for managing temporary files—reads its configuration from
/etc/tmpfiles.d/*.conf
,
/run/tmpfiles.d/*.conf
, and
/usr/lib/tmpfiles.d/*.conf
files. Configuration
placed in /etc/tmpfiles.d/*.conf
overrides related
configurations from the other two directories
(/usr/lib/tmpfiles.d/*.conf
is where packages store
their configuration files).
The configuration format is one line per path containing action and path, and optionally mode, ownership, age and argument fields, depending on the action. The following example unlinks the X11 lock files:
Type Path Mode UID GID Age Argument r /tmp/.X[0-9]*-lock
To get the status the tmpfile timer:
systemctl status systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.timer - Daily Cleanup of Temporary Directories Loaded: loaded (/usr/lib/systemd/system/systemd-tmpfiles-clean.timer; static) Active: active (waiting) since Tue 2014-09-09 15:30:36 CEST; 1 weeks 6 days ago Docs: man:tmpfiles.d(5) man:systemd-tmpfiles(8) Sep 09 15:30:36 jupiter systemd[1]: Starting Daily Cleanup of Temporary Directories. Sep 09 15:30:36 jupiter systemd[1]: Started Daily Cleanup of Temporary Directories.
For more information on temporary files handling, see man 5
tmpfiles.d
.
Section 10.6.8, “Debugging Services” explains
how to view log messages for a given service. However, displaying log
messages is not restricted to service logs. You can also access and
query the complete log messages written by systemd
—the
so-called “Journal”. Use the command
systemd-journalctl
to display the complete log
messages starting with the oldest entries. Refer to man 1
systemd-journalctl
for options such as applying filters or
changing the output format.
You can save the current state of systemd
to a named snapshot and
later revert to it with the isolate
subcommand. This
is useful when testing services or custom targets, because it allows you
to return to a defined state at any time. A snapshot is only available
in the current session and will automatically be deleted on reboot. A
snapshot name must end in .snapshot
.
systemctl snapshot <my_snapshot>.snapshot
systemctl delete <my_snapshot>.snapshot
systemctl show <my_snapshot>.snapshot
systemctl isolate <my_snapshot>.snapshot
With systemd
, kernel modules can automatically be loaded at boot
time via a configuration file in
/etc/modules-load.d
. The file should be named
module.conf and have the following content:
# load module module at boot time module
In case a package installes a configuration file for loading a Kernel
module, the file gets installed to
/usr/lib/modules-load.d
. If two configuration files
with the same name exist, the one in
/etc/modules-load.d
tales precedence.
For more information, see the modules-load.d(5)
man page.
With System V init actions that need to be performed before loading a
service, needed to be specified in /etc/init.d/before.local
. This procedure is no longer supported with systemd. If you
need to do actions before starting services, do the following:
Create a drop-in file in /etc/modules-load.d
directory (see man modules-load.d
for the syntax)
Create a drop-in file in /etc/tmpfiles.d
(see
man tmpfiles.d
for the syntax)
Create a system service file, for example
/etc/systemd/system/before.service
, from the
following template:
[Unit] Before=NAME OF THE SERVICE YOU WANT THIS SERVICE TO BE STARTED BEFORE [Service] Type=oneshot RemainAfterExit=true ExecStart=YOUR_COMMAND # beware, executable is run directly, not through a shell, check the man pages # systemd.service and systemd.unit for full syntax [Install] # target in which to start the service WantedBy=multi-user.target #WantedBy=graphical.target
When the service file is created, you should run the following
commands (as root
):
systemctl daemon-reload systemctl enable before
Every time you modify the service file, you need to run:
systemctl daemon-reload
On a traditional System V init system it is not always possible to clearly assign a process to the service that spawned it. Some services, such as Apache, spawn a lot of third-party processes (for example CGI or Java processes), which themselves spawn more processes. This makes a clear assignment difficult or even impossible. Additionally, a service may not terminate correctly, leaving some children alive.
systemd solves this problem by placing each service into its own cgroup. cgroups are a Kernel feature that allows aggregating processes and all their children into hierarchical organized groups. systemd names each cgroup after its service. Since a non-privileged process is not allowed to “leave” its cgroup, this provides an effective way to label all processes spawned by a service with the name of the service.
To list all processes belonging to a service, use the command
systemd-cgls
. The result will look like the following
(shortened) example:
root #
systemd-cgls --no-pager
├─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 20
├─user.slice
│ └─user-1000.slice
│ ├─session-102.scope
│ │ ├─12426 gdm-session-worker [pam/gdm-password]
│ │ ├─15831 gdm-session-worker [pam/gdm-password]
│ │ ├─15839 gdm-session-worker [pam/gdm-password]
│ │ ├─15858 /usr/lib/gnome-terminal-server
[...]
└─system.slice
├─systemd-hostnamed.service
│ └─17616 /usr/lib/systemd/systemd-hostnamed
├─cron.service
│ └─1689 /usr/sbin/cron -n
├─ntpd.service
│ └─1328 /usr/sbin/ntpd -p /var/run/ntp/ntpd.pid -g -u ntp:ntp -c /etc/ntp.conf
├─postfix.service
│ ├─ 1676 /usr/lib/postfix/master -w
│ ├─ 1679 qmgr -l -t fifo -u
│ └─15590 pickup -l -t fifo -u
├─sshd.service
│ └─1436 /usr/sbin/sshd -D
[...]
See Book “System Analysis and Tuning Guide”, Chapter 9 “Kernel Control Groups” for more information about cgroups.
As explained in Section 10.6.6, “Kernel Control Groups (cgroups)”, it is not always possible to assign a process to its parent service process in a System V init system. This makes it difficult to terminate a service and all of its children. Child processes that have not been terminated will remain as zombie processes.
systemd's concept of confining each service into a cgroup makes it
possible to clearly identify all child processes of a service and
therefore allows you to send a signal to each of these processes. Use
systemctl kill
to send signals to services. For a
list of available signals refer to man 7 signals
.
SIGTERM
to a Service
SIGTERM
is the default signal that is sent.
systemctl kill <my_service>
Use the -s
option to specify the signal that should
be sent.
systemctl kill -s SIGNAL <my_service>
By default the kill
command sends the signal to
all
processes of the specified cgroup. You can
restrict it to the control
or the
main
process. The latter is for example useful to
force a service to reload its configuration by sending
SIGHUP
:
systemctl kill -s SIGHUP --kill-who=main <my_service>
By default, systemd is not overly verbose. If a service was started
successfully, no output will be produced. In case of a failure, a short
error message will be displayed. However, systemctl
status
provides means to debug start-up and operation of a
service.
systemd comes with its own logging mechanism (“The
Journal”) that logs system messages. This allows you to display
the service messages together with status messages. The
status
command works similar to
tail
and can also display the log messages in
different formats, making it a powerful debugging tool.
Whenever a service fails to start, use systemctl status
<my_service>
to get a
detailed error message:
root #
systemctl start apache2 Job failed. See system journal and 'systemctl status' for details.root #
systemctl status apache2 Loaded: loaded (/usr/lib/systemd/system/apache2.service; disabled) Active: failed (Result: exit-code) since Mon, 04 Jun 2012 16:52:26 +0200; 29s ago Process: 3088 ExecStart=/usr/sbin/start_apache2 -D SYSTEMD -k start (code=exited, status=1/FAILURE) CGroup: name=systemd:/system/apache2.service Jun 04 16:52:26 g144 start_apache2[3088]: httpd2-prefork: Syntax error on line 205 of /etc/apache2/httpd.conf: Syntax error on li...alHost>
The default behavior of the status
subcommand is
to display the last ten messages a service issued. To change the
number of messages to show, use the
--lines=n
parameter:
systemctl status ntp systemctl --lines=20 status ntp
To display a “live stream” of service messages, use the
--follow
option, which works like
tail
-f
:
systemctl --follow status ntp
The --output=mode
parameter allows you to change the output format of service messages.
The most important modes available are:
short
The default format. Shows the log messages with a human readable time stamp.
verbose
Full output with all fields.
cat
Terse output without time stamps.
For more information on systemd refer to the following online resources:
Lennart Pöttering, one of the systemd authors, has written a series of blog entries (13 at the time of writing this chapter). Find them at http://0pointer.de/blog/projects.