To install KVM and the KVM tools, proceed as follows:

To install Xen and Xen tools, proceed as follows:

If you use VMware ESX as a guest hypervisor on top of KVM bare metal hypervisor, you may experience unstable network communication. This problem happens especially between nested KVM guests and the KVM bare metal hypervisor or external network. The following default CPU configuration of the nested KVM guest is causing the problem:

<cpu mode='host-model' check='partial'/>

To fix it, modify the CPU configuration as follow:

[...]
<cpu mode='host-passthrough' check='none'>
 <cache mode='passthrough'/>
</cpu>
[...]

The network bridge configuration provides a Layer 2 switch for VM Guests, switching Layer 2 Ethernet packets between ports on the bridge based on MAC addresses associated with the ports. This gives the VM Guest Layer 2 access to the VM Host Server's network. This configuration is analogous to connecting the VM Guest's virtual Ethernet cable into a hub that is shared with the host and other VM Guests running on the host. The configuration is often referred to as shared physical device.

The network bridge configuration is the default configuration of openSUSE Leap when configured as a KVM or Xen hypervisor. It is the preferred configuration when you simply want to connect VM Guests to the VM Host Server's LAN.

Which tool to use to create the network bridge depends on the service you use to manage the network connection on the VM Host Server:

libvirt-managed virtual networks are similar to bridged networks, but typically have no Layer 2 connection to the VM Host Server. Connectivity to the VM Host Server's physical network is accomplished with Layer 3 forwarding, which introduces additional packet processing on the VM Host Server as compared to a Layer 2 bridged network. Virtual networks also provide DHCP and DNS services for VM Guests. For more information on libvirt virtual networks, see the Network XML format documentation at https://libvirt.org/formatnetwork.html.

A standard libvirt installation on openSUSE Leap already comes with a predefined virtual network named default. It provides DHCP and DNS services for the network, along with connectivity to the VM Host Server's physical network using the network address translation (NAT) forwarding mode. Although it is predefined, the default virtual network needs to be explicitly enabled by the administrator. For more information on the forwarding modes supported by libvirt, see the Connectivity section of the Network XML format documentation at https://libvirt.org/formatnetwork.html#elementsConnect.

libvirt-managed virtual networks can be used to satisfy a wide range of use cases, but are commonly used on VM Host Servers that have a wireless connection or dynamic/sporadic network connectivity, such as laptops. Virtual networks are also useful when the VM Host Server's network has limited IP addresses, allowing forwarding of packets between the virtual network and the VM Host Server's network. However, most server use cases are better suited for the network bridge configuration, where VM Guests are connected to the VM Host Server's LAN.

Warning: Enabling forwarding mode

Enabling forwarding mode in a libvirt virtual network enables forwarding in the VM Host Server by setting /proc/sys/net/ipv4/ip_forward and /proc/sys/net/ipv6/conf/all/forwarding to 1, which essentially turns the VM Host Server into a router. Restarting the VM Host Server's network may reset the values and disable forwarding. To avoid this behavior, explicitly enable forwarding in the VM Host Server by editing the /etc/sysctl.conf file and adding:

net.ipv4.ip_forward = 1

net.ipv6.conf.all.forwarding = 1

> sudo virsh help network
Networking (help keyword 'network'):
 net-autostart                  autostart a network
        net-create                     create a network from an XML file
        net-define                     define (but don't start) a network from an XML file
        net-destroy                    destroy (stop) a network
        net-dumpxml                    network information in XML
        net-edit                       edit XML configuration for a network
        net-event                      Network Events
        net-info                       network information
        net-list                       list networks
        net-name                       convert a network UUID to network name
        net-start                      start a (previously defined) inactive network
        net-undefine                   undefine an inactive network
        net-update                     update parts of an existing network's configuration
 net-uuid                       convert a network name to network UUID

> sudo virsh help net-create
  NAME
    net-create - create a network from an XML file

  SYNOPSIS
    net-create <file>

  DESCRIPTION
    Create a network.

  OPTIONS
    [--file] <string>  file containing an XML network description

> sudo virsh net-create VNET_DEFINITION.xml

> sudo virsh net-define VNET_DEFINITION.xml

<network>
<name>vnet_nated</name>1
<bridge name="virbr1"/>2
 <forward mode="nat"/>3
 <ip address="192.168.122.1" netmask="255.255.255.0">4
  <dhcp>
   <range start="192.168.122.2" end="192.168.122.254"/>5
   <host mac="52:54:00:c7:92:da" name="host1.testing.com" \
    ip="192.168.1.23.101"/>6
   <host mac="52:54:00:c7:92:db" name="host2.testing.com" \
    ip="192.168.1.23.102"/>
   <host mac="52:54:00:c7:92:dc" name="host3.testing.com" \
    ip="192.168.1.23.103"/>
  </dhcp>
 </ip>
</network>

<network>
 <name>vnet_routed</name>
 <bridge name="virbr1"/>
 <forward mode="route" dev="eth1"/>1
 <ip address="192.168.122.1" netmask="255.255.255.0">
  <dhcp>
   <range start="192.168.122.2" end="192.168.122.254"/>
  </dhcp>
 </ip>
</network>

<network>
 <name>vnet_isolated</name>
 <bridge name="virbr3"/>
 <ip address="192.168.152.1" netmask="255.255.255.0">
  <dhcp>
   <range start="192.168.152.2" end="192.168.152.254"/>
  </dhcp>
 </ip>
 </network>

<network>
        <name>host-bridge</name>
        <forward mode="bridge"/>
        <bridge name="br0"/>
</network>

> sudo virsh net-list --all

 Name                 State      Autostart     Persistent
----------------------------------------------------------
 crowbar              active     yes           yes
 vnet_nated           active     yes           yes
 vnet_routed          active     yes           yes
 vnet_isolated        inactive   yes           yes

> sudo virsh list
 Id    Name                           State
----------------------------------------------------
 1     nated_sles12sp3                running
 ...

> sudo virsh domifaddr nated_sles12sp3 --interface vnet0 --source lease
 Name       MAC address          Protocol     Address
-------------------------------------------------------------------------------
 vnet0      52:54:00:9e:0d:2b    ipv6         fd00:dead:beef:55::140/64
 -          -                    ipv4         192.168.100.168/24

> sudo virsh domiflist nated_sles12sp3
Interface  Type       Source       Model       MAC
---------------------------------------------------------
vnet0      network    vnet_nated   virtio      52:54:00:9e:0d:2b

> sudo virsh net-info vnet_routed
Name:           vnet_routed
UUID:           756b48ff-d0c6-4c0a-804c-86c4c832a498
Active:         yes
Persistent:     yes
Autostart:      yes
Bridge:         virbr5

> sudo virsh net-list --inactive
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 vnet_isolated        inactive   yes           yes

> sudo virsh net-start vnet_isolated
Network vnet_isolated started

> sudo virsh net-list --inactive
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 vnet_isolated        active     yes           yes

> sudo virsh net-destroy vnet_isolated
Network vnet_isolated destroyed

> sudo virsh net-undefine vnet_isolated
Network vnet_isolated has been undefined

Managing storage from the command line is also possible by using virsh. However, creating storage pools is currently not supported by SUSE. Therefore, this section is restricted to documenting functions such as starting, stopping, and deleting pools, and volume management.

A list of all virsh subcommands for managing pools and volumes is available by running virsh help pool and virsh help volume, respectively.

> virsh pool-list --details

> virsh pool-info POOL

> virsh vol-list --details POOL

> virsh vol-create-as POOL1NAME2 12G --format3raw|qcow24 --allocation 4G5

> virsh vol-clone NAME_EXISTING_VOLUME1NAME_NEW_VOLUME2 --pool POOL3

8.2.1.4 Deleting volumes from a storage pool #Edit source

To permanently delete a volume from a pool, use the subcommand vol-delete:

> virsh vol-delete NAME --pool POOL

--pool is optional. libvirt tries to locate the volume automatically. If that fails, specify this parameter.

Warning: No checks upon volume deletion

A volume will be deleted in any case, regardless of whether it is currently used in an active or inactive VM Guest. There is no way to recover a deleted volume.

Whether a volume is used by a VM Guest can only be detected by using by the method described in Procedure 8.2, “Listing all storage volumes currently used on a VM Host Server”.

8.2.1.5 Attaching volumes to a VM Guest #Edit source

After you create a volume as described in Section 8.2.1.3, “Adding volumes to a storage pool”, you can attach it to a virtual machine and use it as a hard disk:

> virsh attach-disk DOMAIN SOURCE_IMAGE_FILE TARGET_DISK_DEVICE

For example:

> virsh attach-disk sles12sp3 /virt/images/example_disk.qcow2 sda2

To check if the new disk is attached, inspect the result of the virsh dumpxml command:

# virsh dumpxml sles12sp3
[...]
<disk type='file' device='disk'>
 <driver name='qemu' type='raw'/>
 <source file='/virt/images/example_disk.qcow2'/>
 <backingStore/>
 <target dev='sda2' bus='scsi'/>
 <alias name='scsi0-0-0'/>
 <address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
[...]

8.2.1.5.1 Hotplug or persistent change #Edit source

You can attach disks to both active and inactive domains. The attachment is controlled by the --live and --config options:

--live

Hotplugs the disk to an active domain. The attachment is not saved in the domain configuration. Using --live on an inactive domain is an error.

--config

Changes the domain configuration persistently. The attached disk is then available after the next domain start.

--live--config

Hotplugs the disk and adds it to the persistent domain configuration.

Tip: virsh attach-device

virsh attach-device is the more generic form of virsh attach-disk. You can use it to attach other types of devices to a domain.

# virsh detach-disk DOMAIN TARGET_DISK_DEVICE

# virsh detach-disk sles12sp3 sda2

The Virtual Machine Manager provides a graphical interface—the Storage Manager—to manage storage volumes and pools. To access it, either right-click a connection and choose Details, or highlight a connection and choose Edit › Connection Details. Select the Storage tab.

8.2.2.2 Managing storage pools #Edit source

Virtual Machine Manager's Storage Manager lets you create or delete volumes in a pool. You may also temporarily deactivate or permanently delete existing storage pools. Changing the basic configuration of a pool is currently not supported by SUSE.

8.2.2.2.1 Starting, stopping, and deleting pools #Edit source

The purpose of storage pools is to provide block devices located on the VM Host Server that can be added to a VM Guest when managing it from remote. To make a pool temporarily inaccessible from remote, click Stop in the bottom left corner of the Storage Manager. Stopped pools are marked with State: Inactive and are grayed out in the list pane. By default, a newly created pool will be automatically started On Boot of the VM Host Server.

To start an inactive pool and make it available from remote again, click Start in the bottom left corner of the Storage Manager.

Note: A pool's state does not affect attached volumes

Volumes from a pool attached to VM Guests are always available, regardless of the pool's state (Active (stopped) or Inactive (started)). The state of the pool solely affects the ability to attach volumes to a VM Guest via remote management.

To permanently make a pool inaccessible, click Delete in the bottom left corner of the Storage Manager. You can only delete inactive pools. Deleting a pool does not physically erase its contents on VM Host Server—it only deletes the pool configuration. However, you need to be extra careful when deleting pools, especially when deleting LVM volume group-based tools:

Warning: Deleting storage pools

Deleting storage pools based on local file system directories, local partitions or disks has no effect on the availability of volumes from these pools currently attached to VM Guests.

Volumes located in pools of type iSCSI, SCSI, LVM group or Network Exported Directory will become inaccessible from the VM Guest if the pool is deleted. Although the volumes themselves will not be deleted, the VM Host Server will no longer have access to the resources.

Volumes on iSCSI/SCSI targets or Network Exported Directory will become accessible again when creating an adequate new pool or when mounting/accessing these resources directly from the host system.

When deleting an LVM group-based storage pool, the LVM group definition will be erased and the LVM group will no longer exist on the host system. The configuration is not recoverable and all volumes from this pool are lost.

8.2.2.2.2 Adding volumes to a storage pool #Edit source

Virtual Machine Manager lets you create volumes in all storage pools, except in pools of types Multipath, iSCSI, or SCSI. A volume in these pools is equivalent to a LUN and cannot be changed from within libvirt.

1. A new volume can either be created using the Storage Manager or while adding a new storage device to a VM Guest. In either case, select a storage pool from the left panel, then click Create new volume.

2. Specify a Name for the image and choose an image format.

   SUSE currently only supports raw or qcow2 images. The latter option is not available on LVM group-based pools.

   Next to Max Capacity, specify the maximum size that the disk image is allowed to reach. Unless you are working with a qcow2 image, you can also set an amount for Allocation that should be allocated initially. If the two values differ, a sparse image file will be created, which grows on demand.

   For qcow2 images, you can use a Backing Store (also called “backing file”), which constitutes a base image. The newly created qcow2 image will then only record the changes that are made to the base image.

3. Start the volume creation by clicking Finish.

8.2.2.2.3 Deleting volumes from a storage pool #Edit source

Deleting a volume can only be done from the Storage Manager, by selecting a volume and clicking Delete Volume. Confirm with Yes.

Warning: Volumes can be deleted even while in use

Volumes can be deleted even if they are currently used in an active or inactive VM Guest. There is no way to recover a deleted volume.

Whether a volume is used by a VM Guest is indicated in the Used By column in the Storage Manager.

PXE boot enables your virtual machine to boot the installation media from the network, instead of from a physical medium or an installation disk image.

To let your VM boot from a PXE server, follow these steps:

Some operating systems such as openSUSE Leap offer to include add-on products in the installation process. If the add-on product installation source is provided via SUSE Customer Center, no special VM Guest configuration is needed. If it is provided via CD/DVD or ISO image, it is necessary to provide the VM Guest installation system with both the standard installation medium image and the image of the add-on product.

If you are using the GUI-based installation, select Customize Configuration Before Install in the last step of the wizard and add the add-on product ISO image via Add Hardware › Storage. Specify the path to the image and set the Device Type to CD-ROM.

If you are installing from the command line, you need to set up the virtual CD/DVD drives with the --disk parameter rather than with --cdrom. The device that is specified first is used for booting. The following example will install SUSE Linux Enterprise Server 15 together with SUSE Enterprise Storage extension:

> virt-install \
 --name sles15+storage \
 --memory 2048 --disk size=10 \
 --disk /path/to/SLE-15-SP3-Full-ARCH-GM-media1.iso-x86_64-GM-DVD1.iso,device=cdrom \
 --disk /path/to/SUSE-Enterprise-Storage-VERSION-DVD-ARCH-Media1.iso,device=cdrom \
 --graphics vnc --os-variant sle15

The main window of the Virtual Machine Manager lists all VM Guests for each VM Host Server it is connected to. Each VM Guest entry contains the machine's name, its status (Running, Paused, or Shutoff) displayed as an icon and literally, and a CPU usage bar.

Use the command virsh list to get a list of VM Guests:

For more information and further options, see virsh help list or man 1 virsh.

Opening a graphical console to a VM Guest lets you interact with the machine like a physical host via a VNC connection. If accessing the VNC server requires authentication, you are prompted to enter a user name (if applicable) and a password.

When you click into the VNC console, the cursor is “grabbed” and cannot be used outside the console anymore. To release it, press Alt–Ctrl.

Tip: Seamless (absolute) cursor movement

To prevent the console from grabbing the cursor and to enable seamless cursor movement, add a tablet input device to the VM Guest. See Section 13.5, “Input devices” for more information.

Certain key combinations such as Ctrl–Alt–Del are interpreted by the host system and are not passed to the VM Guest. To pass such key combinations to a VM Guest, open the Send Key menu from the VNC window and choose the desired key combination entry. The Send Key menu is only available when using Virtual Machine Manager and virt-viewer. With Virtual Machine Manager, you can alternatively use the “sticky key” feature as explained in Tip: Passing key combinations to virtual machines.

Note: Supported VNC viewers

Principally all VNC viewers can connect to the console of a VM Guest. However, if you are using SASL authentication and/or TLS/SSL connection to access the guest, the options are limited. Common VNC viewers such as tightvnc or tigervnc support neither SASL authentication nor TLS/SSL. The only supported alternative to Virtual Machine Manager and virt-viewer is Remmina (refer to Book “Reference”, Chapter 4 “Remote graphical sessions with VNC”, Section 4.2 “Remmina: the remote desktop client”).

10.2.1.2 Opening a graphical console with virt-viewer #Edit source

virt-viewer is a simple VNC viewer with added functionality for displaying VM Guest consoles. For example, it can be started in “wait” mode, where it waits for a VM Guest to start before it connects. It also supports automatically reconnecting to a VM Guest that is rebooted.

virt-viewer addresses VM Guests by name, by ID or by UUID. Use virsh list --all to get this data.

To connect to a guest that is running or paused, use either the ID, UUID, or name. VM Guests that are shut off do not have an ID—you can only connect to them by UUID or name.

Connect to guest with the ID 8

> virt-viewer 8

Connect to the inactive guest named sles12; the connection window will open once the guest starts

> virt-viewer --wait sles12

With the --wait option, the connection will be upheld even if the VM Guest is not running at the moment. When the guest starts, the viewer will be launched.

For more information, see virt-viewer --help or man 1 virt-viewer.

Note: Password input on remote connections with SSH

When using virt-viewer to open a connection to a remote host via SSH, the SSH password needs to be entered twice. The first time for authenticating with libvirt, the second time for authenticating with the VNC server. The second password needs to be provided on the command line where virt-viewer was started.

Accessing the graphical console of a virtual machine requires a graphical environment on the client accessing the VM Guest. As an alternative, virtual machines managed with libvirt can also be accessed from the shell via the serial console and virsh. To open a serial console to a VM Guest named “sles12”, run the following command:

> virsh console sles12

virsh console takes two optional flags: --safe ensures exclusive access to the console, --force disconnects any existing sessions before connecting. Both features need to be supported by the guest operating system.

Being able to connect to a VM Guest via serial console requires that the guest operating system supports serial console access and is properly supported. Refer to the guest operating system manual for more information.

Tip: Enabling serial console access for SUSE Linux Enterprise and openSUSE guests

Serial console access in SUSE Linux Enterprise and openSUSE is disabled by default. To enable it, proceed as follows:

SLES 12 and up/openSUSE

Launch the YaST Boot Loader module and switch to the Kernel Parameters tab. Add console=ttyS0 to the field Optional Kernel Command Line Parameter.

SLES 11

Launch the YaST Boot Loader module and select the boot entry for which to activate serial console access. Choose Edit and add console=ttyS0 to the field Optional Kernel Command Line Parameter. Additionally, edit /etc/inittab and uncomment the line with the following content:

#S0:12345:respawn:/sbin/agetty -L 9600 ttyS0 vt102

Changing a VM Guest's state can be done either from Virtual Machine Manager's main window, or from a VNC window.

In the following examples, the state of a VM Guest named “sles12” is changed.

Save a running VM Guest with the command virsh save and specify the file which it is saved to.

To restore a VM Guest, use virsh restore:

> virsh restore /virtual/saves/opensuse13.vmsave

There are several specific terms used to describe the types of snapshots:

Important: Internal snapshots only

Virtual Machine Manager supports only internal snapshots, either live or offline.

To open the snapshot management view in Virtual Machine Manager, open the VNC window as described in Section 10.2.1.1, “Opening a graphical console with Virtual Machine Manager”. Now either choose View › Snapshots or click Manage VM Snapshots in the toolbar.

The list of existing snapshots for the chosen VM Guest is displayed in the left-hand part of the window. The snapshot that was last started is marked with a green tick. The right-hand part of the window shows details of the snapshot currently marked in the list. These details include the snapshot's title and time stamp, the state of the VM Guest at the time the snapshot was taken and a description. Snapshots of running guests also include a screenshot. The Description can be changed directly from this view. Other snapshot data cannot be changed.

To list all existing snapshots for a domain (admin_server in the following), run the snapshot-list command:

> virsh snapshot-list --domain sle-ha-node1
 Name                 Creation Time             State
------------------------------------------------------------
 sleha_12_sp2_b2_two_node_cluster 2016-06-06 15:04:31 +0200 shutoff
 sleha_12_sp2_b3_two_node_cluster 2016-07-04 14:01:41 +0200 shutoff
 sleha_12_sp2_b4_two_node_cluster 2016-07-14 10:44:51 +0200 shutoff
 sleha_12_sp2_rc3_two_node_cluster 2016-10-10 09:40:12 +0200 shutoff
 sleha_12_sp2_gmc_two_node_cluster 2016-10-24 17:00:14 +0200 shutoff
 sleha_12_sp3_gm_two_node_cluster 2017-08-02 12:19:37 +0200 shutoff
 sleha_12_sp3_rc1_two_node_cluster 2017-06-13 13:34:19 +0200 shutoff
 sleha_12_sp3_rc2_two_node_cluster 2017-06-30 11:51:24 +0200 shutoff
 sleha_15_b6_two_node_cluster 2018-02-07 15:08:09 +0100 shutoff
 sleha_15_rc1_one-node 2018-03-09 16:32:38 +0100 shutoff

The snapshot that was last started is shown with the snapshot-current command:

> virsh snapshot-current --domain admin_server
Basic installation incl. SMT for CLOUD4

Details about a particular snapshot can be obtained by running the snapshot-info command:

> virsh snapshot-info --domain admin_server \
   -name  "Basic installation incl. SMT for CLOUD4"
Name:           Basic installation incl. SMT for CLOUD4
Domain:         admin_server
Current:        yes
State:          shutoff
Location:       internal
Parent:         Basic installation incl. SMT for CLOUD3-HA
Children:       0
Descendants:    0
Metadata:       yes

> virsh snapshot-create-as --domain admin_server1 --name "Snapshot 1"2 \
--description "First snapshot"3

> virsh snapshot-create-as --domain admin_server --name \
 "Offline external snapshot" --disk-only

> virsh snapshot-create-as --domain admin_server --name \
 "Offline external snapshot" \
 --disk-only --diskspec vda,snapshot=external,file=/path/to/snapshot_file

> virsh snapshot-create-as --domain admin_server --name \
 "Offline external snapshot" --live \
 --memspec snapshot=external,file=/path/to/snapshot_file

> virsh snapshot-create-as --domain admin_server --name \
 "Offline external snapshot" --live \
 --memspec snapshot=external,file=/path/to/snapshot_file
 --diskspec vda,snapshot=external,file=/path/to/snapshot_file

> virsh snapshot-delete --domain admin_server --snapshotname "Snapshot 2"

> virsh snapshot-revert --domain admin_server --snapshotname "Snapshot 1"

> virsh snapshot-revert --domain admin_server --current

To delete a VM Guest, it needs to be shut down first. It is not possible to delete a running guest. For information on shutting down, see Section 10.3, “Changing a VM Guest's state: start, stop, pause”.

To delete a VM Guest with virsh, run virsh undefine VM_NAME.

> virsh undefine sles12

There is no option to automatically delete the attached storage files. If they are managed by libvirt, delete them as described in Section 8.2.1.4, “Deleting volumes from a storage pool”.

To successfully migrate a VM Guest to another VM Host Server, the following requirements need to be met:

When using the Virtual Machine Manager to migrate VM Guests, it does not matter on which machine it is started. You can start Virtual Machine Manager on the source or the target host or even on a third host. In the latter case you need to be able to open remote connections to both the target and the source host.

To migrate a VM Guest with virsh migrate, you need to have direct or remote shell access to the VM Host Server, because the command needs to be run on the host. The migration command looks like this:

> virsh migrate [OPTIONS] VM_ID_or_NAME CONNECTION_URI [--migrateuri tcp://REMOTE_HOST:PORT]

The most important options are listed below. See virsh help migrate for a full list.

The following examples use mercury.example.com as the source system and jupiter.example.com as the target system; the VM Guest's name is opensuse131 with Id 37.

Note: Transient compared to persistent migrations

By default virsh migrate creates a temporary (transient) copy of the VM Guest on the target host. A shut down version of the original guest description remains on the source host. A transient copy will be deleted from the server after it is shut down.

To create a permanent copy of a guest on the target host, use the switch --persistent. A shut down version of the original guest description remains on the source host, too. Use the option --undefinesource together with --persistent for a “real” move where a permanent copy is created on the target host and the version on the source host is deleted.

It is not recommended to use --undefinesource without the --persistent option, since this will result in the loss of both VM Guest definitions when the guest is shut down on the target host.

After starting Virtual Machine Manager and connecting to the VM Host Server, a CPU usage graph of all the running guests is displayed.

It is also possible to get information about disk and network usage with this tool, however, you must first activate this in Preferences:

Afterward, the disk and network statistics are also displayed in the main window of the Virtual Machine Manager.

More precise data is available from the VNC window. Open a VNC window as described in Section 10.2.1, “Opening a graphical console”. Choose Details from the toolbar or the View menu. The statistics are displayed from the Performance entry of the left-hand tree menu.

virt-top is a command line tool similar to the well-known process monitoring tool top. virt-top uses libvirt and therefore is capable of showing statistics for VM Guests running on different hypervisors. It is recommended to use virt-top instead of hypervisor-specific tools like xentop.

By default virt-top shows statistics for all running VM Guests. Among the data that is displayed is the percentage of memory used (%MEM) and CPU (%CPU) and the uptime of the guest (TIME). The data is updated regularly (every three seconds by default). The following shows the output on a VM Host Server with seven VM Guests, four of them inactive:

virt-top 13:40:19 - x86_64 8/8CPU 1283MHz 16067MB 7.6% 0.5%
7 domains, 3 active, 3 running, 0 sleeping, 0 paused, 4 inactive D:0 O:0 X:0
CPU: 6.1%  Mem: 3072 MB (3072 MB by guests)

   ID S RDRQ WRRQ RXBY TXBY %CPU %MEM    TIME   NAME
    7 R  123    1  18K  196  5.8  6.0   0:24.35 sled12_sp1
    6 R    1    0  18K    0  0.2  6.0   0:42.51 sles12_sp1
    5 R    0    0  18K    0  0.1  6.0  85:45.67 opensuse_leap
    -                                           (Ubuntu_1410)
    -                                           (debian_780)
    -                                           (fedora_21)
    -                                           (sles11sp3)

By default the output is sorted by ID. Use the following key combinations to change the sort field:

To use any other field for sorting, press Shift–F and select a field from the list. To toggle the sort order, use Shift–R.

virt-top also supports different views on the VM Guests data, which can be changed on-the-fly by pressing the following keys:

virt-top supports more hot keys to change the view of the data and many command line switches that affect the behavior of the program. For more information, see man 1 virt-top.

kvm_stat can be used to trace KVM performance events. It monitors /sys/kernel/debug/kvm, so it needs the debugfs to be mounted. On openSUSE Leap it should be mounted by default. In case it is not mounted, use the following command:

> sudo mount -t debugfs none /sys/kernel/debug

kvm_stat can be used in three different modes:

kvm_stat                    # update in 1 second intervals
kvm_stat -1                 # 1 second snapshot
kvm_stat -l > kvmstats.log  # update in 1 second intervals in log format
                            # can be imported to a spreadsheet

kvm statistics

 efer_reload                  0       0
 exits                 11378946  218130
 fpu_reload               62144     152
 halt_exits              414866     100
 halt_wakeup             260358      50
 host_state_reload       539650     249
 hypercalls                   0       0
 insn_emulation         6227331  173067
 insn_emulation_fail          0       0
 invlpg                  227281      47
 io_exits                113148      18
 irq_exits               168474     127
 irq_injections          482804     123
 irq_window               51270      18
 largepages                   0       0
 mmio_exits                6925       0
 mmu_cache_miss           71820      19
 mmu_flooded              35420       9
 mmu_pde_zapped           64763      20
 mmu_pte_updated              0       0
 mmu_pte_write           213782      29
 mmu_recycled                 0       0
 mmu_shadow_zapped       128690      17
 mmu_unsync                  46      -1
 nmi_injections               0       0
 nmi_window                   0       0
 pf_fixed               1553821     857
 pf_guest               1018832     562
 remote_tlb_flush        174007      37
 request_irq                  0       0
 signal_exits                 0       0
 tlb_flush               394182     148

See http://clalance.blogspot.com/2009/01/kvm-performance-tools.html for further information on how to interpret these values.

libvirtd authentication is configured in /etc/libvirt/libvirtd.conf. The configuration made here applies to all libvirt tools such as the Virtual Machine Manager or virsh.

libvirt offers two sockets: a read-only socket for monitoring purposes and a read-write socket to be used for management operations. Access to both sockets can be configured independently. By default, both sockets are owned by root.root. Default access permissions on the read-write socket are restricted to the user root (0700) and fully open on the read-only socket (0777).

In the following instructions, you will learn how to configure access permissions for the read-write socket. The same instructions also apply to the read-only socket. All configuration steps need to be carried out on the VM Host Server.

Note: Default authentication settings on openSUSE Leap

The default authentication method on openSUSE Leap is access control for Unix sockets. Only the user root may authenticate. When accessing the libvirt tools as a non-root user directly on the VM Host Server, you need to provide the root password through PolKit once. You are then granted access for the current and for future sessions.

Alternatively, you can configure libvirt to allow “system” access to non-privileged users. See Section 11.2.1, ““system” access for non-privileged users” for details.

11.1.1.2 Local access control for Unix sockets with PolKit #Edit source

Access control for Unix sockets with PolKit is the default authentication method on openSUSE Leap for non-remote connections. Therefore, no libvirt configuration changes are needed. With PolKit authorization enabled, permissions on both sockets default to 0777 and each application trying to access a socket needs to authenticate via PolKit.

Important: PolKit authentication for local connections only

Authentication with PolKit can only be used for local connections on the VM Host Server itself, since PolKit does not handle remote authentication.

Two policies for accessing libvirt's sockets exist:

• org.libvirt.unix.monitor: accessing the read-only socket

• org.libvirt.unix.manage: accessing the read-write socket

By default, the policy for accessing the read-write socket is to authenticate with the root password once and grant the privilege for the current and for future sessions.

To grant users access to a socket without having to provide the root password, you need to create a rule in /etc/polkit-1/rules.d. Create the file /etc/polkit-1/rules.d/10-grant-libvirt with the following content to grant access to the read-write socket to all members of the group libvirt:

polkit.addRule(function(action, subject) {
  if (action.id == "org.libvirt.unix.manage" && subject.isInGroup("libvirt")) {
    return polkit.Result.YES;
  }
});

11.1.1.3 User name and password authentication with SASL #Edit source

SASL provides user name and password authentication and data encryption (digest-md5, by default). Since SASL maintains its own user database, the users do not need to exist on the VM Host Server. SASL is required by TCP connections and on top of TLS/SSL connections.

Important: Plain TCP and SASL with digest-md5 encryption

Using digest-md5 encryption on an otherwise not encrypted TCP connection does not provide enough security for production environments. It is recommended to only use it in testing environments.

Tip: SASL authentication on top of TLS/SSL

Access from remote TLS/SSL connections can be indirectly controlled on the client side by restricting access to the certificate's key file. However, this might prove error-prone when dealing with many clients. Using SASL with TLS adds security by additionally controlling access on the server side.

To configure SASL authentication, proceed as follows:

1. Change the configuration in /etc/libvirt/libvirtd.conf as follows:

   1. To enable SASL for TCP connections:

      auth_tcp = "sasl"

   2. To enable SASL for TLS/SSL connections:

      auth_tls = "sasl"

2. Restart libvirtd:

   > sudo systemctl restart libvirtd

3. The libvirt SASL configuration file is located at /etc/sasl2/libvirtd.conf. Normally, there is no need to change the defaults. However, if using SASL on top of TLS, you may turn off session encryption to avoid additional overhead (TLS connections are already encrypted) by commenting the line setting the mech_list parameter. Only do this for TLS/SASL, for TCP connections this parameter must be set to digest-md5.

   #mech_list: digest-md5

4. By default, no SASL users are configured, so no logins are possible. Use the following commands to manage users:

   Add the user tux

   saslpasswd2 -a libvirt tux

   Delete the user tux

   saslpasswd2 -a libvirt -d tux

   List existing users

   sasldblistusers2 -f /etc/libvirt/passwd.db

Tip: virsh and SASL authentication

When using SASL authentication, you will be prompted for a user name and password every time you issue a virsh command. Avoid this by using virsh in shell mode.

Since access to the graphical console of a VM Guest is not controlled by libvirt, but rather by the specific hypervisor, it is always necessary to additionally configure VNC authentication. The main configuration file is /etc/libvirt/<hypervisor>.conf. This section describes the QEMU/KVM hypervisor, so the target configuration file is /etc/libvirt/qemu.conf.

Note: VNC authentication for Xen

In contrast with KVM, Xen does not yet offer VNC authentication more sophisticated than setting a password on a per-VM basis. See the <graphics type='vnc'... libvirt configuration option below.

Two authentication types are available: SASL and single-password authentication. If you are using SASL for libvirt authentication, it is strongly recommended to use it for VNC authentication as well—it is possible to share the same database.

A third method to restrict access to the VM Guest is to enable the use of TLS encryption on the VNC server. This requires the VNC clients to have access to x509 client certificates. By restricting access to these certificates, access can indirectly be controlled on the client side. Refer to Section 11.3.2.4.2, “VNC over TLS/SSL: client configuration” for details.

11.1.2.1 User name and password authentication with SASL #Edit source

SASL provides user name and password authentication and data encryption. Since SASL maintains its own user database, the users do not need to exist on the VM Host Server. As with SASL authentication for libvirt, you may use SASL on top of TLS/SSL connections. Refer to Section 11.3.2.4.2, “VNC over TLS/SSL: client configuration” for details on configuring these connections.

To configure SASL authentication for VNC, proceed as follows:

1. Create a SASL configuration file. It is recommended to use the existing libvirt file. If you have already configured SASL for libvirt and are planning to use the same settings including the same user name and password database, a simple link is suitable:

   > sudo ln -s /etc/sasl2/libvirt.conf /etc/sasl2/qemu.conf

   If are setting up SASL for VNC only or you are planning to use a different configuration than for libvirt, copy the existing file to use as a template:

   > sudo cp /etc/sasl2/libvirt.conf /etc/sasl2/qemu.conf

   Then edit it according to your needs.

2. Change the configuration in /etc/libvirt/qemu.conf as follows:

   vnc_listen = "0.0.0.0"
   vnc_sasl = 1
   sasldb_path: /etc/libvirt/qemu_passwd.db

   The first parameter enables VNC to listen on all public interfaces (rather than to the local host only), and the second parameter enables SASL authentication.

3. By default, no SASL users are configured, so no logins are possible. Use the following commands to manage users:

   Add the user tux

   > saslpasswd2 -f /etc/libvirt/qemu_passwd.db -a qemu tux

   Delete the user tux

   > saslpasswd2 -f /etc/libvirt/qemu_passwd.db -a qemu -d tux

   List existing users

   > sasldblistusers2 -f /etc/libvirt/qemu_passwd.db

4. Restart libvirtd:

   > sudo systemctl restart libvirtd

5. Restart all VM Guests that have been running prior to changing the configuration. VM Guests that have not been restarted will not use SASL authentication for VNC connects.

Note: Supported VNC viewers

SASL authentication is currently supported by Virtual Machine Manager and virt-viewer. Both of these viewers also support TLS/SSL connections.

11.1.2.2 Single password authentication #Edit source

Access to the VNC server may also be controlled by setting a VNC password. You can either set a global password for all VM Guests or set individual passwords for each guest. The latter requires to edit the VM Guest's configuration files.

Note: Always set a global password

If you are using single password authentication, it is good practice to set a global password even if setting passwords for each VM Guest. This will always leave your virtual machines protected with a “fallback” password if you forget to set a per-machine password. The global password will only be used if no other password is set for the machine.

Procedure 11.1: Setting a global VNC password #

1. Change the configuration in /etc/libvirt/qemu.conf as follows:

   vnc_listen = "0.0.0.0"
   vnc_password = "PASSWORD"

   The first parameter enables VNC to listen on all public interfaces (rather than to the local host only), and the second parameter sets the password. The maximum length of the password is eight characters.

2. Restart libvirtd:

   > sudo systemctl restart libvirtd

3. Restart all VM Guests that have been running prior to changing the configuration. VM Guests that have not been restarted will not use password authentication for VNC connects.

Procedure 11.2: Setting a VM Guest specific VNC password #

1. Change the configuration in /etc/libvirt/qemu.conf as follows to enable VNC to listen on all public interfaces (rather than to the local host only).

   vnc_listen = "0.0.0.0"

2. Open the VM Guest's XML configuration file in an editor. Replace VM_NAME in the following example with the name of the VM Guest. The editor that is used defaults to $EDITOR. If that variable is not set, vi is used.

   > virsh edit VM_NAME

3. Search for the element <graphics> with the attribute type='vnc', for example:

   <graphics type='vnc' port='-1' autoport='yes'/>

4. Add the passwd=PASSWORD attribute, save the file and exit the editor. The maximum length of the password is eight characters.

   <graphics type='vnc' port='-1' autoport='yes' passwd='PASSWORD'/>

5. Restart libvirtd:

   > sudo systemctl restart libvirtd

6. Restart all VM Guests that have been running prior to changing the configuration. VM Guests that have not been restarted will not use password authentication for VNC connects.

Warning: Security of the VNC protocol

The VNC protocol is not considered to be safe. Although the password is sent encrypted, it might be vulnerable when an attacker can sniff both the encrypted password and the encryption key. Therefore, it is recommended to use VNC with TLS/SSL or tunneled over SSH. virt-viewer, Virtual Machine Manager and Remmina (refer to Book “Reference”, Chapter 4 “Remote graphical sessions with VNC”, Section 4.2 “Remmina: the remote desktop client”) support both methods.

As mentioned above, a connection to the QEMU hypervisor can be established using two different protocols: session and system. A “session” connection is spawned with the same privileges as the client program. Such a connection is intended for desktop virtualization, since it is restricted (for example no USB/PCI device assignments, no virtual network setup, limited remote access to libvirtd).

The “system” connection intended for server virtualization has no functional restrictions but is, by default, only accessible by root. However, with the addition of the DAC (Discretionary Access Control) driver to libvirt it is now possible to grant non-privileged users “system” access. To grant “system” access to the user tux, proceed as follows:

The Virtual Machine Manager uses a Connection for every VM Host Server it manages. Each connection contains all VM Guests on the respective host. By default, a connection to the local host is already configured and connected.

All configured connections are displayed in the Virtual Machine Manager main window. Active connections are marked with a small triangle, which you can click to fold or unfold the list of VM Guests for this connection.

Inactive connections are listed gray and are marked with Not Connected. Either double-click or right-click it and choose Connect from the context menu. You can also Delete an existing connection from this menu.

Note: Editing existing connections

It is not possible to edit an existing connection. To change a connection, create a new one with the desired parameters and delete the “old” one.

To add a new connection in the Virtual Machine Manager, proceed as follows:

Enabling a remote connection that is tunneled over SSH on the VM Host Server only requires the ability to accept SSH connections. Make sure the SSH daemon is started (systemctl status sshd) and that the ports for service SSH are opened in the firewall.

User authentication for SSH connections can be done using traditional file user/group ownership and permissions as described in Section 11.1.1.1, “Access control for Unix sockets with permissions and group ownership”. Connecting as user tux (qemu+ssh://tuxsIVname;/system or xen+ssh://tuxsIVname;/system) works out of the box and does not require additional configuration on the libvirt side.

When connecting via SSH qemu+ssh://USER@SYSTEM or xen+ssh://USER@SYSTEM you need to provide the password for USER. This can be avoided by copying your public key to ~USER/.ssh/authorized_keys on the VM Host Server as explained in Book “Security and Hardening Guide”, Chapter 23 “Securing network operations with OpenSSH”, Section 23.7.2 “Copying an SSH key”. Using an ssh-agent on the machine from which you are connecting adds even more convenience. For more information, see Book “Security and Hardening Guide”, Chapter 23 “Securing network operations with OpenSSH”, Section 23.7.3 “Using the ssh-agent”.

Using TCP connections with TLS/SSL encryption and authentication via x509 certificates is much more complicated to set up than SSH, but it is a lot more scalable. Use this method if you need to manage several VM Host Servers with a varying number of administrators.

11.3.2.1 Basic concept #Edit source

TLS (Transport Layer Security) encrypts the communication between two computers by using certificates. The computer starting the connection is always considered the “client”, using a “client certificate”, while the receiving computer is always considered the “server”, using a “server certificate”. This scenario applies, for example, if you manage your VM Host Servers from a central desktop.

If connections are initiated from both computers, each needs to have a client and a server certificate. This is the case, for example, if you migrate a VM Guest from one host to another.

Each x509 certificate has a matching private key file. Only the combination of certificate and private key file can identify itself correctly. To assure that a certificate was issued by the assumed owner, it is signed and issued by a central certificate called certificate authority (CA). Both the client and the server certificates must be issued by the same CA.

Important: User authentication

Using a remote TLS/SSL connection only ensures that two computers are allowed to communicate in a certain direction. Restricting access to certain users can indirectly be achieved on the client side by restricting access to the certificates. For more information, see Section 11.3.2.5, “Restricting access (security considerations)”.

libvirt also supports user authentication on the server with SASL. For more information, see Section 11.3.2.6, “Central user authentication with SASL for TLS sockets”.

11.3.2.2 Configuring the VM Host Server #Edit source

The VM Host Server is the machine receiving connections. Therefore, the server certificates need to be installed. The CA certificate needs to be installed, too. When the certificates are in place, TLS support can be turned on for libvirt.

1. Create the server certificate and export it together with the respective CA certificate.

2. Create the following directories on the VM Host Server:

   > sudo mkdir -p /etc/pki/CA/ /etc/pki/libvirt/private/

   Install the certificates as follows:

   > sudo /etc/pki/CA/cacert.pem
   > sudo /etc/pki/libvirt/servercert.pem
   > sudo /etc/pki/libvirt/private/serverkey.pem

   

   Important: Restrict access to certificates

   Make sure to restrict access to certificates as explained in Section 11.3.2.5, “Restricting access (security considerations)”.

3. Enable TLS support by enabling the relevant socket and restarting libvirtd:

   > sudo systemctl stop libvirtd.service
   > sudo systemctl enable --now libvirtd-tls.socket
   > sudo systemctl start libvirtd.service

4. By default, libvirt uses the TCP port 16514 for accepting secure TLS connections. Open this port in the firewall.

Important: Restarting libvirtd with TLS enabled

If you enable TLS for libvirt, the server certificates need to be in place, otherwise restarting libvirtd will fail. You also need to restart libvirtd in case you change the certificates.

11.3.2.4 Enabling VNC for TLS/SSL connections #Edit source

Currently, VNC communication over TLS is only supported by a few tools. Common VNC viewers such as tightvnc or tigervnc do not support TLS/SSL. The only supported alternative to Virtual Machine Manager and virt-viewer is remmina (refer to Book “Reference”, Chapter 4 “Remote graphical sessions with VNC”, Section 4.2 “Remmina: the remote desktop client”).

11.3.2.4.1 VNC over TLS/SSL: VM Host Server configuration #Edit source

To access the graphical console via VNC over TLS/SSL, you need to configure the VM Host Server as follows:

1. Open ports for the service VNC in your firewall.

2. Create a directory /etc/pki/libvirt-vnc and link the certificates into this directory as follows:

   > sudo mkdir -p /etc/pki/libvirt-vnc && cd /etc/pki/libvirt-vnc
   > sudo ln -s /etc/pki/CA/cacert.pem ca-cert.pem
   > sudo ln -s /etc/pki/libvirt/servercert.pem server-cert.pem
   > sudo ln -s /etc/pki/libvirt/private/serverkey.pem server-key.pem

3. Edit /etc/libvirt/qemu.conf and set the following parameters:

   vnc_listen = "0.0.0.0"
           vnc_tls = 1
           vnc_tls_x509_verify = 1

4. Restart the libvirtd:

   > sudo systemctl restart libvirtd

   

   Important: VM Guests need to be restarted

   The VNC TLS setting is only set when starting a VM Guest. Therefore, you need to restart all machines that have been running prior to making the configuration change.

11.3.2.4.2 VNC over TLS/SSL: client configuration #Edit source

The only action needed on the client side is to place the x509 client certificates in a location recognized by the client of choice. Unfortunately, Virtual Machine Manager and virt-viewer expect the certificates in a different location. Virtual Machine Manager can either read from a system-wide location applying to all users, or from a per-user location. Remmina (refer to Book “Reference”, Chapter 4 “Remote graphical sessions with VNC”, Section 4.2 “Remmina: the remote desktop client”) asks for the location of certificates when initializing the connection to the remote VNC session.

Virtual Machine Manager (virt-manager)

To connect to the remote host, Virtual Machine Manager requires the setup explained in Section 11.3.2.3, “Configuring the client and testing the setup”. To be able to connect via VNC, the client certificates also need to be placed in the following locations:

System-wide location

/etc/pki/CA/cacert.pem

/etc/pki/libvirt-vnc/clientcert.pem

/etc/pki/libvirt-vnc/private/clientkey.pem

Per-user location

/etc/pki/CA/cacert.pem

~/.pki/libvirt-vnc/clientcert.pem

~/.pki/libvirt-vnc/private/clientkey.pem

virt-viewer

virt-viewer only accepts certificates from a system-wide location:

/etc/pki/CA/cacert.pem

/etc/pki/libvirt-vnc/clientcert.pem

/etc/pki/libvirt-vnc/private/clientkey.pem

Important: Restrict access to certificates

Make sure to restrict access to certificates as explained in Section 11.3.2.5, “Restricting access (security considerations)”.

tls_allowed_dn_list = [
   "C=US,L=Provo,O=SUSE Linux Products GmbH,OU=*,CN=venus.example.com,EMAIL=*",
   "C=DE,L=Nuremberg,O=SUSE Linux Products GmbH,OU=Documentation,CN=*"]

> certtool -i --infile /etc/pki/libvirt/clientcert.pem | grep "Subject:"

> sudo systemctl restart libvirtd

> ps ax | grep qemu | grep "\-name sles" | awk -F" -vnc " '{ print FS $2 }'

 -vnc 0.0.0.0:0,tls,x509verify=/etc/pki/libvirt

Locking virtual disks is not enabled by default on openSUSE Leap. To enable and automatically start it upon rebooting, perform the following steps:

By default virtlockd is configured to automatically lock all disks configured for your VM Guests. The default setting uses a "direct" lockspace, where the locks are acquired against the actual file paths associated with the VM Guest <disk> devices. For example, flock(2) will be called directly on /var/lib/libvirt/images/my-server/disk0.raw when the VM Guest contains the following <disk> device:

<disk type='file' device='disk'>
 <driver name='qemu' type='raw'/>
 <source file='/var/lib/libvirt/images/my-server/disk0.raw'/>
 <target dev='vda' bus='virtio'/>
</disk>

The virtlockd configuration can be changed by editing the file /etc/libvirt/qemu-lockd.conf. It also contains detailed comments with further information. Make sure to activate configuration changes by reloading virtlockd:

> sudo systemctl reload virtlockd

file_lockspace_dir = "/MY_LOCKSPACE_DIRECTORY"

lvm_lockspace_dir = "/MY_LOCKSPACE_DIRECTORY"
iscsi_lockspace_dir = "/MY_LOCKSPACE_DIRECTORY"