1. Double-click a VM Guest entry in the Virtual Machine Manager to open its console and switch to the Details view with View › Details.

2. Click Add Hardware and choose the PCI Host Device category in the left panel. A list of available PCI devices appears in the right part of the window.

   

   Figure 13.12: Adding a PCI Device #

3. From the list of available PCI devices, choose the one you want to pass to the guest. Confirm with Finish.

1. Identify the host PCI device to assign to the guest. In the following example, we are assigning a DEC network card to the guest:

   tux > sudo lspci -nn
   [...]
   03:07.0 Ethernet controller [0200]: Digital Equipment Corporation DECchip \
   21140 [FasterNet] [1011:0009] (rev 22)
   [...]

   Note down the device ID (03:07.0 in this case).

2. Gather detailed information about the device using virsh nodedev-dumpxml ID. To get the ID, you need to replace colon and period in the device ID (03:07.0) with underscore and prefix the result with “pci_0000_” (pci_0000_03_07_0).

   tux > virsh nodedev-dumpxml pci_0000_03_07_0
   <device>
     <name>pci_0000_03_07_0</name>
     <path>/sys/devices/pci0000:00/0000:00:14.4/0000:03:07.0</path>
     <parent>pci_0000_00_14_4</parent>
     <driver>
       <name>tulip</name>
     </driver>
     <capability type='pci'>
       <domain>0</domain>
       <bus>3</bus>
       <slot>7</slot>
       <function>0</function>
       <product id='0x0009'>DECchip 21140 [FasterNet]</product>
       <vendor id='0x1011'>Digital Equipment Corporation</vendor>
       <numa node='0'/>
     </capability>
   </device>

   Note down the values for domain, bus, and function.

3. Detach the device from the host system prior to attaching it to VM Guest.

   tux > virsh nodedev-detach pci_0000_03_07_0
     Device pci_0000_03_07_0 detached

   

   Tip: Multi-Function PCI Devices

   When using a multi-function PCI device that does not support FLR (function level reset) or PM (power management) reset, you need to detach all its functions from the VM Host Server. The whole device must be reset for security reasons. libvirt will refuse to assign the device if one of its functions is still in use by the VM Host Server or another VM Guest.

4. Convert the domain, bus, slot, and function value from decimal to hexadecimal, and prefix with 0x to tell the system that the value is hexadecimal. In our example, domain = 0, bus = 3, slot = 7, and function = 0. Their hexadecimal values are:

   tux > printf %x 0
   0
   tux > printf %x 3
   3
   tux > printf %x 7
   7

   This results in domain = 0x0000, bus = 0x03, slot = 0x07 and function = 0x00.

5. Run virsh edit on your domain, and add the following device entry in the <devices> section using the values from the previous step:

   <hostdev mode='subsystem' type='pci' managed='yes'>
     <source>
       <address domain='0x0000' bus='0x03' slot='0x07' function='0x00'/>
     </source>
   </hostdev>

   

   Tip: managed Compared to unmanaged

   libvirt recognizes two modes for handling PCI devices: they can be either managed or unmanaged. In the managed case, libvirt will handle all the details of unbinding the device from the existing driver if needed, resetting the device, binding it to vfio-pci before starting the domain, etc. When the domain is terminated or the device is removed from the domain, libvirt will unbind from vfio-pci and rebind to the original driver in the case of a managed device. If the device is unmanaged, the user must take care to ensure all of these management aspects of the device are done before assigning it to a domain, and after the device is no longer used by the domain.

   In the example above, the managed='yes' option means that the device is managed. To switch the device mode to unmanaged, set managed='no' in the listing above. If you do so, you need to take care of the related driver with the virsh nodedev-detach and virsh nodedev-reattach commands. That means you need to run virsh nodedev-detach pci_0000_03_07_0 prior to starting the VM Guest to detach the device from the host. In case the VM Guest is not running, you can make the device available for the host by running virsh nodedev-reattach pci_0000_03_07_0.

6. Shut down the VM Guest and restart it to make the assigned PCI device available.

   

   Tip: SELinux

   If you are running SELinux on your VM Host Server, you need to disable it prior to starting the VM Guest with

   setsebool -P virt_use_sysfs 1

1. Double-click a VM Guest entry in the Virtual Machine Manager to open its console and switch to the Details view with View › Details.

2. Click Add Hardware and choose the USB Host Device category in the left panel. A list of available USB devices appears in the right part of the window.

   

   Figure 13.13: Adding a USB Device #

3. From the list of available USB devices, choose the one you want to pass to the guest. Confirm with Finish. The new USB device appears in the left pane of the Details view.

   

   Tip: USB Device Removal

   To remove the host USB device assignment, click it in the left pane of the Details view and confirm with Remove.

1. Identify the host USB device to assign to the guest:

   tux > sudo lsusb
   [...]
   Bus 001 Device 003: ID 0557:2221 ATEN International Co., Ltd Winbond Hermon
   [...]

   Note down the vendor and product IDs. In our example, the vendor ID is 0557 and the product ID is 2221.

2. Run virsh edit on your domain, and add the following device entry in the <devices> section using the values from the previous step:

   <hostdev mode='subsystem' type='usb'>
     <source startupPolicy='optional'>
      <vendor id='0557'/>
      <product id='2221'/>
     </source>
   </hostdev>

   

   Tip: Vendor/Product or Device's Address

   Instead of defining the host device with <vendor/> and <product/> IDs, you can use the <address/> element as described for host PCI devices in Section 13.10.2, “Adding a PCI Device with virsh”.

3. Shut down the VM Guest and restart it to make the assigned USB device available.

   

   Tip: SELinux

   If you are running SELinux on your VM Host Server, you need to disable it prior to starting the VM Guest with

   setsebool -P virt_use_sysfs 1

1. Before loading the driver, make sure the card is properly detected by running lspci. The following example shows the lspci output for the dual-port Intel 82576NS network card:

   tux > sudo /sbin/lspci | grep 82576
   01:00.0 Ethernet controller: Intel Corporation 82576NS Gigabit Network Connection (rev 01)
   01:00.1 Ethernet controller: Intel Corporation 82576NS Gigabit Network Connection (rev 01)
   04:00.0 Ethernet controller: Intel Corporation 82576NS Gigabit Network Connection (rev 01)
   04:00.1 Ethernet controller: Intel Corporation 82576NS Gigabit Network Connection (rev 01)

   In case the card is not detected, it is likely that the hardware virtualization support in the BIOS/EFI has not been enabled.

2. Check whether the SR-IOV driver is already loaded by running lsmod. In the following example a check for the igb driver (for the Intel 82576NS network card) returns a result. That means the driver is already loaded. If the command returns nothing, the driver is not loaded.

   tux > sudo /sbin/lsmod | egrep "^igb "
   igb                   185649  0

3. Skip this step if the driver is already loaded.

   If the SR-IOV driver is not yet loaded, the non-SR-IOV driver needs to be removed first, before loading the new driver. Use rmmod to unload a driver. The following example unloads the non-SR-IOV driver for the Intel 82576NS network card:

   sudo /sbin/rmmod igbvf

   Load the SR-IOV driver subsequently using the modprobe command—the VF parameter (max_vfs) is mandatory:

   sudo /sbin/modprobe igb max_vfs=8

   Or load the driver via SYSFS:

   Find the PCI ID of the physical NIC by listing Ethernet devices:

   tux > sudo lspci | grep Eth
   06:00.0 Ethernet controller: Emulex Corporation OneConnect NIC (Skyhawk) (rev 10)
   06:00.1 Ethernet controller: Emulex Corporation OneConnect NIC (Skyhawk) (rev 10)

   To enable VFs, echo the number of desired VFs to load to the sriov_numvfs parameter:

   tux > sudo echo 1 > /sys/bus/pci/devices/0000:06:00.1/sriov_numvfs

   Verify that the VF NIC was loaded:

   tux > sudo lspci | grep Eth
   06:00.0 Ethernet controller: Emulex Corporation OneConnect NIC (Skyhawk) (rev 10)
   06:00.1 Ethernet controller: Emulex Corporation OneConnect NIC (Skyhawk) (rev 10)
   06:08.0 Ethernet controller: Emulex Corporation OneConnect NIC (Skyhawk) (rev 10)

   Obtain the maximum number of VFs available:

   tux > sudo lspci -vvv -s 06:00.1 | grep 'Initial VFs'
                          Initial VFs: 32, Total VFs: 32, Number of VFs: 0,
   Function Dependency Link: 01

4. Create a before.service file which loads VF via SYSFS on boot:

   [Unit]
   Before=
   [Service]
   Type=oneshot
   RemainAfterExit=true
   ExecStart=/bin/bash -c "echo 1 > /sys/bus/pci/devices/0000:06:00.1/sriov_numvfs"
   # beware, executable is run directly, not through a shell, check the man pages
   # systemd.service and systemd.unit for full syntax
   [Install]
   # target in which to start the service
   WantedBy=multi-user.target
   #WantedBy=graphical.target

   And copy it to /etc/systemd/system.

   Additionally, it is required to create another service file (after-local.service) pointing to /etc/init.d/after.local script that detaches the NIC prior to starting the VM, otherwise the VM would fail to start:

   [Unit]
   Description=/etc/init.d/after.local Compatibility
   After=libvirtd.service
   Requires=libvirtd.service
   [Service]
   Type=oneshot
   ExecStart=/etc/init.d/after.local
   RemainAfterExit=true
   
   [Install]
   WantedBy=multi-user.target

   And copy it to /etc/systemd/system.

   #! /bin/sh
   #
   # Copyright (c) 2010 SuSE LINUX Products GmbH, Germany.  All rights reserved.
   # ...
   virsh nodedev-detach pci_0000_06_08_0

   Then save it as /etc/init.d/after.local.

5. Reboot the machine and check if the SR-IOV driver is loaded by re-running the lspci command from the first step of this procedure. If the SR-IOV driver was loaded successfully you should see additional lines for the VFs:

   01:00.0 Ethernet controller: Intel Corporation 82576NS Gigabit Network Connection (rev 01)
   01:00.1 Ethernet controller: Intel Corporation 82576NS Gigabit Network Connection (rev 01)
   01:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   01:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   01:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   [...]
   04:00.0 Ethernet controller: Intel Corporation 82576NS Gigabit Network Connection (rev 01)
   04:00.1 Ethernet controller: Intel Corporation 82576NS Gigabit Network Connection (rev 01)
   04:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   04:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   04:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   [...]

Note: The following procedure is using example data. Make sure to replace it by appropriate data from your setup.

1. Use the virsh nodedev-list command to get the PCI address of the VF you want to assign and its corresponding PF. Numerical values from the lspci output shown in Section 13.12.2, “Loading and Configuring the SR-IOV Host Drivers” (for example 01:00.0 or 04:00.1) are transformed by adding the prefix "pci_0000_" and by replacing colons and dots with underscores. So a PCI ID listed as "04:00.0" by lspci is listed as "pci_0000_04_00_0" by virsh. The following example lists the PCI IDs for the second port of the Intel 82576NS network card:

   tux > sudo virsh nodedev-list | grep 0000_04_
   pci_0000_04_00_0
   pci_0000_04_00_1
   pci_0000_04_10_0
   pci_0000_04_10_1
   pci_0000_04_10_2
   pci_0000_04_10_3
   pci_0000_04_10_4
   pci_0000_04_10_5
   pci_0000_04_10_6
   pci_0000_04_10_7
   pci_0000_04_11_0
   pci_0000_04_11_1
   pci_0000_04_11_2
   pci_0000_04_11_3
   pci_0000_04_11_4
   pci_0000_04_11_5

   The first two entries represent the PFs, whereas the other entries represent the VFs.

2. Get more data that will be needed by running the command virsh nodedev-dumpxml on the PCI ID of the VF you want to add:

   tux > sudo virsh nodedev-dumpxml pci_0000_04_10_0
   <device>
     <name>pci_0000_04_10_0</name>
     <parent>pci_0000_00_02_0</parent>
     <capability type='pci'>
       <domain>0</domain>
       <bus>4</bus>
       <slot>16</slot>
       <function>0</function>
       <product id='0x10ca'>82576 Virtual Function</product>
       <vendor id='0x8086'>Intel Corporation</vendor>
       <capability type='phys_function'>
         <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
       </capability>
     </capability>
   </device>

   The following data is needed for the next step:

   • <domain>0</domain>

   • <bus>4</bus>

   • <slot>16</slot>

   • <function>0</function>

3. Create a temporary XML file (for example /tmp/vf-interface.xml containing the data necessary to add a VF network device to an existing VM Guest. The minimal content of the file needs to look like the following:

   <interface type='hostdev'>1
    <source>
     <address type='pci' domain='0' bus='11' slot='16' function='0'2/>2
    </source>
   </interface>

   1

   VFs do not get a fixed MAC address; it changes every time the host reboots. When adding network devices the “traditional” way with <hostdev>, it would require to reconfigure the VM Guest's network device after each reboot of the host, because of the MAC address change. To avoid this kind of problem, libvirt introduced the “interface type='hostdev'” directive, which sets up network-specific data before assigning the device.

   2

   Specify the data you acquired in the previous step here.

4. In case a device is already attached to the host, it cannot be attached to a guest. To make it available for guests, detach it from the host first:

   virsh nodedev-detach pci_0000_04_10_0

5. Last, add the VF interface to an existing VM Guest:

   virsh attach-device GUEST /tmp/vf-interface.xml --OPTION

   GUEST needs to be replaced by the domain name, id or uuid of the VM Guest and --OPTION can be one of the following:

   --persistent

   This option will always add the device to the domain's persistent XML. In addition, if the domain is running, it will be hotplugged.

   --config

   This option will only affect the persistent XML, even if the domain is running. The device will only show up in the guest on next boot.

   --live

   This option will only affect a running domain. If the domain is inactive, the operation will fail. The device is not persisted in the XML and will not be available in the guest on next boot.

   --current

   This option affects the current state of the domain. If the domain is inactive, the device is added to the persistent XML and will be available on next boot. If the domain is active, the device is hotplugged but not added to the persistent XML.

   To detach a VF interface, use the virsh detach-device command, which also takes the options listed above.