1. Double-click a VM Guest entry in the Virtual Machine Manager to open its console and switch to the Details view with View › Details.

2. Click Add Hardware and choose the PCI Host Device category in the left pane. A list of available PCI devices appears in the right part of the window.

   

   Figure 14.12: Adding a PCI Device #

3. From the list of available PCI devices, choose the one you want to pass to the guest. Confirm with Finish.

1. Identify the host PCI device to assign to the guest. In the following example we are assigning a DEC network card to the guest:

   tux > sudo lspci -nn
   [...]
   03:07.0 Ethernet controller [0200]: Digital Equipment Corporation DECchip \
   21140 [FasterNet] [1011:0009] (rev 22)
   [...]

   Note down the device ID (03:07.0 in this case).

2. Gather detailed information about the device using virsh nodedev-dumpxml ID. To get the ID, you need to replace colon and period in the device ID (03:07.0) with underscore and prefix the result with “pci_0000_” (pci_0000_03_07_0).

   tux > virsh nodedev-dumpxml pci_0000_03_07_0
   <device>
     <name>pci_0000_03_07_0</name>
     <path>/sys/devices/pci0000:00/0000:00:14.4/0000:03:07.0</path>
     <parent>pci_0000_00_14_4</parent>
     <driver>
       <name>tulip</name>
     </driver>
     <capability type='pci'>
       <domain>0</domain>
       <bus>3</bus>
       <slot>7</slot>
       <function>0</function>
       <product id='0x0009'>DECchip 21140 [FasterNet]</product>
       <vendor id='0x1011'>Digital Equipment Corporation</vendor>
       <numa node='0'/>
     </capability>
   </device>

   Note down the values for domain, bus, and function.

3. Detach the device from the host system prior to attaching it to VM Guest.

   tux > virsh nodedev-detach pci_0000_03_07_0
     Device pci_0000_03_07_0 detached

   

   Tip: Multi-Function PCI Devices

   When using a multi-function PCI device that does not support FLR (function level reset) or PM (power management) reset, you need to detach all its functions from the VM Host Server. The whole device must be reset for security reasons. libvirt will refuse to assign the device if one of its functions is still in use by the VM Host Server or another VM Guest.

4. Convert the domain, bus, slot, and function value from decimal to hexadecimal, and prepend 0x to tell the system that the value is hexadecimal. In our example, domain = 0, bus = 3, slot = 7, and function = 0. Their hexadecimal values are:

   tux > printf %x 0
   0
   tux > printf %x 3
   3
   tux > printf %x 7
   7

   This results in domain = 0x0000, bus = 0x03, slot = 0x07 and function = 0x00.

5. Run virsh edit on your domain, and add the following device entry in the <devices> section using the values from the previous step:

   <hostdev mode='subsystem' type='pci' managed='yes'>
     <source>
       <address domain='0x0000' bus='0x03' slot='0x07' function='0x00'/>
     </source>
   </hostdev>

   

   Tip: managed vs. unmanaged

   libvirt recognizes two modes for handling PCI devices: they can be either managed or unmanaged. In the managed case, libvirt will handle all the details of unbinding the device from the existing driver if needed, resetting the device, binding it to vfio-pci before starting the domain, etc. When the domain is terminated or the device is removed from the domain, libvirt will unbind from vfio-pci and rebind to the original driver in the case of a managed device. If the device is unmanaged, the user must take care to ensure all of these management aspects of the device are done before assigning it to a domain, and after the device is no longer used by the domain.

   In the example above, the managed='yes' option means that the device is managed. To switch the device mode to unmanaged, set managed='no' in the listing above. If you do so, you need to take care of the related driver with the virsh nodedev-detach and virsh nodedev-reattach commands. That means you need to run virsh nodedev-detach pci_0000_03_07_0 prior to starting the VM Guest to detach the device from the host. In case the VM Guest is not running, you can make the device available for the host by running virsh nodedev-reattach pci_0000_03_07_0.

6. Shut down the VM Guest and restart it to make the assigned PCI device available.

   

   Tip: SELinux

   If you are running SELinux on your VM Host Server, you need to disable it prior to starting the VM Guest with

   setsebool -P virt_use_sysfs 1

1. Before loading the driver, make sure the card is properly detected by running lspci. The following example shows the lspci output for the dual-port Intel 82576NS network card:

   tux > sudo /sbin/lspci | grep 82576
   01:00.0 Ethernet controller: Intel Corporation 82576NS Gigabit Network Connection (rev 01)
   01:00.1 Ethernet controller: Intel Corporation 82576NS Gigabit Network Connection (rev 01)
   04:00.0 Ethernet controller: Intel Corporation 82576NS Gigabit Network Connection (rev 01)
   04:00.1 Ethernet controller: Intel Corporation 82576NS Gigabit Network Connection (rev 01)

   In case the card is not detected, it is likely that the hardware virtualization support in the BIOS/EFI has not been enabled.

2. Check whether the SR-IOV driver is already loaded by running lsmod. In the following example a check for the igb driver (for the Intel 82576NS network card) returns a result. That means the driver is already loaded. If the command returns nothing, the driver is not loaded.

   tux > sudo /sbin/lsmod | egrep "^igb "
   igb                   185649  0

3. Skip this step if the driver is already loaded.

   If the SR-IOV driver is not yet loaded, the non-SR-IOV driver needs to be removed first, before loading the new driver. Use rmmod to unload a driver. The following example unloads the non-SR-IOV driver for the Intel 82576NS network card:

   sudo /sbin/rmmod igbvf

   Load the SR-IOV driver subsequently using the modprobe command:

   sudo /sbin/modprobe igb

4. Configure the driver by adding the number of VFs you want to make available and, if necessary, by blacklisting the non-SR-IOV driver:

   sudo echo -e "options igb max_vfs=8\nblacklist igbvf" >> /etc/modprobe.d/50-igb/

   Make sure to replace the example values igb, igbvf and 50-igb by values appropriate for your driver.

5. Now make sure the driver is loaded on boot by creating the file /etc/modules-load.d/igb.conf with the following content:

   # Load the igb driver at boot
   igb

   Make sure to replace the example value igb by a string appropriate for your driver.

6. Reboot the machine and check if the SR-IOV driver is loaded by re-running the lspci command from the first step of this procedure. If the SR-IOV driver was loaded successfully you should see additional lines for the VFs:

   01:00.0 Ethernet controller: Intel Corporation 82576NS Gigabit Network Connection (rev 01)
   01:00.1 Ethernet controller: Intel Corporation 82576NS Gigabit Network Connection (rev 01)
   01:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   01:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   01:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   [...]
   04:00.0 Ethernet controller: Intel Corporation 82576NS Gigabit Network Connection (rev 01)
   04:00.1 Ethernet controller: Intel Corporation 82576NS Gigabit Network Connection (rev 01)
   04:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   04:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   04:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
   [...]

Note: The following procedure is using example data. Make sure to replace it by appropriate data from your setup.

1. Use the virsh nodedev-list command to get the PCI address of the VF you want to assign and its corresponding PF. Numerical values from the lspci output shown in Section 14.11.2, “Loading and Configuring the SR-IOV Host Drivers” (for example 01:00.0 or 04:00.1) are transformed by adding the prefix "pci_0000_" and by replacing colons and dots with underscores. So a PCI ID listed as "04:00.0" by lspci is listed as "pci_0000_04_00_0" by virsh. The following example lists the PCI IDs for the second port of the Intel 82576NS network card:

   tux > sudo virsh nodedev-list | grep 0000_04_
   pci_0000_04_00_0
   pci_0000_04_00_1
   pci_0000_04_10_0
   pci_0000_04_10_1
   pci_0000_04_10_2
   pci_0000_04_10_3
   pci_0000_04_10_4
   pci_0000_04_10_5
   pci_0000_04_10_6
   pci_0000_04_10_7
   pci_0000_04_11_0
   pci_0000_04_11_1
   pci_0000_04_11_2
   pci_0000_04_11_3
   pci_0000_04_11_4
   pci_0000_04_11_5

   The first two entries represent the PFs, whereas the other entries represent the VFs.

2. Get more data that will be needed by running the command virsh nodedev-dumpxml on the PCI ID of the VF you want to add:

   tux > sudo virsh nodedev-dumpxml pci_0000_04_10_0
   <device>
     <name>pci_0000_04_10_0</name>
     <parent>pci_0000_00_02_0</parent>
     <capability type='pci'>
       <domain>0</domain>
       <bus>4</bus>
       <slot>16</slot>
       <function>0</function>
       <product id='0x10ca'>82576 Virtual Function</product>
       <vendor id='0x8086'>Intel Corporation</vendor>
       <capability type='phys_function'>
         <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
       </capability>
     </capability>
   </device>

   The following data is needed for the next step:

   • <domain>0</domain>

   • <bus>4</bus>

   • <slot>16</slot>

   • <function>0</function>

3. Create a temporary XML file (for example /tmp/vf-interface.xml containing the data necessary to add a VF network device to an existing VM Guest. The minimal content of the file needs to look like the following:

   <interface type='hostdev'>1
    <source>
     <address type='pci' domain='0' bus='11' slot='16' function='0'2/>2
    </source>
   </interface>

   1

   VFs do not get a fixed MAC address; it changes every time the host reboots. When adding network devices the “traditional” way with <hostdev>, it would require to reconfigure the VM Guest's network device after each reboot of the host, because of the MAC address change. To avoid this kind of problem, libvirt introduced the “interface type='hostdev'” directive, which sets up network-specific data before assigning the device.

   2

   Specify the data you acquired in the previous step here.

4. In case a device is already attached to the host, it cannot be attached to a guest. To make it available for guests, detach it from the host first:

   virsh nodedev-detach pci_0000_04_10_0

5. Last, add the VF interface to an existing VM Guest:

   virsh attach-device GUEST /tmp/vf-interface.xml --OPTION

   GUEST needs to be replaced by the domain name, id or uuid of the VM Guest and --OPTION can be one of the following:

   --persistent

   This option will always add the device to the domain's persistent XML. In addition, if the domain is running, it will be hotplugged.

   --config

   This option will only affect the persistent XML, even if the domain is running. The device will only show up in the guest on next boot.

   --live

   This option will only affect a running domain. If the domain is inactive, the operation will fail. The device is not persisted in the XML and will not be available in the guest on next boot.

   --current

   This option affects the current state of the domain. If the domain is inactive, the device is added to the persistent XML and will be available on next boot. If the domain is active, the device is hotplugged but not added to the persistent XML.

   To detach a VF interface, use the virsh detach-device command, which also takes the options listed above.