Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
Virtualization Guide › Configuring GPU Pass-Through for NVIDIA cards
ContentsContents
Virtualization Guide
  1. Preface
  2. I Introduction
    1. 1 Virtualization technology
    2. 2 Virtualization scenarios
    3. 3 Introduction to Xen virtualization
    4. 4 Introduction to KVM virtualization
    5. 5 Virtualization tools
    6. 6 Installation of virtualization components
  3. II Managing virtual machines with libvirt
    1. 7 Starting and stopping libvirtd
    2. 8 Preparing the VM Host Server
    3. 9 Guest installation
    4. 10 Basic VM Guest management
    5. 11 Connecting and authorizing
    6. 12 Advanced storage topics
    7. 13 Configuring virtual machines with Virtual Machine Manager
    8. 14 Configuring virtual machines with virsh
    9. 15 Managing virtual machines with Vagrant
    10. 16 Xen to KVM migration guide
  4. III Hypervisor-independent features
    1. 17 Disk cache modes
    2. 18 VM Guest clock settings
    3. 19 libguestfs
    4. 20 QEMU guest agent
    5. 21 Software TPM emulator
  5. IV Managing virtual machines with Xen
    1. 22 Setting up a virtual machine host
    2. 23 Virtual networking
    3. 24 Managing a virtualization environment
    4. 25 Block devices in Xen
    5. 26 Virtualization: configuration options and settings
    6. 27 Administrative tasks
    7. 28 XenStore: configuration database shared between domains
    8. 29 Xen as a high-availability virtualization host
    9. 30 Xen: converting a paravirtual (PV) guest into a fully virtual (FV/HVM) guest
  6. V Managing virtual machines with QEMU
    1. 31 QEMU overview
    2. 32 Setting up a KVM VM Host Server
    3. 33 Guest installation
    4. 34 Running virtual machines with qemu-system-ARCH
    5. 35 Virtual machine administration using QEMU monitor
  7. VI Troubleshooting
    1. 36 Integrated help and package documentation
    2. 37 Gathering system information and logs
  8. Glossary
  9. A Configuring GPU Pass-Through for NVIDIA cards
  10. B GNU licenses
Navigation←→
Virtualization Guide › Configuring GPU Pass-Through for NVIDIA cards
Top
←→
Applies to openSUSE Leap 15.3

A Configuring GPU Pass-Through for NVIDIA cards Edit source

A.1 Introduction
A.2 Prerequisites
A.3 Configuring the host
A.4 Configuring the guest

A.1 Introduction Edit source

This article describes how to assign an NVIDIA GPU graphics card on the host machine to a virtualized guest.

A.2 Prerequisites Edit source

  • GPU pass-through is supported on the AMD64/Intel 64 architecture only.

  • The host operating system needs to be SLES 12 SP3 or newer.

  • This article deals with a set of instructions based on V100/T1000 NVIDIA cards, and is meant for GPU computation purposes only.

  • Verify that you are using an NVIDIA Tesla product—Maxwell, Pascal, or Volta.

  • To be able to manage the host system, you need an additional display card on the host that you can use when configuring the GPU Pass-Through, or a functional SSH environment.

A.3 Configuring the host Edit source

A.3.1 Verify the host environment Edit source

  1. Verify that the host operating system is SLES 12 SP3 or newer: 

    > cat /etc/issue
Welcome to SUSE Linux Enterprise Server 15  (x86_64) - Kernel \r (\l).

  2. Verify that the host supports VT-d technology and that it is already enabled in the firmware settings: 

    > dmesg | grep -e "Directed I/O"
[   12.819760] DMAR: Intel(R) Virtualization Technology for Directed I/O

    If VT-d is not enabled in the firmware, enable it and reboot the host.

  3. Verify that the host has an extra GPU or VGA card: 

    > lspci | grep -i "vga"
07:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. \
  MGA G200e [Pilot] ServerEngines (SEP1) (rev 05)

    With a Tesla V100 card: 

    > lspci | grep -i nvidia
03:00.0 3D controller: NVIDIA Corporation GV100 [Tesla V100 PCIe] (rev a1)

    With a T1000 Mobile (available on Dell 5540): 

    > lspci | grep -i nvidia
01:00.0 3D controller: NVIDIA Corporation TU117GLM [Quadro T1000 Mobile] (rev a1)

A.3.2 Enable IOMMU Edit source

IOMMU is disabled by default. You need to enable it at boot time in the /etc/default/grub configuration file.

  1. For Intel-based hosts: 

    GRUB_CMDLINE_LINUX="intel_iommu=on iommu=pt rd.driver.pre=vfio-pci"

    For AMD-based hosts: 

    GRUB_CMDLINE_LINUX="iommu=pt amd_iommu=on rd.driver.pre=vfio-pci"

  2. When you save the modified /etc/default/grub file, re-generate the main GRUB 2 configuration file /boot/grub2/grub.cfg: 

    > sudo grub2-mkconfig -o /boot/grub2/grub.cfg

  3. Reboot the host and verify that IOMMU is enabled: 

    > dmesg |  grep -e DMAR -e IOMMU

A.3.3 Blacklist the Nouveau driver Edit source

Because we want to assign the NVIDIA card to a VM guest, we need to avoid use of the card by the host OS's built-in driver for NVIDIA GPUs. The open source NVIDIA driver is called nouveau. Edit the file /etc/modprobe.d/50-blacklist.conf and append the following line to its end: 

blacklist nouveau

A.3.4 Configure VFIO and isolate the GPU used for pass-through Edit source

  1. Find the card vendor and model IDs. Utilize the bus number identified in Section A.3.1, “Verify the host environment”, for example 03:00.0: 

    > lspci -nn | grep 03:00.0
03:00.0 3D controller [0302]: NVIDIA Corporation GV100 [Tesla V100 PCIe] [10de:1db4] (rev a1)

  2. Create the file vfio.conf in the /etc/modprobe.d/ directory with the following content: 

    options vfio-pci ids=10de:1db4
    Note
    Note

    Verify that your card does not need an extra ids= parameter. For some cards, you must specify the audio device too, so that device's ID must also be added to the list, otherwise you will not be able to use the card.

A.3.5 Load the VFIO driver Edit source

There are three ways you can load the VFIO driver.

A.3.5.1 Including the driver in the initrd file Edit source

  1. Create the file /etc/dracut.conf.d/gpu-passthrough.conf and add the following content (mind the leading whitespace): 

    add_drivers+=" vfio vfio_iommu_type1 vfio_pci vfio_virqfd"

  2. Re-generate the initrd file: 

    > sudo dracut --force /boot/initrd $(uname -r)

A.3.5.2 Adding the driver to the list of auto-loaded modules Edit source

Create the file /etc/modules-load.d/vfio-pci.conf and add the following content: 

pci_stub
vfio
vfio_iommu_type1
vfio_pci
kvm
kvm_intel

A.3.5.3 Loading the driver manually Edit source

To load the driver manually at run-time, execute the following command: 

> sudo modprobe vfio-pci

A.3.6 Disable MSR for Microsoft Windows guests Edit source

For Microsoft Windows guests, we recommend disabling MSR (model-specific register) to avoid the guest crashing. Create the file /etc/modprobe.d/kvm.conf and add the following content: 

options kvm ignore_msrs=1

A.3.7 Install and enable UEFI firmware Edit source

For proper GPU Pass-Through functionality, the host needs to boot using UEFI firmware (that is, not using a legacy-style BIOS boot sequence).

  1. Install the qemu-ovmf package which includes UEFI firmware images: 

    > sudo zypper install qemu-ovmf

  2. Get the list of OVMF bin and vars files by filtering the results of the following command: 

    > rpm -ql qemu-ovmf

  3. Enable OVMF in the libvirt QEMU configuration in the file /etc/libvirt/qemu.conf by using the list obtained from the previous step. It should look similar to the following: 

    nvram = [
"/usr/share/qemu/ovmf-x86_64-4m.bin:/usr/share/qemu/ovmf-x86_64-4m-vars.bin",
"/usr/share/qemu/ovmf-x86_64-4m-code.bin:/usr/share/qemu/ovmf-x86_64-4m-vars.bin",
"/usr/share/qemu/ovmf-x86_64-smm-ms-code.bin:/usr/share/qemu/ovmf-x86_64-smm-ms-vars.bin",
"/usr/share/qemu/ovmf-x86_64-smm-opensuse-code.bin:/usr/share/qemu/ovmf-x86_64-smm-opensuse-vars.bin",
"/usr/share/qemu/ovmf-x86_64-ms-4m-code.bin:/usr/share/qemu/ovmf-x86_64-ms-4m-vars.bin",
"/usr/share/qemu/ovmf-x86_64-smm-suse-code.bin:/usr/share/qemu/ovmf-x86_64-smm-suse-vars.bin",
"/usr/share/qemu/ovmf-x86_64-ms-code.bin:/usr/share/qemu/ovmf-x86_64-ms-vars.bin",
"/usr/share/qemu/ovmf-x86_64-smm-code.bin:/usr/share/qemu/ovmf-x86_64-smm-vars.bin",
"/usr/share/qemu/ovmf-x86_64-opensuse-4m-code.bin:/usr/share/qemu/ovmf-x86_64-opensuse-4m-vars.bin",
"/usr/share/qemu/ovmf-x86_64-suse-4m-code.bin:/usr/share/qemu/ovmf-x86_64-suse-4m-vars.bin",
"/usr/share/qemu/ovmf-x86_64-suse-code.bin:/usr/share/qemu/ovmf-x86_64-suse-vars.bin",
"/usr/share/qemu/ovmf-x86_64-opensuse-code.bin:/usr/share/qemu/ovmf-x86_64-opensuse-vars.bin",
"/usr/share/qemu/ovmf-x86_64-code.bin:/usr/share/qemu/ovmf-x86_64-vars.bin",
]

A.3.8 Reboot the host machine Edit source

For most of the changes in the above steps to take effect, you need to reboot the host machine: 

> sudo shutdown -r now

A.4 Configuring the guest Edit source

This section describes how to configure the guest virtual machine so that it can use the host's NVIDIA GPU. Use Virtual Machine Manager or virt-install to install the guest VM. Find more details in Chapter 9, Guest installation.

A.4.1 Requirements for the guest configuration Edit source

During the guest VM installation, select Customize configuration before install and configure the following devices:

  • Use Q35 chipset if possible.

  • Install the guest VM using UEFI firmware.

  • Add the following emulated devices:

    Graphic: Spice or VNC

    Device: qxl, VGA, or Virtio

    Find more information in Section 13.6, “Video”.

  • Add the host PCI device (03:00.0 in our example) to the guest. Find more information in Section 13.12, “Assigning a host PCI device to a VM Guest”.

  • For best performance, we recommend using virtio drivers for the network card and storage.

A.4.2 Install the graphic card driver Edit source

A.4.2.1 Linux guest Edit source

Procedure A.1: RPM-based distributions

  1. Download the driver RPM package from http://www.nvidia.com/download/driverResults.aspx/131159/en-us.

  2. Install the downloaded RPM package: 

    > sudo rpm -i nvidia-diag-driver-local-repo-sles123-390.30-1.0-1.x86_64.rpm

  3. Refresh repositories and install cuda-drivers. This step will be different for non-SUSE distributions: 

    > sudo zypper refresh && zypper install cuda-drivers

  4. Reboot the guest VM: 

    > sudo shutdown -r now
Procedure A.2: Generic installer

  1. Because the installer needs to compile the NVIDIA driver modules, install the gcc-c++ and kernel-devel packages.

  2. Disable Secure Boot on the guest, because NVIDIA's driver modules are unsigned. On SUSE distributions, you can use the YaST GRUB 2 module to disable Secure Boot. Find more information in Book “Reference”, Chapter 14 “UEFI (Unified Extensible Firmware Interface)”, Section 14.1.1 “Implementation on openSUSE Leap”.

  3. Download the driver installation script from https://www.nvidia.com/Download/index.aspx?lang=en-us, make it executable, and run it to complete the driver installation: 

    > chmod +x NVIDIA-Linux-x86_64-460.73.01.run
> sudo ./NVIDIA-Linux-x86_64-460.73.01.run

  4. Download CUDA drivers from https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=SLES&target_version=15&target_type=rpmlocal and install following the on-screen instructions.

Note
Note: Display issues

After you have installed the NVIDIA drivers, the Virtual Machine Manager display will lose its connection to the guest OS. To access the guest VM, you must either login via ssh, change to the console interface, or install a dedicated VNC server in the guest. To avoid a flickering screen, stop and disable the display manager: 

> sudo systemctl stop display-manager && systemctl disable display-manager
Procedure A.3: Testing the Linux driver installation

  1. Change the directory to the CUDA sample templates: 

    > cd /usr/local/cuda-9.1/samples/0_Simple/simpleTemplates

  2. Compile and run the simpleTemplates file: 

    > make && ./simpleTemplates
runTest<float,32>
GPU Device 0: "Tesla V100-PCIE-16GB" with compute capability 7.0
CUDA device [Tesla V100-PCIE-16GB] has 80 Multi-Processors
Processing time: 495.006000 (ms)
Compare OK
runTest<int,64>
GPU Device 0: "Tesla V100-PCIE-16GB" with compute capability 7.0
CUDA device [Tesla V100-PCIE-16GB] has 80 Multi-Processors
Processing time: 0.203000 (ms)
Compare OK
[simpleTemplates] -> Test Results: 0 Failures

A.4.2.2 Microsoft Windows guest Edit source

Important
Important

Before you install the NVIDIA drivers, you need to hide the hypervisor from the drivers by using the <hidden state='on'/> directive in the guest's libvirt definition.

  1. Download and install the NVIDIA driver from https://www.nvidia.com/Download/index.aspx.

  2. Download and install the CUDA toolkit from https://developer.nvidia.com/cuda-downloads?target_os=Windows&target_arch=x86_64.

  3. Find some NVIDIA demo samples in the directory Program Files\Nvidia GPU Computing Toolkit\CUDA\v10.2\extras\demo_suite on the guest.

→Appendix B GNU licenses←Glossary
Print this page

© 2022 SUSE