qm-pci-passthrough.adoc

   1 [[qm_pci_passthrough]]
   2 PCI(e) Passthrough
   3 ------------------
   4
   5 PCI(e) passthrough is a mechanism to give a virtual machine control over
   6 a pci device usually only available for the host. This can have some
   7 advantages over using virtualized hardware, for example lower latency,
   8 higher performance, or more features (e.g., offloading).
   9
  10 If you pass through a device to a virtual machine, you cannot use that
  11 device anymore on the host or in any other VM.
  12
  13 General Requirements
  14 ~~~~~~~~~~~~~~~~~~~~
  15
  16 Since passthrough is a feature which also needs hardware support, there are
  17 some requirements and steps before it can work.
  18
  19 Hardware
  20 ^^^^^^^^
  21
  22 Your hardware has to support IOMMU interrupt remapping, this includes CPU and
  23 Mainboard.
  24
  25 Generally Intel systems with VT-d, and AMD systems with AMD-Vi support this,
  26 but it is not guaranteed that everything will work out of the box, due
  27 to bad hardware implementation or missing/low quality drivers.
  28
  29 In most cases, server grade hardware has better support than consumer grade
  30 hardware, but even then, many modern system can support this.
  31
  32 Please refer to your hardware vendor if this is a feature that is supported
  33 under Linux.
  34
  35 Configuration
  36 ^^^^^^^^^^^^^
  37
  38 To enable PCI(e) passthrough, there are some configurations needed.
  39
  40 First, the iommu has to be activated on the kernel commandline.
  41 The easiest way is to enable it in */etc/default/grub*. Just add
  42
  43  intel_iommu=on
  44
  45 or if you have AMD hardware:
  46
  47  amd_iommu=on
  48
  49 to GRUB_CMDLINE_LINUX_DEFAULT
  50
  51 After that, make sure you run 'update grub' to update grub.
  52
  53 Second, you have to make sure the following modules are loaded.
  54 This can be achieved by adding them to */etc/modules*
  55
  56  vfio
  57  vfio_iommu_type1
  58  vfio_pci
  59  vfio_virqfd
  60
  61 After changing anything modules related, you need to refresh your
  62 initramfs with
  63
  64 ----
  65 update-initramfs -u -k all
  66 ----
  67
  68 Finally reboot and check that it is indeed enabled.
  69
  70 ----
  71 dmesg -e DMAR -e IOMMU -e AMD-Vi
  72 ----
  73
  74 should display that IOMMU, Directed I/O or Interrupt Remapping is enabled.
  75 (The exact message can vary, depending on hardware and kernel version)
  76
  77 It is also important that the device(s) you want to pass through
  78 are in a seperate IOMMU group.  This can be checked with:
  79
  80 ----
  81 find /sys/kernel/iommu_groups/ -type l
  82 ----
  83
  84 It is okay if the device is in an IOMMU group together with its functions
  85 (e.g. a GPU with the HDMI Audio device) or with its root port or PCI(e) bridge.
  86
  87 .PCI(e) slots
  88 [NOTE]
  89 ====
  90 Some platforms handle their PCI(e) slots differently, so if you
  91 do not get the desired IOMMU group separation, it may be helpful to
  92 try to put the card in a another PCI(e) slot.
  93 ====
  94
  95 .Unsafe interrupts
  96 [NOTE]
  97 ====
  98 For some platforms, it may be necessary to allow unsafe interrupts.
  99 This can most easily enabled with adding the following line
 100 in a .conf file in */etc/modprobe.d/*.
 101
 102  options vfio_iommu_type1 allow_unsafe_interrupts=1
 103
 104 Please be aware that this option can make your system unstable.
 105 ====
 106
 107 Host Device Passhtrough
 108 ~~~~~~~~~~~~~~~~~~~~~~~
 109
 110 The most used variant of PCI(e) passthrough is to pass through a whole
 111 PCI(e) card, for example a GPU or network card.
 112
 113 Host Configuration
 114 ^^^^^^^^^^^^^^^^^^
 115
 116 In this case, the host can not use the card. This can be achieved by two
 117 methods:
 118
 119 Either add the ids to the options of the vfio-pci modules. This works
 120 with adding
 121
 122  options vfio-pci ids=1234:5678,4321:8765
 123
 124 to a .conf file in */etc/modprobe.d/* where 1234:5678 and 4321:8765 are
 125 the vendor and device ids obtained by:
 126
 127 ----
 128 lcpci -nn
 129 ----
 130
 131 Or simply blacklist the driver completely on the host with
 132
 133  blacklist DRIVERNAME
 134
 135 also in a .conf file in */etc/modprobe.d/*.  Again update the initramfs
 136 and reboot after that.
 137
 138 VM Configuration
 139 ^^^^^^^^^^^^^^^^
 140
 141 To pass through the device you set *hostpciX* on the VM with
 142
 143 ----
 144 qm set VMID -hostpci0 00:02.0
 145 ----
 146
 147 If your device has multiple functions, you can pass them through all together
 148 with the shortened syntax
 149
 150  00:02
 151
 152 There are some options to which may be necessary, depending on the device
 153 and guest OS.
 154
 155 * *x-vga=on|off* marks the PCI(e) device the primary GPU of the VM.
 156 With this enabled the *vga* parameter of the config will be ignored.
 157 * *pcie=on|off* tells {pve} to use a PCIe or PCI port. Some guests/device
 158 combination require PCIe rather than PCI (only available for q35 machine types).
 159 * *rombar=on|off* makes the firmware ROM visible for the guest. Default is on.
 160 Some PCI(e) devices need this disabled.
 161 * *romfile=<path>*, is an optional path to a ROM file for the device to use.
 162 this is a relative path under */usr/share/kvm/*.
 163
 164 An example of PCIe passthrough with a GPU set to primary:
 165
 166 ----
 167 qm set VMID -hostpci0 02:00,pcie=on,x-vga=on
 168 ----
 169
 170 Other considerations
 171 ^^^^^^^^^^^^^^^^^^^^
 172
 173 When passing through a GPU, the best compatibility is reached when using
 174 q35 as machine type, OVMF instead of SeaBIOS and PCIe instead of PCI.
 175 Note that if you want to use OVMF for GPU passthrough, the GPU needs
 176 to have an EFI capable ROM, otherwise use SeaBIOS instead.
 177
 178 SR-IOV
 179 ~~~~~~
 180
 181 Another variant of passing through PCI(e) devices, is to use the hardware
 182 virtualization features of your devices.
 183
 184 SR-IOV (Single-root input/output virtualization) enables a single device
 185 to provide multiple vf (virtual functions) to the system, so that each
 186 vf can be used in a different VM, with full hardware features, better
 187 performance and lower latency than software virtualized devices.
 188
 189 The most used devices for this are NICs with SR-IOV which can provide
 190 multiple vf per physical port, allowing features such as
 191 checksum offloading, etc. to be used inside a VM, reducing CPU overhead.
 192
 193 Host Configuration
 194 ^^^^^^^^^^^^^^^^^^
 195
 196 Generally there are 2 methods for enabling virtual functions on a device.
 197
 198 In some cases there is an option for the driver module e.g. for some
 199 Intel drivers
 200
 201  max_vfs=4
 202
 203 which could be put in a file in a .conf file in */etc/modprobe.d/*.
 204 (Do not forget to update your initramfs after that)
 205
 206 Please refer to your driver module documentation for the exact
 207 parameters and options.
 208
 209 The second (more generic) approach is via the sysfs.
 210 If a device and driver supports this you can change the number of vfs on
 211 the fly. For example 4 vfs on device 0000:01:00.0 with:
 212
 213 ----
 214 echo 4 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs
 215 ----
 216
 217 To make this change persistent you can use sysfsutils.
 218 Just install them via
 219
 220 ----
 221 apt install sysfsutils
 222 ----
 223
 224 and configure it via */etc/sysfs.conf* or */etc/sysfs.d/*.
 225
 226 VM Configuration
 227 ^^^^^^^^^^^^^^^^
 228
 229 After creating vfs, you should see them as seperate PCI(e) devices, which
 230 can be passed through like a normal PCI(e) device.
 231
 232 Other considerations
 233 ^^^^^^^^^^^^^^^^^^^^
 234
 235 For this feature, platform support is especially important. It may be necessary
 236 to enable this feature in the BIOS or to use a specific PCI(e) port for it
 237 to work. In doubt, consult the manual of the platform or contact the vendor.