5 PCI(e) passthrough is a mechanism to give a virtual machine control over
6 a pci device usually only available for the host. This can have some
7 advantages over using virtualized hardware, for example lower latency,
8 higher performance, or more features (e.g., offloading).
10 If you pass through a device to a virtual machine, you cannot use that
11 device anymore on the host or in any other VM.
16 Since passthrough is a feature which also needs hardware support, there are
17 some requirements and steps before it can work.
22 Your hardware has to support IOMMU interrupt remapping, this includes CPU and
25 Generally Intel systems with VT-d, and AMD systems with AMD-Vi support this,
26 but it is not guaranteed that everything will work out of the box, due
27 to bad hardware implementation or missing/low quality drivers.
29 In most cases, server grade hardware has better support than consumer grade
30 hardware, but even then, many modern system can support this.
32 Please refer to your hardware vendor if this is a feature that is supported
38 To enable PCI(e) passthrough, there are some configurations needed.
40 First, the iommu has to be activated on the kernel commandline.
41 The easiest way is to enable it in */etc/default/grub*. Just add
45 or if you have AMD hardware:
49 to GRUB_CMDLINE_LINUX_DEFAULT
51 After that, make sure you run 'update grub' to update grub.
53 Second, you have to make sure the following modules are loaded.
54 This can be achieved by adding them to */etc/modules*
61 After changing anything modules related, you need to refresh your
65 update-initramfs -u -k all
68 Finally reboot and check that it is indeed enabled.
71 dmesg -e DMAR -e IOMMU -e AMD-Vi
74 should display that IOMMU, Directed I/O or Interrupt Remapping is enabled.
75 (The exact message can vary, depending on hardware and kernel version)
77 It is also important that the device(s) you want to pass through
78 are in a seperate IOMMU group. This can be checked with:
81 find /sys/kernel/iommu_groups/ -type l
84 It is okay if the device is in an IOMMU group together with its functions
85 (e.g. a GPU with the HDMI Audio device) or with its root port or PCI(e) bridge.
90 Some platforms handle their PCI(e) slots differently, so if you
91 do not get the desired IOMMU group separation, it may be helpful to
92 try to put the card in a another PCI(e) slot.
98 For some platforms, it may be necessary to allow unsafe interrupts.
99 This can most easily enabled with adding the following line
100 in a .conf file in */etc/modprobe.d/*.
102 options vfio_iommu_type1 allow_unsafe_interrupts=1
104 Please be aware that this option can make your system unstable.
107 Host Device Passhtrough
108 ~~~~~~~~~~~~~~~~~~~~~~~
110 The most used variant of PCI(e) passthrough is to pass through a whole
111 PCI(e) card, for example a GPU or network card.
116 In this case, the host can not use the card. This can be achieved by two
119 Either add the ids to the options of the vfio-pci modules. This works
122 options vfio-pci ids=1234:5678,4321:8765
124 to a .conf file in */etc/modprobe.d/* where 1234:5678 and 4321:8765 are
125 the vendor and device ids obtained by:
131 Or simply blacklist the driver completely on the host with
135 also in a .conf file in */etc/modprobe.d/*. Again update the initramfs
136 and reboot after that.
141 To pass through the device you set *hostpciX* on the VM with
144 qm set VMID -hostpci0 00:02.0
147 If your device has multiple functions, you can pass them through all together
148 with the shortened syntax
152 There are some options to which may be necessary, depending on the device
155 * *x-vga=on|off* marks the PCI(e) device the primary GPU of the VM.
156 With this enabled the *vga* parameter of the config will be ignored.
157 * *pcie=on|off* tells {pve} to use a PCIe or PCI port. Some guests/device
158 combination require PCIe rather than PCI (only available for q35 machine types).
159 * *rombar=on|off* makes the firmware ROM visible for the guest. Default is on.
160 Some PCI(e) devices need this disabled.
161 * *romfile=<path>*, is an optional path to a ROM file for the device to use.
162 this is a relative path under */usr/share/kvm/*.
164 An example of PCIe passthrough with a GPU set to primary:
167 qm set VMID -hostpci0 02:00,pcie=on,x-vga=on
173 When passing through a GPU, the best compatibility is reached when using
174 q35 as machine type, OVMF instead of SeaBIOS and PCIe instead of PCI.
175 Note that if you want to use OVMF for GPU passthrough, the GPU needs
176 to have an EFI capable ROM, otherwise use SeaBIOS instead.
181 Another variant of passing through PCI(e) devices, is to use the hardware
182 virtualization features of your devices.
184 SR-IOV (Single-root input/output virtualization) enables a single device
185 to provide multiple vf (virtual functions) to the system, so that each
186 vf can be used in a different VM, with full hardware features, better
187 performance and lower latency than software virtualized devices.
189 The most used devices for this are NICs with SR-IOV which can provide
190 multiple vf per physical port, allowing features such as
191 checksum offloading, etc. to be used inside a VM, reducing CPU overhead.
196 Generally there are 2 methods for enabling virtual functions on a device.
198 In some cases there is an option for the driver module e.g. for some
203 which could be put in a file in a .conf file in */etc/modprobe.d/*.
204 (Do not forget to update your initramfs after that)
206 Please refer to your driver module documentation for the exact
207 parameters and options.
209 The second (more generic) approach is via the sysfs.
210 If a device and driver supports this you can change the number of vfs on
211 the fly. For example 4 vfs on device 0000:01:00.0 with:
214 echo 4 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs
217 To make this change persistent you can use sysfsutils.
218 Just install them via
221 apt install sysfsutils
224 and configure it via */etc/sysfs.conf* or */etc/sysfs.d/*.
229 After creating vfs, you should see them as seperate PCI(e) devices, which
230 can be passed through like a normal PCI(e) device.
235 For this feature, platform support is especially important. It may be necessary
236 to enable this feature in the BIOS or to use a specific PCI(e) port for it
237 to work. In doubt, consult the manual of the platform or contact the vendor.