5 PCI(e) passthrough is a mechanism to give a virtual machine control over
6 a PCI device from the host. This can have some advantages over using
7 virtualized hardware, for example lower latency, higher performance, or more
8 features (e.g., offloading).
10 But, if you pass through a device to a virtual machine, you cannot use that
11 device anymore on the host or in any other VM.
16 Since passthrough is a feature which also needs hardware support, there are
17 some requirements to check and preparations to be done to make it work.
22 Your hardware needs to support `IOMMU` (*I*/*O* **M**emory **M**anagement
23 **U**nit) interrupt remapping, this includes the CPU and the mainboard.
25 Generally, Intel systems with VT-d, and AMD systems with AMD-Vi support this.
26 But it is not guaranteed that everything will work out of the box, due
27 to bad hardware implementation and missing or low quality drivers.
29 Further, server grade hardware has often better support than consumer grade
30 hardware, but even then, many modern system can support this.
32 Please refer to your hardware vendor to check if they support this feature
33 under Linux for your specific setup
39 Once you ensured that your hardware supports passthrough, you will need to do
40 some configuration to enable PCI(e) passthrough.
46 The IOMMU has to be activated on the kernel commandline. The easiest way is to
47 enable trough grub. Edit `'/etc/default/grub'' and add the following to th
48 'GRUB_CMDLINE_LINUX_DEFAULT' variable:
61 To bring this change in effect, make sure you run:
70 You have to make sure the following modules are loaded. This can be achieved by
71 adding them to `'/etc/modules''
80 [[qm_pci_passthrough_update_initramfs]]
81 After changing anything modules related, you need to refresh your
82 `initramfs`. On {pve} this can be done by executing:
85 # update-initramfs -u -k all
91 Finally reboot to bring the changes into effect and check that it is indeed
95 # dmesg -e DMAR -e IOMMU -e AMD-Vi
98 should display that `IOMMU`, `Directed I/O` or `Interrupt Remapping` is
99 enabled, depending on hardware and kernel the exact message can vary.
101 It is also important that the device(s) you want to pass through
102 are in a *separate* `IOMMU` group. This can be checked with:
105 # find /sys/kernel/iommu_groups/ -type l
108 It is okay if the device is in an `IOMMU` group together with its functions
109 (e.g. a GPU with the HDMI Audio device) or with its root port or PCI(e) bridge.
114 Some platforms handle their physical PCI(e) slots differently. So, sometimes
115 it can help to put the card in a another PCI(e) slot, if you do not get the
116 desired `IOMMU` group separation.
122 For some platforms, it may be necessary to allow unsafe interrupts.
123 For this add the following line in a file ending with `.conf' file in
127 options vfio_iommu_type1 allow_unsafe_interrupts=1
130 Please be aware that this option can make your system unstable.
133 Host Device Passthrough
134 ~~~~~~~~~~~~~~~~~~~~~~~
136 The most used variant of PCI(e) passthrough is to pass through a whole
137 PCI(e) card, for example a GPU or a network card.
143 In this case, the host cannot use the card. There are two methods to achieve
146 * pass the device IDs to the options of the 'vfio-pci' modules by adding
149 options vfio-pci ids=1234:5678,4321:8765
152 to a .conf file in */etc/modprobe.d/* where `1234:5678` and `4321:8765` are
153 the vendor and device IDs obtained by:
159 * blacklist the driver completely on the host, ensuring that it is free to bind
160 for passthrough, with
166 in a .conf file in */etc/modprobe.d/*.
168 For both methods you need to
169 xref:qm_pci_passthrough_update_initramfs[update the `initramfs`] again and
172 [[qm_pci_passthrough_vm_config]]
175 To pass through the device you need to set the *hostpciX* option in the VM
176 configuration, for example by executing:
179 # qm set VMID -hostpci0 00:02.0
182 If your device has multiple functions, you can pass them through all together
183 with the shortened syntax ``00:02`'
185 There are some options to which may be necessary, depending on the device
188 * *x-vga=on|off* marks the PCI(e) device as the primary GPU of the VM.
189 With this enabled the *vga* configuration option will be ignored.
191 * *pcie=on|off* tells {pve} to use a PCIe or PCI port. Some guests/device
192 combination require PCIe rather than PCI. PCIe is only available for 'q35'
195 * *rombar=on|off* makes the firmware ROM visible for the guest. Default is on.
196 Some PCI(e) devices need this disabled.
198 * *romfile=<path>*, is an optional path to a ROM file for the device to use.
199 This is a relative path under */usr/share/kvm/*.
204 An example of PCIe passthrough with a GPU set to primary:
207 # qm set VMID -hostpci0 02:00,pcie=on,x-vga=on
214 When passing through a GPU, the best compatibility is reached when using
215 'q35' as machine type, 'OVMF' ('EFI' for VMs) instead of SeaBIOS and PCIe
216 instead of PCI. Note that if you want to use 'OVMF' for GPU passthrough, the
217 GPU needs to have an EFI capable ROM, otherwise use SeaBIOS instead.
222 Another variant for passing through PCI(e) devices, is to use the hardware
223 virtualization features of your devices, if available.
225 'SR-IOV' (**S**ingle-**R**oot **I**nput/**O**utput **V**irtualization) enables
226 a single device to provide multiple 'VF' (**V**irtual **F**unctions) to the
227 system. Each of those 'VF' can be used in a different VM, with full hardware
228 features and also better performance and lower latency than software
231 Currently, the most common use case for this are NICs (**N**etwork
232 **I**nterface **C**ard) with SR-IOV support, which can provide multiple VFs per
233 physical port. This allows using features such as checksum offloading, etc. to
234 be used inside a VM, reducing the (host) CPU overhead.
240 Generally, there are two methods for enabling virtual functions on a device.
242 * sometimes there is an option for the driver module e.g. for some
249 which could be put file with '.conf' ending under */etc/modprobe.d/*.
250 (Do not forget to update your initramfs after that)
252 Please refer to your driver module documentation for the exact
253 parameters and options.
255 * The second, more generic, approach is using the `sysfs`.
256 If a device and driver supports this you can change the number of VFs on
257 the fly. For example, to setup 4 VFs on device 0000:01:00.0 execute:
260 # echo 4 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs
263 To make this change persistent you can use the `sysfsutils` Debian package.
264 After installation configure it via */etc/sysfs.conf* or a `FILE.conf' inf
270 After creating VFs, you should see them as separate PCI(e) devices when
271 outputting them with `lspci`. Get their ID and pass them through like a
272 xref:qm_pci_passthrough_vm_config[normal PCI(e) device].
277 For this feature, platform support is especially important. It may be necessary
278 to enable this feature in the BIOS/EFI first, or to use a specific PCI(e) port
279 for it to work. In doubt, consult the manual of the platform or contact its