8 PCI(e) passthrough is a mechanism to give a virtual machine control over
9 a PCI device from the host. This can have some advantages over using
10 virtualized hardware, for example lower latency, higher performance, or more
11 features (e.g., offloading).
13 But, if you pass through a device to a virtual machine, you cannot use that
14 device anymore on the host or in any other VM.
19 Since passthrough is a feature which also needs hardware support, there are
20 some requirements to check and preparations to be done to make it work.
25 Your hardware needs to support `IOMMU` (*I*/*O* **M**emory **M**anagement
26 **U**nit) interrupt remapping, this includes the CPU and the mainboard.
28 Generally, Intel systems with VT-d, and AMD systems with AMD-Vi support this.
29 But it is not guaranteed that everything will work out of the box, due
30 to bad hardware implementation and missing or low quality drivers.
32 Further, server grade hardware has often better support than consumer grade
33 hardware, but even then, many modern system can support this.
35 Please refer to your hardware vendor to check if they support this feature
36 under Linux for your specific setup.
42 Once you ensured that your hardware supports passthrough, you will need to do
43 some configuration to enable PCI(e) passthrough.
47 First, you have to enable IOMMU support in your BIOS/UEFI. Usually the
48 corresponding setting is called `IOMMU` or `VT-d`,but you should find the exact
49 option name in the manual of your motherboard.
51 For Intel CPUs, you may also need to enable the IOMMU on the
52 xref:sysboot_edit_kernel_cmdline[kernel command line] for older (pre-5.15)
59 For AMD CPUs it should be enabled automatically.
61 .IOMMU Passthrough Mode
63 If your hardware supports IOMMU passthrough mode, enabling this mode might
65 This is because VMs then bypass the (default) DMA translation normally
66 performed by the hyper-visor and instead pass DMA requests directly to the
67 hardware IOMMU. To enable these options, add:
73 to the xref:sysboot_edit_kernel_cmdline[kernel commandline].
77 You have to make sure the following modules are loaded. This can be achieved by
78 adding them to `'/etc/modules''
87 [[qm_pci_passthrough_update_initramfs]]
88 After changing anything modules related, you need to refresh your
89 `initramfs`. On {pve} this can be done by executing:
92 # update-initramfs -u -k all
97 Finally reboot to bring the changes into effect and check that it is indeed
101 # dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
104 should display that `IOMMU`, `Directed I/O` or `Interrupt Remapping` is
105 enabled, depending on hardware and kernel the exact message can vary.
107 It is also important that the device(s) you want to pass through
108 are in a *separate* `IOMMU` group. This can be checked with:
111 # find /sys/kernel/iommu_groups/ -type l
114 It is okay if the device is in an `IOMMU` group together with its functions
115 (e.g. a GPU with the HDMI Audio device) or with its root port or PCI(e) bridge.
120 Some platforms handle their physical PCI(e) slots differently. So, sometimes
121 it can help to put the card in a another PCI(e) slot, if you do not get the
122 desired `IOMMU` group separation.
128 For some platforms, it may be necessary to allow unsafe interrupts.
129 For this add the following line in a file ending with `.conf' file in
133 options vfio_iommu_type1 allow_unsafe_interrupts=1
136 Please be aware that this option can make your system unstable.
139 GPU Passthrough Notes
140 ^^^^^^^^^^^^^^^^^^^^^
142 It is not possible to display the frame buffer of the GPU via NoVNC or SPICE on
143 the {pve} web interface.
145 When passing through a whole GPU or a vGPU and graphic output is wanted, one
146 has to either physically connect a monitor to the card, or configure a remote
147 desktop software (for example, VNC or RDP) inside the guest.
149 If you want to use the GPU as a hardware accelerator, for example, for
150 programs using OpenCL or CUDA, this is not required.
152 Host Device Passthrough
153 ~~~~~~~~~~~~~~~~~~~~~~~
155 The most used variant of PCI(e) passthrough is to pass through a whole
156 PCI(e) card, for example a GPU or a network card.
162 In this case, the host must not use the card. There are two methods to achieve
165 * pass the device IDs to the options of the 'vfio-pci' modules by adding
168 options vfio-pci ids=1234:5678,4321:8765
171 to a .conf file in */etc/modprobe.d/* where `1234:5678` and `4321:8765` are
172 the vendor and device IDs obtained by:
178 * blacklist the driver completely on the host, ensuring that it is free to bind
179 for passthrough, with
185 in a .conf file in */etc/modprobe.d/*.
187 For both methods you need to
188 xref:qm_pci_passthrough_update_initramfs[update the `initramfs`] again and
191 .Verify Configuration
193 To check if your changes were successful, you can use
199 and check your device entry. If it says
202 Kernel driver in use: vfio-pci
205 or the 'in use' line is missing entirely, the device is ready to be used for
208 [[qm_pci_passthrough_vm_config]]
211 To pass through the device you need to set the *hostpciX* option in the VM
212 configuration, for example by executing:
215 # qm set VMID -hostpci0 00:02.0
218 If your device has multiple functions (e.g., ``00:02.0`' and ``00:02.1`' ),
219 you can pass them through all together with the shortened syntax ``00:02`'.
220 This is equivalent with checking the ``All Functions`' checkbox in the
223 There are some options to which may be necessary, depending on the device
226 * *x-vga=on|off* marks the PCI(e) device as the primary GPU of the VM.
227 With this enabled the *vga* configuration option will be ignored.
229 * *pcie=on|off* tells {pve} to use a PCIe or PCI port. Some guests/device
230 combination require PCIe rather than PCI. PCIe is only available for 'q35'
233 * *rombar=on|off* makes the firmware ROM visible for the guest. Default is on.
234 Some PCI(e) devices need this disabled.
236 * *romfile=<path>*, is an optional path to a ROM file for the device to use.
237 This is a relative path under */usr/share/kvm/*.
241 An example of PCIe passthrough with a GPU set to primary:
244 # qm set VMID -hostpci0 02:00,pcie=on,x-vga=on
249 You can override the PCI vendor ID, device ID, and subsystem IDs that will be
250 seen by the guest. This is useful if your device is a variant with an ID that
251 your guest's drivers don't recognize, but you want to force those drivers to be
252 loaded anyway (e.g. if you know your device shares the same chipset as a
255 The available options are `vendor-id`, `device-id`, `sub-vendor-id`, and
256 `sub-device-id`. You can set any or all of these to override your device's
262 # qm set VMID -hostpci0 02:00,device-id=0x10f6,sub-vendor-id=0x0000
269 When passing through a GPU, the best compatibility is reached when using
270 'q35' as machine type, 'OVMF' ('EFI' for VMs) instead of SeaBIOS and PCIe
271 instead of PCI. Note that if you want to use 'OVMF' for GPU passthrough, the
272 GPU needs to have an EFI capable ROM, otherwise use SeaBIOS instead.
277 Another variant for passing through PCI(e) devices, is to use the hardware
278 virtualization features of your devices, if available.
280 'SR-IOV' (**S**ingle-**R**oot **I**nput/**O**utput **V**irtualization) enables
281 a single device to provide multiple 'VF' (**V**irtual **F**unctions) to the
282 system. Each of those 'VF' can be used in a different VM, with full hardware
283 features and also better performance and lower latency than software
286 Currently, the most common use case for this are NICs (**N**etwork
287 **I**nterface **C**ard) with SR-IOV support, which can provide multiple VFs per
288 physical port. This allows using features such as checksum offloading, etc. to
289 be used inside a VM, reducing the (host) CPU overhead.
295 Generally, there are two methods for enabling virtual functions on a device.
297 * sometimes there is an option for the driver module e.g. for some
304 which could be put file with '.conf' ending under */etc/modprobe.d/*.
305 (Do not forget to update your initramfs after that)
307 Please refer to your driver module documentation for the exact
308 parameters and options.
310 * The second, more generic, approach is using the `sysfs`.
311 If a device and driver supports this you can change the number of VFs on
312 the fly. For example, to setup 4 VFs on device 0000:01:00.0 execute:
315 # echo 4 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs
318 To make this change persistent you can use the `sysfsutils` Debian package.
319 After installation configure it via */etc/sysfs.conf* or a `FILE.conf' in
325 After creating VFs, you should see them as separate PCI(e) devices when
326 outputting them with `lspci`. Get their ID and pass them through like a
327 xref:qm_pci_passthrough_vm_config[normal PCI(e) device].
332 For this feature, platform support is especially important. It may be necessary
333 to enable this feature in the BIOS/EFI first, or to use a specific PCI(e) port
334 for it to work. In doubt, consult the manual of the platform or contact its
337 Mediated Devices (vGPU, GVT-g)
338 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
340 Mediated devices are another method to reuse features and performance from
341 physical hardware for virtualized hardware. These are found most common in
342 virtualized GPU setups such as Intel's GVT-g and NVIDIA's vGPUs used in their
345 With this, a physical Card is able to create virtual cards, similar to SR-IOV.
346 The difference is that mediated devices do not appear as PCI(e) devices in the
347 host, and are such only suited for using in virtual machines.
353 In general your card's driver must support that feature, otherwise it will
354 not work. So please refer to your vendor for compatible drivers and how to
357 Intel's drivers for GVT-g are integrated in the Kernel and should work
358 with 5th, 6th and 7th generation Intel Core Processors, as well as E3 v4, E3
359 v5 and E3 v6 Xeon Processors.
361 To enable it for Intel Graphics, you have to make sure to load the module
362 'kvmgt' (for example via `/etc/modules`) and to enable it on the
363 xref:sysboot_edit_kernel_cmdline[Kernel commandline] and add the following parameter:
369 After that remember to
370 xref:qm_pci_passthrough_update_initramfs[update the `initramfs`],
371 and reboot your host.
376 To use a mediated device, simply specify the `mdev` property on a `hostpciX`
377 VM configuration option.
379 You can get the supported devices via the 'sysfs'. For example, to list the
380 supported types for the device '0000:00:02.0' you would simply execute:
383 # ls /sys/bus/pci/devices/0000:00:02.0/mdev_supported_types
386 Each entry is a directory which contains the following important files:
388 * 'available_instances' contains the amount of still available instances of
389 this type, each 'mdev' use in a VM reduces this.
390 * 'description' contains a short description about the capabilities of the type
391 * 'create' is the endpoint to create such a device, {pve} does this
392 automatically for you, if a 'hostpciX' option with `mdev` is configured.
394 Example configuration with an `Intel GVT-g vGPU` (`Intel Skylake 6700k`):
397 # qm set VMID -hostpci0 00:02.0,mdev=i915-GVTg_V5_4
400 With this set, {pve} automatically creates such a device on VM start, and
401 cleans it up again when the VM stops.
408 * link:/wiki/Pci_passthrough[PCI Passthrough Examples]