5 PCI(e) passthrough is a mechanism to give a virtual machine control over
6 a PCI device from the host. This can have some advantages over using
7 virtualized hardware, for example lower latency, higher performance, or more
8 features (e.g., offloading).
10 But, if you pass through a device to a virtual machine, you cannot use that
11 device anymore on the host or in any other VM.
16 Since passthrough is a feature which also needs hardware support, there are
17 some requirements to check and preparations to be done to make it work.
22 Your hardware needs to support `IOMMU` (*I*/*O* **M**emory **M**anagement
23 **U**nit) interrupt remapping, this includes the CPU and the mainboard.
25 Generally, Intel systems with VT-d, and AMD systems with AMD-Vi support this.
26 But it is not guaranteed that everything will work out of the box, due
27 to bad hardware implementation and missing or low quality drivers.
29 Further, server grade hardware has often better support than consumer grade
30 hardware, but even then, many modern system can support this.
32 Please refer to your hardware vendor to check if they support this feature
33 under Linux for your specific setup
39 Once you ensured that your hardware supports passthrough, you will need to do
40 some configuration to enable PCI(e) passthrough.
45 The IOMMU has to be activated on the kernel commandline. The easiest way is to
46 enable trough grub. Edit `'/etc/default/grub'' and add the following to the
47 'GRUB_CMDLINE_LINUX_DEFAULT' variable:
60 [[qm_pci_passthrough_update_grub]]
61 To bring this change in effect, make sure you run:
69 You have to make sure the following modules are loaded. This can be achieved by
70 adding them to `'/etc/modules''
79 [[qm_pci_passthrough_update_initramfs]]
80 After changing anything modules related, you need to refresh your
81 `initramfs`. On {pve} this can be done by executing:
84 # update-initramfs -u -k all
89 Finally reboot to bring the changes into effect and check that it is indeed
93 # dmesg -e DMAR -e IOMMU -e AMD-Vi
96 should display that `IOMMU`, `Directed I/O` or `Interrupt Remapping` is
97 enabled, depending on hardware and kernel the exact message can vary.
99 It is also important that the device(s) you want to pass through
100 are in a *separate* `IOMMU` group. This can be checked with:
103 # find /sys/kernel/iommu_groups/ -type l
106 It is okay if the device is in an `IOMMU` group together with its functions
107 (e.g. a GPU with the HDMI Audio device) or with its root port or PCI(e) bridge.
112 Some platforms handle their physical PCI(e) slots differently. So, sometimes
113 it can help to put the card in a another PCI(e) slot, if you do not get the
114 desired `IOMMU` group separation.
120 For some platforms, it may be necessary to allow unsafe interrupts.
121 For this add the following line in a file ending with `.conf' file in
125 options vfio_iommu_type1 allow_unsafe_interrupts=1
128 Please be aware that this option can make your system unstable.
131 GPU Passthrough Notes
132 ^^^^^^^^^^^^^^^^^^^^^
134 It is not possible to display the frame buffer of the GPU via NoVNC or SPICE on
135 the {pve} web interface.
137 When passing through a whole GPU or a vGPU and graphic output is wanted, one
138 has to either physically connect a monitor to the card, or configure a remote
139 desktop software (for example, VNC or RDP) inside the guest.
141 If you want to use the GPU as a hardware accelerator, for example, for
142 programs using OpenCL or CUDA, this is not required.
144 Host Device Passthrough
145 ~~~~~~~~~~~~~~~~~~~~~~~
147 The most used variant of PCI(e) passthrough is to pass through a whole
148 PCI(e) card, for example a GPU or a network card.
154 In this case, the host cannot use the card. There are two methods to achieve
157 * pass the device IDs to the options of the 'vfio-pci' modules by adding
160 options vfio-pci ids=1234:5678,4321:8765
163 to a .conf file in */etc/modprobe.d/* where `1234:5678` and `4321:8765` are
164 the vendor and device IDs obtained by:
170 * blacklist the driver completely on the host, ensuring that it is free to bind
171 for passthrough, with
177 in a .conf file in */etc/modprobe.d/*.
179 For both methods you need to
180 xref:qm_pci_passthrough_update_initramfs[update the `initramfs`] again and
183 [[qm_pci_passthrough_vm_config]]
186 To pass through the device you need to set the *hostpciX* option in the VM
187 configuration, for example by executing:
190 # qm set VMID -hostpci0 00:02.0
193 If your device has multiple functions (e.g., ``00:02.0`' and ``00:02.1`'),
194 you can pass them through all together with the shortened syntax ``00:02`'
196 There are some options to which may be necessary, depending on the device
199 * *x-vga=on|off* marks the PCI(e) device as the primary GPU of the VM.
200 With this enabled the *vga* configuration option will be ignored.
202 * *pcie=on|off* tells {pve} to use a PCIe or PCI port. Some guests/device
203 combination require PCIe rather than PCI. PCIe is only available for 'q35'
206 * *rombar=on|off* makes the firmware ROM visible for the guest. Default is on.
207 Some PCI(e) devices need this disabled.
209 * *romfile=<path>*, is an optional path to a ROM file for the device to use.
210 This is a relative path under */usr/share/kvm/*.
214 An example of PCIe passthrough with a GPU set to primary:
217 # qm set VMID -hostpci0 02:00,pcie=on,x-vga=on
224 When passing through a GPU, the best compatibility is reached when using
225 'q35' as machine type, 'OVMF' ('EFI' for VMs) instead of SeaBIOS and PCIe
226 instead of PCI. Note that if you want to use 'OVMF' for GPU passthrough, the
227 GPU needs to have an EFI capable ROM, otherwise use SeaBIOS instead.
232 Another variant for passing through PCI(e) devices, is to use the hardware
233 virtualization features of your devices, if available.
235 'SR-IOV' (**S**ingle-**R**oot **I**nput/**O**utput **V**irtualization) enables
236 a single device to provide multiple 'VF' (**V**irtual **F**unctions) to the
237 system. Each of those 'VF' can be used in a different VM, with full hardware
238 features and also better performance and lower latency than software
241 Currently, the most common use case for this are NICs (**N**etwork
242 **I**nterface **C**ard) with SR-IOV support, which can provide multiple VFs per
243 physical port. This allows using features such as checksum offloading, etc. to
244 be used inside a VM, reducing the (host) CPU overhead.
250 Generally, there are two methods for enabling virtual functions on a device.
252 * sometimes there is an option for the driver module e.g. for some
259 which could be put file with '.conf' ending under */etc/modprobe.d/*.
260 (Do not forget to update your initramfs after that)
262 Please refer to your driver module documentation for the exact
263 parameters and options.
265 * The second, more generic, approach is using the `sysfs`.
266 If a device and driver supports this you can change the number of VFs on
267 the fly. For example, to setup 4 VFs on device 0000:01:00.0 execute:
270 # echo 4 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs
273 To make this change persistent you can use the `sysfsutils` Debian package.
274 After installation configure it via */etc/sysfs.conf* or a `FILE.conf' in
280 After creating VFs, you should see them as separate PCI(e) devices when
281 outputting them with `lspci`. Get their ID and pass them through like a
282 xref:qm_pci_passthrough_vm_config[normal PCI(e) device].
287 For this feature, platform support is especially important. It may be necessary
288 to enable this feature in the BIOS/EFI first, or to use a specific PCI(e) port
289 for it to work. In doubt, consult the manual of the platform or contact its
292 Mediated Devices (vGPU, GVT-g)
293 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
295 Mediated devices are another method to use reuse features and performance from
296 physical hardware for virtualized hardware. These are found most common in
297 virtualized GPU setups such as Intels GVT-g and Nvidias vGPUs used in their
300 With this, a physical Card is able to create virtual cards, similar to SR-IOV.
301 The difference is that mediated devices do not appear as PCI(e) devices in the
302 host, and are such only suited for using in virtual machines.
308 In general your card's driver must support that feature, otherwise it will
309 not work. So please refer to your vendor for compatbile drivers and how to
312 Intels drivers for GVT-g are integraded in the Kernel and should work
313 with the 5th, 6th and 7th generation Intel Core Processors, further E3 v4, E3
314 v5 and E3 v6 Xeon Processors are supported.
316 To enable it for Intel Graphcs, you have to make sure to load the module
317 'kvmgt' (for example via `/etc/modules`) and to enable it on the Kernel
318 commandline. For this you can edit `'/etc/default/grub'' and add the following
319 to the 'GRUB_CMDLINE_LINUX_DEFAULT' variable:
325 After that remember to
326 xref:qm_pci_passthrough_update_initramfs[update the `initramfs`],
327 xref:qm_pci_passthrough_update_grub[update grub] and
333 To use a mediated device, simply specify the `mdev` property on a `hostpciX`
334 VM configuration option.
336 You can get the supported devices via the 'sysfs'. For example, to list the
337 supported types for the device '0000:00:02.0' you would simply execute:
340 # ls /sys/bus/pci/devices/0000:00:02.0/mdev_supported_types
343 Each entry is a directory which contains the following important files:
345 * 'available_instances' contains the amount of still available instances of
346 this type, each 'mdev' use in a VM reduces this.
347 * 'description' contains a short description about the capabilities of the type
348 * 'create' is the endpoint to create such a device, {pve} does this
349 automatically for you, if a 'hostpciX' option with `mdev` is configured.
351 Example configuration with an `Intel GVT-g vGPU` (`Intel Skylake 6700k`):
354 # qm set VMID -hostpci0 00:02.0,mdev=i915-GVTg_V5_4
357 With this set, {pve} automatically creates such a device on VM start, and
358 cleans it up again when the VM stops.