X-Git-Url: https://git.proxmox.com/?p=pve-docs.git;a=blobdiff_plain;f=qm-pci-passthrough.adoc;h=a347e31a5e4b277028e767190d513583b8d76435;hp=95e4ae1b5f412c40ad3137f1861cc2c6272f7a13;hb=7d6078845fa6a3bd308c7dc843273e56be33f315;hpb=6e4c46c4cb08bcd49c4384c2b0817923b1d63585 diff --git a/qm-pci-passthrough.adoc b/qm-pci-passthrough.adoc index 95e4ae1..a347e31 100644 --- a/qm-pci-passthrough.adoc +++ b/qm-pci-passthrough.adoc @@ -1,237 +1,370 @@ [[qm_pci_passthrough]] PCI(e) Passthrough ------------------ +ifdef::wiki[] +:pve-toplevel: +endif::wiki[] PCI(e) passthrough is a mechanism to give a virtual machine control over -a pci device usually only available for the host. This can have some -advantages over using virtualized hardware, for example lower latency, -higher performance, or more features (e.g., offloading). +a PCI device from the host. This can have some advantages over using +virtualized hardware, for example lower latency, higher performance, or more +features (e.g., offloading). -If you pass through a device to a virtual machine, you cannot use that +But, if you pass through a device to a virtual machine, you cannot use that device anymore on the host or in any other VM. General Requirements ~~~~~~~~~~~~~~~~~~~~ Since passthrough is a feature which also needs hardware support, there are -some requirements and steps before it can work. +some requirements to check and preparations to be done to make it work. + Hardware ^^^^^^^^ +Your hardware needs to support `IOMMU` (*I*/*O* **M**emory **M**anagement +**U**nit) interrupt remapping, this includes the CPU and the mainboard. -Your hardware has to support IOMMU interrupt remapping, this includes CPU and -Mainboard. - -Generally Intel systems with VT-d, and AMD systems with AMD-Vi support this, -but it is not guaranteed that everything will work out of the box, due -to bad hardware implementation or missing/low quality drivers. +Generally, Intel systems with VT-d, and AMD systems with AMD-Vi support this. +But it is not guaranteed that everything will work out of the box, due +to bad hardware implementation and missing or low quality drivers. -In most cases, server grade hardware has better support than consumer grade +Further, server grade hardware has often better support than consumer grade hardware, but even then, many modern system can support this. -Please refer to your hardware vendor if this is a feature that is supported -under Linux. +Please refer to your hardware vendor to check if they support this feature +under Linux for your specific setup + Configuration ^^^^^^^^^^^^^ -To enable PCI(e) passthrough, there are some configurations needed. +Once you ensured that your hardware supports passthrough, you will need to do +some configuration to enable PCI(e) passthrough. -First, the iommu has to be activated on the kernel commandline. -The easiest way is to enable it in */etc/default/grub*. Just add - intel_iommu=on +.IOMMU -or if you have AMD hardware: +The IOMMU has to be activated on the kernel commandline. The easiest way is to +enable trough grub. Edit `'/etc/default/grub'' and add the following to the +'GRUB_CMDLINE_LINUX_DEFAULT' variable: +* for Intel CPUs: ++ +---- + intel_iommu=on +---- +* for AMD CPUs: ++ +---- amd_iommu=on +---- + +[[qm_pci_passthrough_update_grub]] +To bring this change in effect, make sure you run: -to GRUB_CMDLINE_LINUX_DEFAULT +---- +# update-grub +---- -After that, make sure you run 'update grub' to update grub. +.Kernel Modules -Second, you have to make sure the following modules are loaded. -This can be achieved by adding them to */etc/modules* +You have to make sure the following modules are loaded. This can be achieved by +adding them to `'/etc/modules'' +---- vfio vfio_iommu_type1 vfio_pci vfio_virqfd +---- +[[qm_pci_passthrough_update_initramfs]] After changing anything modules related, you need to refresh your -initramfs with +`initramfs`. On {pve} this can be done by executing: ---- -update-initramfs -u -k all +# update-initramfs -u -k all ---- -Finally reboot and check that it is indeed enabled. +.Finish Configuration + +Finally reboot to bring the changes into effect and check that it is indeed +enabled. ---- -dmesg -e DMAR -e IOMMU -e AMD-Vi +# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi ---- -should display that IOMMU, Directed I/O or Interrupt Remapping is enabled. -(The exact message can vary, depending on hardware and kernel version) +should display that `IOMMU`, `Directed I/O` or `Interrupt Remapping` is +enabled, depending on hardware and kernel the exact message can vary. It is also important that the device(s) you want to pass through -are in a seperate IOMMU group. This can be checked with: +are in a *separate* `IOMMU` group. This can be checked with: ---- -find /sys/kernel/iommu_groups/ -type l +# find /sys/kernel/iommu_groups/ -type l ---- -It is okay if the device is in an IOMMU group together with its functions +It is okay if the device is in an `IOMMU` group together with its functions (e.g. a GPU with the HDMI Audio device) or with its root port or PCI(e) bridge. .PCI(e) slots [NOTE] ==== -Some platforms handle their PCI(e) slots differently, so if you -do not get the desired IOMMU group separation, it may be helpful to -try to put the card in a another PCI(e) slot. +Some platforms handle their physical PCI(e) slots differently. So, sometimes +it can help to put the card in a another PCI(e) slot, if you do not get the +desired `IOMMU` group separation. ==== .Unsafe interrupts [NOTE] ==== For some platforms, it may be necessary to allow unsafe interrupts. -This can most easily enabled with adding the following line -in a .conf file in */etc/modprobe.d/*. +For this add the following line in a file ending with `.conf' file in +*/etc/modprobe.d/*: +---- options vfio_iommu_type1 allow_unsafe_interrupts=1 +---- Please be aware that this option can make your system unstable. ==== -Host Device Passhtrough +GPU Passthrough Notes +^^^^^^^^^^^^^^^^^^^^^ + +It is not possible to display the frame buffer of the GPU via NoVNC or SPICE on +the {pve} web interface. + +When passing through a whole GPU or a vGPU and graphic output is wanted, one +has to either physically connect a monitor to the card, or configure a remote +desktop software (for example, VNC or RDP) inside the guest. + +If you want to use the GPU as a hardware accelerator, for example, for +programs using OpenCL or CUDA, this is not required. + +Host Device Passthrough ~~~~~~~~~~~~~~~~~~~~~~~ The most used variant of PCI(e) passthrough is to pass through a whole -PCI(e) card, for example a GPU or network card. +PCI(e) card, for example a GPU or a network card. + Host Configuration ^^^^^^^^^^^^^^^^^^ -In this case, the host can not use the card. This can be achieved by two -methods: - -Either add the ids to the options of the vfio-pci modules. This works -with adding +In this case, the host cannot use the card. There are two methods to achieve +this: +* pass the device IDs to the options of the 'vfio-pci' modules by adding ++ +---- options vfio-pci ids=1234:5678,4321:8765 - -to a .conf file in */etc/modprobe.d/* where 1234:5678 and 4321:8765 are -the vendor and device ids obtained by: - ---- -lcpci -nn ++ +to a .conf file in */etc/modprobe.d/* where `1234:5678` and `4321:8765` are +the vendor and device IDs obtained by: ++ +---- +# lcpci -nn ---- -Or simply blacklist the driver completely on the host with - +* blacklist the driver completely on the host, ensuring that it is free to bind +for passthrough, with ++ +---- blacklist DRIVERNAME +---- ++ +in a .conf file in */etc/modprobe.d/*. -also in a .conf file in */etc/modprobe.d/*. Again update the initramfs -and reboot after that. +For both methods you need to +xref:qm_pci_passthrough_update_initramfs[update the `initramfs`] again and +reboot after that. +[[qm_pci_passthrough_vm_config]] VM Configuration ^^^^^^^^^^^^^^^^ - -To pass through the device you set *hostpciX* on the VM with +To pass through the device you need to set the *hostpciX* option in the VM +configuration, for example by executing: ---- -qm set VMID -hostpci0 00:02.0 +# qm set VMID -hostpci0 00:02.0 ---- -If your device has multiple functions, you can pass them through all together -with the shortened syntax - - 00:02 +If your device has multiple functions (e.g., ``00:02.0`' and ``00:02.1`' ), +you can pass them through all together with the shortened syntax ``00:02`' There are some options to which may be necessary, depending on the device -and guest OS. +and guest OS: + +* *x-vga=on|off* marks the PCI(e) device as the primary GPU of the VM. +With this enabled the *vga* configuration option will be ignored. -* *x-vga=on|off* marks the PCI(e) device the primary GPU of the VM. -With this enabled the *vga* parameter of the config will be ignored. * *pcie=on|off* tells {pve} to use a PCIe or PCI port. Some guests/device -combination require PCIe rather than PCI (only available for q35 machine types). +combination require PCIe rather than PCI. PCIe is only available for 'q35' +machine types. + * *rombar=on|off* makes the firmware ROM visible for the guest. Default is on. Some PCI(e) devices need this disabled. + * *romfile=*, is an optional path to a ROM file for the device to use. -this is a relative path under */usr/share/kvm/*. +This is a relative path under */usr/share/kvm/*. + +.Example An example of PCIe passthrough with a GPU set to primary: ---- -qm set VMID -hostpci0 02:00,pcie=on,x-vga=on +# qm set VMID -hostpci0 02:00,pcie=on,x-vga=on ---- + Other considerations ^^^^^^^^^^^^^^^^^^^^ When passing through a GPU, the best compatibility is reached when using -q35 as machine type, OVMF instead of SeaBIOS and PCIe instead of PCI. -Note that if you want to use OVMF for GPU passthrough, the GPU needs -to have an EFI capable ROM, otherwise use SeaBIOS instead. +'q35' as machine type, 'OVMF' ('EFI' for VMs) instead of SeaBIOS and PCIe +instead of PCI. Note that if you want to use 'OVMF' for GPU passthrough, the +GPU needs to have an EFI capable ROM, otherwise use SeaBIOS instead. SR-IOV ~~~~~~ -Another variant of passing through PCI(e) devices, is to use the hardware -virtualization features of your devices. +Another variant for passing through PCI(e) devices, is to use the hardware +virtualization features of your devices, if available. + +'SR-IOV' (**S**ingle-**R**oot **I**nput/**O**utput **V**irtualization) enables +a single device to provide multiple 'VF' (**V**irtual **F**unctions) to the +system. Each of those 'VF' can be used in a different VM, with full hardware +features and also better performance and lower latency than software +virtualized devices. -SR-IOV (Single-root input/output virtualization) enables a single device -to provide multiple vf (virtual functions) to the system, so that each -vf can be used in a different VM, with full hardware features, better -performance and lower latency than software virtualized devices. +Currently, the most common use case for this are NICs (**N**etwork +**I**nterface **C**ard) with SR-IOV support, which can provide multiple VFs per +physical port. This allows using features such as checksum offloading, etc. to +be used inside a VM, reducing the (host) CPU overhead. -The most used devices for this are NICs with SR-IOV which can provide -multiple vf per physical port, allowing features such as -checksum offloading, etc. to be used inside a VM, reducing CPU overhead. Host Configuration ^^^^^^^^^^^^^^^^^^ -Generally there are 2 methods for enabling virtual functions on a device. +Generally, there are two methods for enabling virtual functions on a device. -In some cases there is an option for the driver module e.g. for some +* sometimes there is an option for the driver module e.g. for some Intel drivers - ++ +---- max_vfs=4 - -which could be put in a file in a .conf file in */etc/modprobe.d/*. +---- ++ +which could be put file with '.conf' ending under */etc/modprobe.d/*. (Do not forget to update your initramfs after that) - ++ Please refer to your driver module documentation for the exact parameters and options. -The second (more generic) approach is via the sysfs. -If a device and driver supports this you can change the number of vfs on -the fly. For example 4 vfs on device 0000:01:00.0 with: - +* The second, more generic, approach is using the `sysfs`. +If a device and driver supports this you can change the number of VFs on +the fly. For example, to setup 4 VFs on device 0000:01:00.0 execute: ++ ---- -echo 4 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs +# echo 4 > /sys/bus/pci/devices/0000:01:00.0/sriov_numvfs ---- ++ +To make this change persistent you can use the `sysfsutils` Debian package. +After installation configure it via */etc/sysfs.conf* or a `FILE.conf' in +*/etc/sysfs.d/*. + +VM Configuration +^^^^^^^^^^^^^^^^ + +After creating VFs, you should see them as separate PCI(e) devices when +outputting them with `lspci`. Get their ID and pass them through like a +xref:qm_pci_passthrough_vm_config[normal PCI(e) device]. + +Other considerations +^^^^^^^^^^^^^^^^^^^^ + +For this feature, platform support is especially important. It may be necessary +to enable this feature in the BIOS/EFI first, or to use a specific PCI(e) port +for it to work. In doubt, consult the manual of the platform or contact its +vendor. + +Mediated Devices (vGPU, GVT-g) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Mediated devices are another method to use reuse features and performance from +physical hardware for virtualized hardware. These are found most common in +virtualized GPU setups such as Intels GVT-g and Nvidias vGPUs used in their +GRID technology. + +With this, a physical Card is able to create virtual cards, similar to SR-IOV. +The difference is that mediated devices do not appear as PCI(e) devices in the +host, and are such only suited for using in virtual machines. + + +Host Configuration +^^^^^^^^^^^^^^^^^^ + +In general your card's driver must support that feature, otherwise it will +not work. So please refer to your vendor for compatbile drivers and how to +configure them. -To make this change persistent you can use sysfsutils. -Just install them via +Intels drivers for GVT-g are integraded in the Kernel and should work +with the 5th, 6th and 7th generation Intel Core Processors, further E3 v4, E3 +v5 and E3 v6 Xeon Processors are supported. + +To enable it for Intel Graphcs, you have to make sure to load the module +'kvmgt' (for example via `/etc/modules`) and to enable it on the Kernel +commandline. For this you can edit `'/etc/default/grub'' and add the following +to the 'GRUB_CMDLINE_LINUX_DEFAULT' variable: ---- -apt install sysfsutils + i915.enable_gvt=1 ---- -and configure it via */etc/sysfs.conf* or */etc/sysfs.d/*. +After that remember to +xref:qm_pci_passthrough_update_initramfs[update the `initramfs`], +xref:qm_pci_passthrough_update_grub[update grub] and +reboot your host. VM Configuration ^^^^^^^^^^^^^^^^ -After creating vfs, you should see them as seperate PCI(e) devices, which -can be passed through like a normal PCI(e) device. +To use a mediated device, simply specify the `mdev` property on a `hostpciX` +VM configuration option. -Other considerations -^^^^^^^^^^^^^^^^^^^^ +You can get the supported devices via the 'sysfs'. For example, to list the +supported types for the device '0000:00:02.0' you would simply execute: -For this feature, platform support is especially important. It may be necessary -to enable this feature in the BIOS or to use a specific PCI(e) port for it -to work. In doubt, consult the manual of the platform or contact the vendor. +---- +# ls /sys/bus/pci/devices/0000:00:02.0/mdev_supported_types +---- + +Each entry is a directory which contains the following important files: + +* 'available_instances' contains the amount of still available instances of +this type, each 'mdev' use in a VM reduces this. +* 'description' contains a short description about the capabilities of the type +* 'create' is the endpoint to create such a device, {pve} does this +automatically for you, if a 'hostpciX' option with `mdev` is configured. + +Example configuration with an `Intel GVT-g vGPU` (`Intel Skylake 6700k`): + +---- +# qm set VMID -hostpci0 00:02.0,mdev=i915-GVTg_V5_4 +---- + +With this set, {pve} automatically creates such a device on VM start, and +cleans it up again when the VM stops. + +ifdef::wiki[] + +See Also +~~~~~~~~ + +* link:/wiki/Pci_passthrough[PCI Passthrough Examples] + +endif::wiki[]