Emulated devices and paravirtualized devices
--------------------------------------------
-The PC hardware emulated by QEMU includes a mainboard, network controllers,
+The PC hardware emulated by QEMU includes a motherboard, network controllers,
SCSI, IDE and SATA controllers, serial ports (the complete list can be seen in
the `kvm(1)` man page) all of them emulated in software. All these devices
are the exact software equivalent of existing hardware devices, and if the OS
running in the guest has the proper drivers it will use the devices as if it
-were running on real hardware. This allows QEMU to runs _unmodified_ operating
+were running on real hardware. This allows QEMU to run _unmodified_ operating
systems.
This however has a performance cost, as running in software what was meant to
{pve} allows to boot VMs with different firmware and machine types, namely
xref:qm_bios_and_uefi[SeaBIOS and OVMF]. In most cases you want to switch from
the default SeaBIOS to OVMF only if you plan to use
-xref:qm_pci_passthrough[PCIe pass through]. A VMs 'Machine Type' defines the
-hardware layout of the VM's virtual motherboard. You can choose between the
-default https://en.wikipedia.org/wiki/Intel_440FX[Intel 440FX] or the
+xref:qm_pci_passthrough[PCIe passthrough].
+
+[[qm_machine_type]]
+
+Machine Type
+^^^^^^^^^^^^
+
+A VM's 'Machine Type' defines the hardware layout of the VM's virtual
+motherboard. You can choose between the default
+https://en.wikipedia.org/wiki/Intel_440FX[Intel 440FX] or the
https://ark.intel.com/content/www/us/en/ark/products/31918/intel-82q35-graphics-and-memory-controller.html[Q35]
-chipset, which also provides a virtual PCIe bus, and thus may be desired if
-one wants to pass through PCIe hardware.
+chipset, which also provides a virtual PCIe bus, and thus may be
+desired if you want to pass through PCIe hardware.
+Additionally, you can select a xref:qm_pci_viommu[vIOMMU] implementation.
+
+Machine Version
++++++++++++++++
+
+Each machine type is versioned in QEMU and a given QEMU binary supports many
+machine versions. New versions might bring support for new features, fixes or
+general improvements. However, they also change properties of the virtual
+hardware. To avoid sudden changes from the guest's perspective and ensure
+compatibility of the VM state, live-migration and snapshots with RAM will keep
+using the same machine version in the new QEMU instance.
+
+For Windows guests, the machine version is pinned during creation, because
+Windows is sensitive to changes in the virtual hardware - even between cold
+boots. For example, the enumeration of network devices might be different with
+different machine versions. Other OSes like Linux can usually deal with such
+changes just fine. For those, the 'Latest' machine version is used by default.
+This means that after a fresh start, the newest machine version supported by the
+QEMU binary is used (e.g. the newest machine version QEMU 8.1 supports is
+version 8.1 for each machine type).
+
+[[qm_machine_update]]
+
+Update to a Newer Machine Version
++++++++++++++++++++++++++++++++++
+
+Very old machine versions might become deprecated in QEMU. For example, this is
+the case for versions 1.4 to 1.7 for the i440fx machine type. It is expected
+that support for these machine versions will be dropped at some point. If you
+see a deprecation warning, you should change the machine version to a newer one.
+Be sure to have a working backup first and be prepared for changes to how the
+guest sees hardware. In some scenarios, re-installing certain drivers might be
+required. You should also check for snapshots with RAM that were taken with
+these machine versions (i.e. the `runningmachine` configuration entry).
+Unfortunately, there is no way to change the machine version of a snapshot, so
+you'd need to load the snapshot to salvage any data from it.
[[qm_hard_disk]]
Hard Disk
Resource Limits
^^^^^^^^^^^^^^^
-In addition to the number of virtual cores, you can configure how much resources
-a VM can get in relation to the host CPU time and also in relation to other
-VMs.
-With the *cpulimit* (``Host CPU Time'') option you can limit how much CPU time
-the whole VM can use on the host. It is a floating point value representing CPU
-time in percent, so `1.0` is equal to `100%`, `2.5` to `250%` and so on. If a
-single process would fully use one single core it would have `100%` CPU Time
-usage. If a VM with four cores utilizes all its cores fully it would
-theoretically use `400%`. In reality the usage may be even a bit higher as QEMU
-can have additional threads for VM peripherals besides the vCPU core ones.
-This setting can be useful if a VM should have multiple vCPUs, as it runs a few
-processes in parallel, but the VM as a whole should not be able to run all
-vCPUs at 100% at the same time. Using a specific example: lets say we have a VM
-which would profit from having 8 vCPUs, but at no time all of those 8 cores
-should run at full load - as this would make the server so overloaded that
-other VMs and CTs would get to less CPU. So, we set the *cpulimit* limit to
-`4.0` (=400%). If all cores do the same heavy work they would all get 50% of a
-real host cores CPU time. But, if only 4 would do work they could still get
-almost 100% of a real core each.
+*cpulimit*
+
+In addition to the number of virtual cores, the total available ``Host CPU
+Time'' for the VM can be set with the *cpulimit* option. It is a floating point
+value representing CPU time in percent, so `1.0` is equal to `100%`, `2.5` to
+`250%` and so on. If a single process would fully use one single core it would
+have `100%` CPU Time usage. If a VM with four cores utilizes all its cores
+fully it would theoretically use `400%`. In reality the usage may be even a bit
+higher as QEMU can have additional threads for VM peripherals besides the vCPU
+core ones.
+
+This setting can be useful when a VM should have multiple vCPUs because it is
+running some processes in parallel, but the VM as a whole should not be able to
+run all vCPUs at 100% at the same time.
+
+For example, suppose you have a virtual machine that would benefit from having 8
+virtual CPUs, but you don't want the VM to be able to max out all 8 cores
+running at full load - because that would overload the server and leave other
+virtual machines and containers with too little CPU time. To solve this, you
+could set *cpulimit* to `4.0` (=400%). This means that if the VM fully utilizes
+all 8 virtual CPUs by running 8 processes simultaneously, each vCPU will receive
+a maximum of 50% CPU time from the physical cores. However, if the VM workload
+only fully utilizes 4 virtual CPUs, it could still receive up to 100% CPU time
+from a physical core, for a total of 400%.
NOTE: VMs can, depending on their configuration, use additional threads, such
as for networking or IO operations but also live migration. Thus a VM can show
up to use more CPU time than just its virtual CPUs could use. To ensure that a
-VM never uses more CPU time than virtual CPUs assigned set the *cpulimit*
-setting to the same value as the total core count.
+VM never uses more CPU time than vCPUs assigned, set the *cpulimit* to
+the same value as the total core count.
+
+*cpuuntis*
+
+With the *cpuunits* option, nowadays often called CPU shares or CPU weight, you
+can control how much CPU time a VM gets compared to other running VMs. It is a
+relative weight which defaults to `100` (or `1024` if the host uses legacy
+cgroup v1). If you increase this for a VM it will be prioritized by the
+scheduler in comparison to other VMs with lower weight.
-The second CPU resource limiting setting, *cpuunits* (nowadays often called CPU
-shares or CPU weight), controls how much CPU time a VM gets compared to other
-running VMs. It is a relative weight which defaults to `100` (or `1024` if the
-host uses legacy cgroup v1). If you increase this for a VM it will be
-prioritized by the scheduler in comparison to other VMs with lower weight. For
-example, if VM 100 has set the default `100` and VM 200 was changed to `200`,
-the latter VM 200 would receive twice the CPU bandwidth than the first VM 100.
+For example, if VM 100 has set the default `100` and VM 200 was changed to
+`200`, the latter VM 200 would receive twice the CPU bandwidth than the first
+VM 100.
For more information see `man systemd.resource-control`, here `CPUQuota`
-corresponds to `cpulimit` and `CPUWeight` corresponds to our `cpuunits`
-setting, visit its Notes section for references and implementation details.
+corresponds to `cpulimit` and `CPUWeight` to our `cpuunits` setting. Visit its
+Notes section for references and implementation details.
-The third CPU resource limiting setting, *affinity*, controls what host cores
-the virtual machine will be permitted to execute on. E.g., if an affinity value
-of `0-3,8-11` is provided, the virtual machine will be restricted to using the
-host cores `0,1,2,3,8,9,10,` and `11`. Valid *affinity* values are written in
-cpuset `List Format`. List Format is a comma-separated list of CPU numbers and
-ranges of numbers, in ASCII decimal.
+*affinity*
-NOTE: CPU *affinity* uses the `taskset` command to restrict virtual machines to
-a given set of cores. This restriction will not take effect for some types of
-processes that may be created for IO. *CPU affinity is not a security feature.*
+With the *affinity* option, you can specify the physical CPU cores that are used
+to run the VM's vCPUs. Peripheral VM processes, such as those for I/O, are not
+affected by this setting. Note that the *CPU affinity is not a security
+feature*.
-For more information regarding *affinity* see `man cpuset`. Here the
-`List Format` corresponds to valid *affinity* values. Visit its `Formats`
-section for more examples.
+Forcing a CPU *affinity* can make sense in certain cases but is accompanied by
+an increase in complexity and maintenance effort. For example, if you want to
+add more VMs later or migrate VMs to nodes with fewer CPU cores. It can also
+easily lead to asynchronous and therefore limited system performance if some
+CPUs are fully utilized while others are almost idle.
+
+The *affinity* is set through the `taskset` CLI tool. It accepts the host CPU
+numbers (see `lscpu`) in the `List Format` from `man cpuset`. This ASCII decimal
+list can contain numbers but also number ranges. For example, the *affinity*
+`0-1,8-11` (expanded `0, 1, 8, 9, 10, 11`) would allow the VM to run on only
+these six specific host cores.
CPU Type
^^^^^^^^
QEMU can emulate a number different of *CPU types* from 486 to the latest Xeon
processors. Each new processor generation adds new features, like hardware
-assisted 3d rendering, random number generation, memory protection, etc ...
+assisted 3d rendering, random number generation, memory protection, etc. Also,
+a current generation can be upgraded through
+xref:chapter_firmware_updates[microcode update] with bug or security fixes.
+
Usually you should select for your VM a processor type which closely matches the
CPU of the host system, as it means that the host CPU features (also called _CPU
flags_ ) will be available in your VMs. If you want an exact match, you can set
as your host system.
This has a downside though. If you want to do a live migration of VMs between
-different hosts, your VM might end up on a new system with a different CPU type.
-If the CPU flags passed to the guest are missing, the qemu process will stop. To
-remedy this QEMU has also its own CPU type *kvm64*, that {pve} uses by defaults.
-kvm64 is a Pentium 4 look a like CPU type, which has a reduced CPU flags set,
-but is guaranteed to work everywhere.
+different hosts, your VM might end up on a new system with a different CPU type
+or a different microcode version.
+If the CPU flags passed to the guest are missing, the QEMU process will stop. To
+remedy this QEMU has also its own virtual CPU types, that {pve} uses by default.
+
+The backend default is 'kvm64' which works on essentially all x86_64 host CPUs
+and the UI default when creating a new VM is 'x86-64-v2-AES', which requires a
+host CPU starting from Westmere for Intel or at least a fourth generation
+Opteron for AMD.
+
+In short:
+
+If you don’t care about live migration or have a homogeneous cluster where all
+nodes have the same CPU and same microcode version, set the CPU type to host, as
+in theory this will give your guests maximum performance.
+
+If you care about live migration and security, and you have only Intel CPUs or
+only AMD CPUs, choose the lowest generation CPU model of your cluster.
-In short, if you care about live migration and moving VMs between nodes, leave
-the kvm64 default. If you don’t care about live migration or have a homogeneous
-cluster where all nodes have the same CPU, set the CPU type to host, as in
-theory this will give your guests maximum performance.
+If you care about live migration without security, or have mixed Intel/AMD
+cluster, choose the lowest compatible virtual QEMU CPU type.
+
+NOTE: Live migrations between Intel and AMD host CPUs have no guarantee to work.
+
+See also
+xref:chapter_qm_vcpu_list[List of AMD and Intel CPU Types as Defined in QEMU].
+
+QEMU CPU Types
+^^^^^^^^^^^^^^
+
+QEMU also provide virtual CPU types, compatible with both Intel and AMD host
+CPUs.
+
+NOTE: To mitigate the Spectre vulnerability for virtual CPU types, you need to
+add the relevant CPU flags, see
+xref:qm_meltdown_spectre[Meltdown / Spectre related CPU flags].
+
+Historically, {pve} had the 'kvm64' CPU model, with CPU flags at the level of
+Pentium 4 enabled, so performance was not great for certain workloads.
+
+In the summer of 2020, AMD, Intel, Red Hat, and SUSE collaborated to define
+three x86-64 microarchitecture levels on top of the x86-64 baseline, with modern
+flags enabled. For details, see the
+https://gitlab.com/x86-psABIs/x86-64-ABI[x86-64-ABI specification].
+
+NOTE: Some newer distributions like CentOS 9 are now built with 'x86-64-v2'
+flags as a minimum requirement.
+
+* 'kvm64 (x86-64-v1)': Compatible with Intel CPU >= Pentium 4, AMD CPU >=
+Phenom.
++
+* 'x86-64-v2': Compatible with Intel CPU >= Nehalem, AMD CPU >= Opteron_G3.
+Added CPU flags compared to 'x86-64-v1': '+cx16', '+lahf-lm', '+popcnt', '+pni',
+'+sse4.1', '+sse4.2', '+ssse3'.
++
+* 'x86-64-v2-AES': Compatible with Intel CPU >= Westmere, AMD CPU >= Opteron_G4.
+Added CPU flags compared to 'x86-64-v2': '+aes'.
++
+* 'x86-64-v3': Compatible with Intel CPU >= Broadwell, AMD CPU >= EPYC. Added
+CPU flags compared to 'x86-64-v2-AES': '+avx', '+avx2', '+bmi1', '+bmi2',
+'+f16c', '+fma', '+movbe', '+xsave'.
++
+* 'x86-64-v4': Compatible with Intel CPU >= Skylake, AMD CPU >= EPYC v4 Genoa.
+Added CPU flags compared to 'x86-64-v3': '+avx512f', '+avx512bw', '+avx512cd',
+'+avx512dq', '+avx512vl'.
Custom CPU Types
^^^^^^^^^^^^^^^^
privilege on `/nodes`. When configuring a custom CPU type for a VM via the CLI
or API, the name needs to be prefixed with 'custom-'.
+[[qm_meltdown_spectre]]
Meltdown / Spectre related CPU flags
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
attacks and is able to utilize the CPU feature
Otherwise you need to set the desired CPU flag of the virtual CPU, either by
-editing the CPU options in the WebUI, or by setting the 'flags' property of the
+editing the CPU options in the web UI, or by setting the 'flags' property of the
'cpu' option in the VM configuration file.
For Spectre v1,v2,v4 fixes, your CPU or system vendor also needs to provide a
-so-called ``microcode update'' footnote:[You can use `intel-microcode' /
-`amd-microcode' from Debian non-free if your vendor does not provide such an
-update. Note that not all affected CPUs can be updated to support spec-ctrl.]
-for your CPU.
+so-called ``microcode update'' for your CPU, see
+xref:chapter_firmware_updates[chapter Firmware Updates]. Note that not all
+affected CPUs can be updated to support spec-ctrl.
To check if the {pve} host is vulnerable, execute the following command as root:
for f in /sys/devices/system/cpu/vulnerabilities/*; do echo "${f##*/} -" $(cat "$f"); done
----
-A community script is also available to detect is the host is still vulnerable.
+A community script is also available to detect if the host is still vulnerable.
footnote:[spectre-meltdown-checker https://meltdown.ovh/]
Intel processors
In {pve} the maximal number of plugged CPUs is always `cores * sockets`.
To start a VM with less than this total core count of CPUs you may use the
-*vpus* setting, it denotes how many vCPUs should be plugged in at VM start.
+*vcpus* setting, it denotes how many vCPUs should be plugged in at VM start.
Currently only this feature is only supported on Linux, a kernel newer than 3.10
is needed, a kernel newer than 4.7 is recommended.
database VM when spare RAM is available. For this you assign a Shares property
of 3000 to the database VM, leaving the other VMs to the Shares default setting
of 1000. The host server has 32GB of RAM, and is currently using 16GB, leaving 32
-* 80/100 - 16 = 9GB RAM to be allocated to the VMs. The database VM will get 9 *
-3000 / (3000 + 1000 + 1000 + 1000) = 4.5 GB extra RAM and each HTTP server will
-get 1.5 GB.
+* 80/100 - 16 = 9GB RAM to be allocated to the VMs on top of their configured
+minimum memory amount. The database VM will benefit from 9 * 3000 / (3000 +
+1000 + 1000 + 1000) = 4.5 GB extra RAM and each HTTP server from 1.5 GB.
All Linux distributions released after 2010 have the balloon kernel driver
included. For Windows OSes, the balloon driver needs to be added manually and can
provide network access. This built-in DHCP will serve addresses in the private
10.0.2.0/24 range. The NAT mode is much slower than the bridged mode, and
should only be used for testing. This mode is only available via CLI or the API,
-but not via the WebUI.
+but not via the web UI.
You can also skip adding a network device when creating a VM by selecting *No
network device*.
network queues to the host kernel for each NIC.
//https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Tuning_and_Optimization_Guide/sect-Virtualization_Tuning_Optimization_Guide-Networking-Techniques.html#sect-Virtualization_Tuning_Optimization_Guide-Networking-Multi-queue_virtio-net
-When using Multiqueue, it is recommended to set it to a value equal
-to the number of Total Cores of your guest. You also need to set in
-the VM the number of multi-purpose channels on each VirtIO NIC with the ethtool
-command:
+When using Multiqueue, it is recommended to set it to a value equal to the
+number of vCPUs of your guest. Remember that the number of vCPUs is the number
+of sockets times the number of cores configured for the VM. You also need to set
+the number of multi-purpose channels on each VirtIO NIC in the VM with this
+ethtool command:
`ethtool -L ens1 combined X`
-where X is the number of the number of vcpus of the VM.
+where X is the number of the number of vCPUs of the VM.
+
+To configure a Windows guest for Multiqueue install the
+https://pve.proxmox.com/wiki/Windows_VirtIO_Drivers[Redhat VirtIO Ethernet
+Adapter drivers], then adapt the NIC's configuration as follows. Open the
+device manager, right click the NIC under "Network adapters", and select
+"Properties". Then open the "Advanced" tab and select "Receive Side Scaling"
+from the list on the left. Make sure it is set to "Enabled". Next, navigate to
+"Maximum number of RSS Queues" in the list and set it to the number of vCPUs of
+your VM. Once you verified that the settings are correct, click "OK" to confirm
+them.
You should note that setting the Multiqueue parameter to a value greater
than one will increase the CPU load on the host and guest systems as the
the Web Console to the selected serial port. A configured display 'memory'
setting will be ignored in that case.
+.VNC clipboard
+You can enable the VNC clipboard by setting `clipboard` to `vnc`.
+
+----
+# qm set <vmid> -vga <displaytype>,clipboard=vnc
+----
+
+In order to use the clipboard feature, you must first install the
+SPICE guest tools. On Debian-based distributions, this can be achieved
+by installing `spice-vdagent`. For other Operating Systems search for it
+in the offical repositories or see: https://www.spice-space.org/download.html
+
+Once you have installed the spice guest tools, you can use the VNC clipboard
+function (e.g. in the noVNC console panel). However, if you're using
+SPICE, virtio or virgl, you'll need to choose which clipboard to use.
+This is because the default *SPICE* clipboard will be replaced by the
+*VNC* clipboard, if `clipboard` is set to `vnc`.
+
[[qm_usb_passthrough]]
USB Passthrough
~~~~~~~~~~~~~~~
a VM online to another host, since the hardware is only available
on the host the VM is currently residing.
-The second type of passthrough is SPICE USB passthrough. This is useful
-if you use a SPICE client which supports it. If you add a SPICE USB port
-to your VM, you can passthrough a USB device from where your SPICE client is,
-directly to the VM (for example an input device or hardware dongle).
+The second type of passthrough is SPICE USB passthrough. If you add one or more
+SPICE USB ports to your VM, you can dynamically pass a local USB device from
+your SPICE client through to the VM. This can be useful to redirect an input
+device or hardware dongle temporarily.
+It is also possible to map devices on a cluster level, so that they can be
+properly used with HA and hardware changes are detected and non root users
+can configure them. See xref:resource_mapping[Resource Mapping]
+for details on that.
[[qm_bios_and_uefi]]
BIOS and UEFI
parameters:
* *Start/Shutdown order*: Defines the start order priority. For example, set it
-* to 1 if
-you want the VM to be the first to be started. (We use the reverse startup
-order for shutdown, so a machine with a start order of 1 would be the last to
-be shut down). If multiple VMs have the same order defined on a host, they will
-additionally be ordered by 'VMID' in ascending order.
+to 1 if you want the VM to be the first to be started. (We use the reverse
+startup order for shutdown, so a machine with a start order of 1 would be the
+last to be shut down). If multiple VMs have the same order defined on a host,
+they will additionally be ordered by 'VMID' in ascending order.
* *Startup delay*: Defines the interval between this VM start and subsequent
VMs starts. For example, set it to 240 if you want to wait 240 seconds before
starting other VMs.
https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso[Fedora
VirtIO driver ISO].
+[[qm_qga_enable]]
Enable Guest Agent Communication
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
*Options* panel. A fresh start of the VM is necessary for the changes to take
effect.
+[[qm_qga_auto_trim]]
+Automatic TRIM Using QGA
+^^^^^^^^^^^^^^^^^^^^^^^^
+
It is possible to enable the 'Run guest-trim' option. With this enabled,
{pve} will issue a trim command to the guest after the following
operations that have the potential to write out zeros to the storage:
run as expected. Subsequent ones, until the next reboot, will only consider
parts of the filesystem that changed since then.
+[[qm_qga_fsfreeze]]
+Filesystem Freeze & Thaw on Backup
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+By default, guest filesystems are synced via the 'fs-freeze' QEMU Guest Agent
+Command when a backup is performed, to provide consistency.
+
+On Windows guests, some applications might handle consistent backups themselves
+by hooking into the Windows VSS (Volume Shadow Copy Service) layer, a
+'fs-freeze' then might interfere with that. For example, it has been observed
+that calling 'fs-freeze' with some SQL Servers triggers VSS to call the SQL
+Writer VSS module in a mode that breaks the SQL Server backup chain for
+differential backups.
+
+For such setups you can configure {pve} to not issue a freeze-and-thaw cycle on
+backup by setting the `freeze-fs-on-backup` QGA option to `0`. This can also be
+done via the GUI with the 'Freeze/thaw guest filesystems on backup for
+consistency' option.
+
+IMPORTANT: Disabling this option can potentially lead to backups with inconsistent
+filesystems and should therefore only be disabled if you know what you are
+doing.
+
Troubleshooting
^^^^^^^^^^^^^^^
~~~~~~~~~~~~~~~~
If your VM is running and no locally bound resources are configured (such as
-passed-through devices), you can initiate a live migration with the `--online`
-flag in the `qm migration` command evocation. The web-interface defaults to
+devices that are passed through), you can initiate a live migration with the `--online`
+flag in the `qm migration` command evocation. The web interface defaults to
live migration when the VM is running.
How it works
If you have local resources, you can still migrate your VMs offline as long as
all disk are on storage defined on both hosts.
Migration then copies the disks to the target host over the network, as with
-online migration. Note that any hardware pass-through configuration may need to
+online migration. Note that any hardware passthrough configuration may need to
be adapted to the device location on the target host.
// TODO: mention hardware map IDs as better way to solve that, once available
footnote:[https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/get-started/virtual-dc/virtualized-domain-controller-architecture])
on snapshot rollback, backup restore or a whole VM clone operation.
-Importing Virtual Machines and disk images
-------------------------------------------
+[[qm_import_virtual_machines]]
+Importing Virtual Machines
+--------------------------
+
+Importing existing virtual machines from foreign hypervisors or other {pve}
+clusters can be achieved through various methods, the most common ones are:
+
+* Using the native import wizard, which utilizes the 'import' content type, such
+ as provided by the ESXi special storage.
+* Performing a backup on the source and then restoring on the target. This
+ method works best when migrating from another {pve} instance.
+* using the OVF-specific import command of the `qm` command-line tool.
+
+If you import VMs to {pve} from other hypervisors, it’s recommended to
+familiarize yourself with the
+https://pve.proxmox.com/wiki/Migrate_to_Proxmox_VE#Concepts[concepts of {pve}].
+
+Import Wizard
+~~~~~~~~~~~~~
+
+[thumbnail="screenshot/gui-import-wizard-general.png"]
+
+{pve} provides an integrated VM importer using the storage plugin system for
+native integration into the API and web-based user interface. You can use this
+to import the VM as a whole, with most of its config mapped to {pve}'s config
+model and reduced downtime.
+
+NOTE: The import wizard was added during the {pve} 8.2 development cycle and is
+in tech preview state. While it's already promising and working stable, it's
+still under active development, focusing on adding other import-sources, like
+for example OVF/OVA files, in the future.
+
+To use the import wizard you have to first set up a new storage for an import
+source, you can do so on the web-interface under _Datacenter -> Storage -> Add_.
+
+Then you can select the new storage in the resource tree and use the 'Virtual
+Guests' content tab to see all available guests that can be imported.
+
+[thumbnail="screenshot/gui-import-wizard-advanced.png"]
+
+Select one and use the 'Import' button (or double-click) to open the import
+wizard. You can modify a subset of the available options here and then start the
+import. Please note that you can do more advanced modifications after the import
+finished.
+
+TIP: The import wizard is currently (2024-03) available for ESXi and has been
+tested with ESXi versions 6.5 through 8.0. Note that guests using vSAN storage
+cannot be directly imported directly; their disks must first be moved to another
+storage. While it is possible to use a vCenter as the import source, performance
+is dramatically degraded (5 to 10 times slower).
+
+For a step-by-step guide and tips for how to adapt the virtual guest to the new
+hyper-visor see our
+https://pve.proxmox.com/wiki/Migrate_to_Proxmox_VE#Migration[migrate to {pve}
+wiki article].
+
+Import OVF/OVA Through CLI
+~~~~~~~~~~~~~~~~~~~~~~~~~~
A VM export from a foreign hypervisor takes usually the form of one or more disk
images, with a configuration file describing the settings of the VM (RAM,
cases due to the problems above.
Step-by-step example of a Windows OVF import
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Microsoft provides
https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/[Virtual Machines downloads]
to demonstrate the OVF import feature.
Download the Virtual Machine zip
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+++++++++++++++++++++++++++++++++
After getting informed about the user agreement, choose the _Windows 10
Enterprise (Evaluation - Build)_ for the VMware platform, and download the zip.
Extract the disk image from the zip
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
++++++++++++++++++++++++++++++++++++
Using the `unzip` utility or any archiver of your choice, unpack the zip,
and copy via ssh/scp the ovf and vmdk files to your {pve} host.
Import the Virtual Machine
-^^^^^^^^^^^^^^^^^^^^^^^^^^
+++++++++++++++++++++++++++
This will create a new virtual machine, using cores, memory and
VM name as read from the OVF manifest, and import the disks to the +local-lvm+
The VM is ready to be started.
Adding an external disk image to a Virtual Machine
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+++++++++++++++++++++++++++++++++++++++++++++++++++
You can also add an existing disk image to a VM, either coming from a
foreign hypervisor, or one that you created yourself.
3. The first non-shared storage from any VM disk.
4. The storage `local` as a fallback.
+[[resource_mapping]]
+Resource Mapping
+----------------
+
+[thumbnail="screenshot/gui-datacenter-resource-mappings.png"]
+
+When using or referencing local resources (e.g. address of a pci device), using
+the raw address or id is sometimes problematic, for example:
+
+* when using HA, a different device with the same id or path may exist on the
+ target node, and if one is not careful when assigning such guests to HA
+ groups, the wrong device could be used, breaking configurations.
+
+* changing hardware can change ids and paths, so one would have to check all
+ assigned devices and see if the path or id is still correct.
+
+To handle this better, one can define cluster wide resource mappings, such that
+a resource has a cluster unique, user selected identifier which can correspond
+to different devices on different hosts. With this, HA won't start a guest with
+a wrong device, and hardware changes can be detected.
+
+Creating such a mapping can be done with the {pve} web GUI under `Datacenter`
+in the relevant tab in the `Resource Mappings` category, or on the cli with
+
+----
+# pvesh create /cluster/mapping/<type> <options>
+----
+
+[thumbnail="screenshot/gui-datacenter-mapping-pci-edit.png"]
+
+Where `<type>` is the hardware type (currently either `pci` or `usb`) and
+`<options>` are the device mappings and other configuration parameters.
+
+Note that the options must include a map property with all identifying
+properties of that hardware, so that it's possible to verify the hardware did
+not change and the correct device is passed through.
+
+For example to add a PCI device as `device1` with the path `0000:01:00.0` that
+has the device id `0001` and the vendor id `0002` on the node `node1`, and
+`0000:02:00.0` on `node2` you can add it with:
+
+----
+# pvesh create /cluster/mapping/pci --id device1 \
+ --map node=node1,path=0000:01:00.0,id=0002:0001 \
+ --map node=node2,path=0000:02:00.0,id=0002:0001
+----
+
+You must repeat the `map` parameter for each node where that device should have
+a mapping (note that you can currently only map one USB device per node per
+mapping).
+
+Using the GUI makes this much easier, as the correct properties are
+automatically picked up and sent to the API.
+
+[thumbnail="screenshot/gui-datacenter-mapping-usb-edit.png"]
+
+It's also possible for PCI devices to provide multiple devices per node with
+multiple map properties for the nodes. If such a device is assigned to a guest,
+the first free one will be used when the guest is started. The order of the
+paths given is also the order in which they are tried, so arbitrary allocation
+policies can be implemented.
+
+This is useful for devices with SR-IOV, since some times it is not important
+which exact virtual function is passed through.
+
+You can assign such a device to a guest either with the GUI or with
+
+----
+# qm set ID -hostpci0 <name>
+----
+
+for PCI devices, or
+
+----
+# qm set <vmid> -usb0 <name>
+----
+
+for USB devices.
+
+Where `<vmid>` is the guests id and `<name>` is the chosen name for the created
+mapping. All usual options for passing through the devices are allowed, such as
+`mdev`.
+
+To create mappings `Mapping.Modify` on `/mapping/<type>/<name>` is necessary
+(where `<type>` is the device type and `<name>` is the name of the mapping).
+
+To use these mappings, `Mapping.Use` on `/mapping/<type>/<name>` is necessary
+(in addition to the normal guest privileges to edit the configuration).
+
Managing Virtual Machines with `qm`
------------------------------------
# qm shutdown 300 && qm wait 300 -timeout 40
----
+If the VM does not shut down, force-stop it and overrule any running shutdown
+tasks. As stopping VMs may incur data loss, use it with caution.
+
+----
+# qm stop 300 -overrule-shutdown 1
+----
+
Destroying a VM always removes it from Access Control Lists and it always
removes the firewall configuration of the VM. You have to activate
'--purge', if you want to additionally remove the VM from replication jobs,
CAUTION: Only do that if you are sure the action which set the lock is
no longer running.
-
ifdef::wiki[]
See Also
projects / pve-docs.git / blobdiff