Emulated devices and paravirtualized devices
--------------------------------------------
-The PC hardware emulated by QEMU includes a mainboard, network controllers,
+The PC hardware emulated by QEMU includes a motherboard, network controllers,
SCSI, IDE and SATA controllers, serial ports (the complete list can be seen in
the `kvm(1)` man page) all of them emulated in software. All these devices
are the exact software equivalent of existing hardware devices, and if the OS
https://ark.intel.com/content/www/us/en/ark/products/31918/intel-82q35-graphics-and-memory-controller.html[Q35]
chipset, which also provides a virtual PCIe bus, and thus may be
desired if you want to pass through PCIe hardware.
+Additionally, you can select a xref:qm_pci_viommu[vIOMMU] implementation.
Machine Version
+++++++++++++++
Resource Limits
^^^^^^^^^^^^^^^
-In addition to the number of virtual cores, you can configure how much resources
-a VM can get in relation to the host CPU time and also in relation to other
-VMs.
-With the *cpulimit* (``Host CPU Time'') option you can limit how much CPU time
-the whole VM can use on the host. It is a floating point value representing CPU
-time in percent, so `1.0` is equal to `100%`, `2.5` to `250%` and so on. If a
-single process would fully use one single core it would have `100%` CPU Time
-usage. If a VM with four cores utilizes all its cores fully it would
-theoretically use `400%`. In reality the usage may be even a bit higher as QEMU
-can have additional threads for VM peripherals besides the vCPU core ones.
-This setting can be useful if a VM should have multiple vCPUs, as it runs a few
-processes in parallel, but the VM as a whole should not be able to run all
-vCPUs at 100% at the same time. Using a specific example: lets say we have a VM
-which would profit from having 8 vCPUs, but at no time all of those 8 cores
-should run at full load - as this would make the server so overloaded that
-other VMs and CTs would get to less CPU. So, we set the *cpulimit* limit to
-`4.0` (=400%). If all cores do the same heavy work they would all get 50% of a
-real host cores CPU time. But, if only 4 would do work they could still get
-almost 100% of a real core each.
+*cpulimit*
+
+In addition to the number of virtual cores, the total available ``Host CPU
+Time'' for the VM can be set with the *cpulimit* option. It is a floating point
+value representing CPU time in percent, so `1.0` is equal to `100%`, `2.5` to
+`250%` and so on. If a single process would fully use one single core it would
+have `100%` CPU Time usage. If a VM with four cores utilizes all its cores
+fully it would theoretically use `400%`. In reality the usage may be even a bit
+higher as QEMU can have additional threads for VM peripherals besides the vCPU
+core ones.
+
+This setting can be useful when a VM should have multiple vCPUs because it is
+running some processes in parallel, but the VM as a whole should not be able to
+run all vCPUs at 100% at the same time.
+
+For example, suppose you have a virtual machine that would benefit from having 8
+virtual CPUs, but you don't want the VM to be able to max out all 8 cores
+running at full load - because that would overload the server and leave other
+virtual machines and containers with too little CPU time. To solve this, you
+could set *cpulimit* to `4.0` (=400%). This means that if the VM fully utilizes
+all 8 virtual CPUs by running 8 processes simultaneously, each vCPU will receive
+a maximum of 50% CPU time from the physical cores. However, if the VM workload
+only fully utilizes 4 virtual CPUs, it could still receive up to 100% CPU time
+from a physical core, for a total of 400%.
NOTE: VMs can, depending on their configuration, use additional threads, such
as for networking or IO operations but also live migration. Thus a VM can show
up to use more CPU time than just its virtual CPUs could use. To ensure that a
-VM never uses more CPU time than virtual CPUs assigned set the *cpulimit*
-setting to the same value as the total core count.
+VM never uses more CPU time than vCPUs assigned, set the *cpulimit* to
+the same value as the total core count.
-The second CPU resource limiting setting, *cpuunits* (nowadays often called CPU
-shares or CPU weight), controls how much CPU time a VM gets compared to other
-running VMs. It is a relative weight which defaults to `100` (or `1024` if the
-host uses legacy cgroup v1). If you increase this for a VM it will be
-prioritized by the scheduler in comparison to other VMs with lower weight. For
-example, if VM 100 has set the default `100` and VM 200 was changed to `200`,
-the latter VM 200 would receive twice the CPU bandwidth than the first VM 100.
+*cpuuntis*
+
+With the *cpuunits* option, nowadays often called CPU shares or CPU weight, you
+can control how much CPU time a VM gets compared to other running VMs. It is a
+relative weight which defaults to `100` (or `1024` if the host uses legacy
+cgroup v1). If you increase this for a VM it will be prioritized by the
+scheduler in comparison to other VMs with lower weight.
+
+For example, if VM 100 has set the default `100` and VM 200 was changed to
+`200`, the latter VM 200 would receive twice the CPU bandwidth than the first
+VM 100.
For more information see `man systemd.resource-control`, here `CPUQuota`
-corresponds to `cpulimit` and `CPUWeight` corresponds to our `cpuunits`
-setting, visit its Notes section for references and implementation details.
+corresponds to `cpulimit` and `CPUWeight` to our `cpuunits` setting. Visit its
+Notes section for references and implementation details.
-The third CPU resource limiting setting, *affinity*, controls what host cores
-the virtual machine will be permitted to execute on. E.g., if an affinity value
-of `0-3,8-11` is provided, the virtual machine will be restricted to using the
-host cores `0,1,2,3,8,9,10,` and `11`. Valid *affinity* values are written in
-cpuset `List Format`. List Format is a comma-separated list of CPU numbers and
-ranges of numbers, in ASCII decimal.
+*affinity*
-NOTE: CPU *affinity* uses the `taskset` command to restrict virtual machines to
-a given set of cores. This restriction will not take effect for some types of
-processes that may be created for IO. *CPU affinity is not a security feature.*
+With the *affinity* option, you can specify the physical CPU cores that are used
+to run the VM's vCPUs. Peripheral VM processes, such as those for I/O, are not
+affected by this setting. Note that the *CPU affinity is not a security
+feature*.
-For more information regarding *affinity* see `man cpuset`. Here the
-`List Format` corresponds to valid *affinity* values. Visit its `Formats`
-section for more examples.
+Forcing a CPU *affinity* can make sense in certain cases but is accompanied by
+an increase in complexity and maintenance effort. For example, if you want to
+add more VMs later or migrate VMs to nodes with fewer CPU cores. It can also
+easily lead to asynchronous and therefore limited system performance if some
+CPUs are fully utilized while others are almost idle.
+
+The *affinity* is set through the `taskset` CLI tool. It accepts the host CPU
+numbers (see `lscpu`) in the `List Format` from `man cpuset`. This ASCII decimal
+list can contain numbers but also number ranges. For example, the *affinity*
+`0-1,8-11` (expanded `0, 1, 8, 9, 10, 11`) would allow the VM to run on only
+these six specific host cores.
CPU Type
^^^^^^^^
attacks and is able to utilize the CPU feature
Otherwise you need to set the desired CPU flag of the virtual CPU, either by
-editing the CPU options in the WebUI, or by setting the 'flags' property of the
+editing the CPU options in the web UI, or by setting the 'flags' property of the
'cpu' option in the VM configuration file.
For Spectre v1,v2,v4 fixes, your CPU or system vendor also needs to provide a
In {pve} the maximal number of plugged CPUs is always `cores * sockets`.
To start a VM with less than this total core count of CPUs you may use the
-*vpus* setting, it denotes how many vCPUs should be plugged in at VM start.
+*vcpus* setting, it denotes how many vCPUs should be plugged in at VM start.
Currently only this feature is only supported on Linux, a kernel newer than 3.10
is needed, a kernel newer than 4.7 is recommended.
database VM when spare RAM is available. For this you assign a Shares property
of 3000 to the database VM, leaving the other VMs to the Shares default setting
of 1000. The host server has 32GB of RAM, and is currently using 16GB, leaving 32
-* 80/100 - 16 = 9GB RAM to be allocated to the VMs. The database VM will get 9 *
-3000 / (3000 + 1000 + 1000 + 1000) = 4.5 GB extra RAM and each HTTP server will
-get 1.5 GB.
+* 80/100 - 16 = 9GB RAM to be allocated to the VMs on top of their configured
+minimum memory amount. The database VM will benefit from 9 * 3000 / (3000 +
+1000 + 1000 + 1000) = 4.5 GB extra RAM and each HTTP server from 1.5 GB.
All Linux distributions released after 2010 have the balloon kernel driver
included. For Windows OSes, the balloon driver needs to be added manually and can
provide network access. This built-in DHCP will serve addresses in the private
10.0.2.0/24 range. The NAT mode is much slower than the bridged mode, and
should only be used for testing. This mode is only available via CLI or the API,
-but not via the WebUI.
+but not via the web UI.
You can also skip adding a network device when creating a VM by selecting *No
network device*.
network queues to the host kernel for each NIC.
//https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Tuning_and_Optimization_Guide/sect-Virtualization_Tuning_Optimization_Guide-Networking-Techniques.html#sect-Virtualization_Tuning_Optimization_Guide-Networking-Multi-queue_virtio-net
-When using Multiqueue, it is recommended to set it to a value equal
-to the number of Total Cores of your guest. You also need to set in
-the VM the number of multi-purpose channels on each VirtIO NIC with the ethtool
-command:
+When using Multiqueue, it is recommended to set it to a value equal to the
+number of vCPUs of your guest. Remember that the number of vCPUs is the number
+of sockets times the number of cores configured for the VM. You also need to set
+the number of multi-purpose channels on each VirtIO NIC in the VM with this
+ethtool command:
`ethtool -L ens1 combined X`
-where X is the number of the number of vcpus of the VM.
+where X is the number of the number of vCPUs of the VM.
+
+To configure a Windows guest for Multiqueue install the
+https://pve.proxmox.com/wiki/Windows_VirtIO_Drivers[Redhat VirtIO Ethernet
+Adapter drivers], then adapt the NIC's configuration as follows. Open the
+device manager, right click the NIC under "Network adapters", and select
+"Properties". Then open the "Advanced" tab and select "Receive Side Scaling"
+from the list on the left. Make sure it is set to "Enabled". Next, navigate to
+"Maximum number of RSS Queues" in the list and set it to the number of vCPUs of
+your VM. Once you verified that the settings are correct, click "OK" to confirm
+them.
You should note that setting the Multiqueue parameter to a value greater
than one will increase the CPU load on the host and guest systems as the
the Web Console to the selected serial port. A configured display 'memory'
setting will be ignored in that case.
+.VNC clipboard
+You can enable the VNC clipboard by setting `clipboard` to `vnc`.
+
+----
+# qm set <vmid> -vga <displaytype>,clipboard=vnc
+----
+
+In order to use the clipboard feature, you must first install the
+SPICE guest tools. On Debian-based distributions, this can be achieved
+by installing `spice-vdagent`. For other Operating Systems search for it
+in the offical repositories or see: https://www.spice-space.org/download.html
+
+Once you have installed the spice guest tools, you can use the VNC clipboard
+function (e.g. in the noVNC console panel). However, if you're using
+SPICE, virtio or virgl, you'll need to choose which clipboard to use.
+This is because the default *SPICE* clipboard will be replaced by the
+*VNC* clipboard, if `clipboard` is set to `vnc`.
+
[[qm_usb_passthrough]]
USB Passthrough
~~~~~~~~~~~~~~~
a VM online to another host, since the hardware is only available
on the host the VM is currently residing.
-The second type of passthrough is SPICE USB passthrough. This is useful
-if you use a SPICE client which supports it. If you add a SPICE USB port
-to your VM, you can passthrough a USB device from where your SPICE client is,
-directly to the VM (for example an input device or hardware dongle).
+The second type of passthrough is SPICE USB passthrough. If you add one or more
+SPICE USB ports to your VM, you can dynamically pass a local USB device from
+your SPICE client through to the VM. This can be useful to redirect an input
+device or hardware dongle temporarily.
It is also possible to map devices on a cluster level, so that they can be
properly used with HA and hardware changes are detected and non root users
parameters:
* *Start/Shutdown order*: Defines the start order priority. For example, set it
-* to 1 if
-you want the VM to be the first to be started. (We use the reverse startup
-order for shutdown, so a machine with a start order of 1 would be the last to
-be shut down). If multiple VMs have the same order defined on a host, they will
-additionally be ordered by 'VMID' in ascending order.
+to 1 if you want the VM to be the first to be started. (We use the reverse
+startup order for shutdown, so a machine with a start order of 1 would be the
+last to be shut down). If multiple VMs have the same order defined on a host,
+they will additionally be ordered by 'VMID' in ascending order.
* *Startup delay*: Defines the interval between this VM start and subsequent
VMs starts. For example, set it to 240 if you want to wait 240 seconds before
starting other VMs.
~~~~~~~~~~~~~~~~
If your VM is running and no locally bound resources are configured (such as
-passed-through devices), you can initiate a live migration with the `--online`
-flag in the `qm migration` command evocation. The web-interface defaults to
+devices that are passed through), you can initiate a live migration with the `--online`
+flag in the `qm migration` command evocation. The web interface defaults to
live migration when the VM is running.
How it works
If you have local resources, you can still migrate your VMs offline as long as
all disk are on storage defined on both hosts.
Migration then copies the disks to the target host over the network, as with
-online migration. Note that any hardware pass-through configuration may need to
+online migration. Note that any hardware passthrough configuration may need to
be adapted to the device location on the target host.
// TODO: mention hardware map IDs as better way to solve that, once available
footnote:[https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/get-started/virtual-dc/virtualized-domain-controller-architecture])
on snapshot rollback, backup restore or a whole VM clone operation.
-Importing Virtual Machines and disk images
-------------------------------------------
+[[qm_import_virtual_machines]]
+Importing Virtual Machines
+--------------------------
+
+Importing existing virtual machines from foreign hypervisors or other {pve}
+clusters can be achieved through various methods, the most common ones are:
+
+* Using the native import wizard, which utilizes the 'import' content type, such
+ as provided by the ESXi special storage.
+* Performing a backup on the source and then restoring on the target. This
+ method works best when migrating from another {pve} instance.
+* using the OVF-specific import command of the `qm` command-line tool.
+
+If you import VMs to {pve} from other hypervisors, it’s recommended to
+familiarize yourself with the
+https://pve.proxmox.com/wiki/Migrate_to_Proxmox_VE#Concepts[concepts of {pve}].
+
+Import Wizard
+~~~~~~~~~~~~~
+
+[thumbnail="screenshot/gui-import-wizard-general.png"]
+
+{pve} provides an integrated VM importer using the storage plugin system for
+native integration into the API and web-based user interface. You can use this
+to import the VM as a whole, with most of its config mapped to {pve}'s config
+model and reduced downtime.
+
+NOTE: The import wizard was added during the {pve} 8.2 development cycle and is
+in tech preview state. While it's already promising and working stable, it's
+still under active development, focusing on adding other import-sources, like
+for example OVF/OVA files, in the future.
+
+To use the import wizard you have to first set up a new storage for an import
+source, you can do so on the web-interface under _Datacenter -> Storage -> Add_.
+
+Then you can select the new storage in the resource tree and use the 'Virtual
+Guests' content tab to see all available guests that can be imported.
+
+[thumbnail="screenshot/gui-import-wizard-advanced.png"]
+
+Select one and use the 'Import' button (or double-click) to open the import
+wizard. You can modify a subset of the available options here and then start the
+import. Please note that you can do more advanced modifications after the import
+finished.
+
+TIP: The import wizard is currently (2024-03) available for ESXi and has been
+tested with ESXi versions 6.5 through 8.0. Note that guests using vSAN storage
+cannot be directly imported directly; their disks must first be moved to another
+storage. While it is possible to use a vCenter as the import source, performance
+is dramatically degraded (5 to 10 times slower).
+
+For a step-by-step guide and tips for how to adapt the virtual guest to the new
+hyper-visor see our
+https://pve.proxmox.com/wiki/Migrate_to_Proxmox_VE#Migration[migrate to {pve}
+wiki article].
+
+Import OVF/OVA Through CLI
+~~~~~~~~~~~~~~~~~~~~~~~~~~
A VM export from a foreign hypervisor takes usually the form of one or more disk
images, with a configuration file describing the settings of the VM (RAM,
cases due to the problems above.
Step-by-step example of a Windows OVF import
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Microsoft provides
https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/[Virtual Machines downloads]
to demonstrate the OVF import feature.
Download the Virtual Machine zip
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+++++++++++++++++++++++++++++++++
After getting informed about the user agreement, choose the _Windows 10
Enterprise (Evaluation - Build)_ for the VMware platform, and download the zip.
Extract the disk image from the zip
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
++++++++++++++++++++++++++++++++++++
Using the `unzip` utility or any archiver of your choice, unpack the zip,
and copy via ssh/scp the ovf and vmdk files to your {pve} host.
Import the Virtual Machine
-^^^^^^^^^^^^^^^^^^^^^^^^^^
+++++++++++++++++++++++++++
This will create a new virtual machine, using cores, memory and
VM name as read from the OVF manifest, and import the disks to the +local-lvm+
The VM is ready to be started.
Adding an external disk image to a Virtual Machine
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+++++++++++++++++++++++++++++++++++++++++++++++++++
You can also add an existing disk image to a VM, either coming from a
foreign hypervisor, or one that you created yourself.
# qm shutdown 300 && qm wait 300 -timeout 40
----
+If the VM does not shut down, force-stop it and overrule any running shutdown
+tasks. As stopping VMs may incur data loss, use it with caution.
+
+----
+# qm stop 300 -overrule-shutdown 1
+----
+
Destroying a VM always removes it from Access Control Lists and it always
removes the firewall configuration of the VM. You have to activate
'--purge', if you want to additionally remove the VM from replication jobs,
projects / pve-docs.git / blobdiff