update link qemu documentation non web.archive
[pve-docs.git] / qm.adoc
1 [[chapter_virtual_machines]]
2 ifdef::manvolnum[]
3 qm(1)
4 =====
5 :pve-toplevel:
6
7 NAME
8 ----
9
10 qm - Qemu/KVM Virtual Machine Manager
11
12
13 SYNOPSIS
14 --------
15
16 include::qm.1-synopsis.adoc[]
17
18 DESCRIPTION
19 -----------
20 endif::manvolnum[]
21 ifndef::manvolnum[]
22 Qemu/KVM Virtual Machines
23 =========================
24 :pve-toplevel:
25 endif::manvolnum[]
26
27 // deprecates
28 // http://pve.proxmox.com/wiki/Container_and_Full_Virtualization
29 // http://pve.proxmox.com/wiki/KVM
30 // http://pve.proxmox.com/wiki/Qemu_Server
31
32 Qemu (short form for Quick Emulator) is an open source hypervisor that emulates a
33 physical computer. From the perspective of the host system where Qemu is
34 running, Qemu is a user program which has access to a number of local resources
35 like partitions, files, network cards which are then passed to an
36 emulated computer which sees them as if they were real devices.
37
38 A guest operating system running in the emulated computer accesses these
39 devices, and runs as it were running on real hardware. For instance you can pass
40 an iso image as a parameter to Qemu, and the OS running in the emulated computer
41 will see a real CDROM inserted in a CD drive.
42
43 Qemu can emulate a great variety of hardware from ARM to Sparc, but {pve} is
44 only concerned with 32 and 64 bits PC clone emulation, since it represents the
45 overwhelming majority of server hardware. The emulation of PC clones is also one
46 of the fastest due to the availability of processor extensions which greatly
47 speed up Qemu when the emulated architecture is the same as the host
48 architecture.
49
50 NOTE: You may sometimes encounter the term _KVM_ (Kernel-based Virtual Machine).
51 It means that Qemu is running with the support of the virtualization processor
52 extensions, via the Linux kvm module. In the context of {pve} _Qemu_ and
53 _KVM_ can be used interchangeably as Qemu in {pve} will always try to load the kvm
54 module.
55
56 Qemu inside {pve} runs as a root process, since this is required to access block
57 and PCI devices.
58
59
60 Emulated devices and paravirtualized devices
61 --------------------------------------------
62
63 The PC hardware emulated by Qemu includes a mainboard, network controllers,
64 scsi, ide and sata controllers, serial ports (the complete list can be seen in
65 the `kvm(1)` man page) all of them emulated in software. All these devices
66 are the exact software equivalent of existing hardware devices, and if the OS
67 running in the guest has the proper drivers it will use the devices as if it
68 were running on real hardware. This allows Qemu to runs _unmodified_ operating
69 systems.
70
71 This however has a performance cost, as running in software what was meant to
72 run in hardware involves a lot of extra work for the host CPU. To mitigate this,
73 Qemu can present to the guest operating system _paravirtualized devices_, where
74 the guest OS recognizes it is running inside Qemu and cooperates with the
75 hypervisor.
76
77 Qemu relies on the virtio virtualization standard, and is thus able to present
78 paravirtualized virtio devices, which includes a paravirtualized generic disk
79 controller, a paravirtualized network card, a paravirtualized serial port,
80 a paravirtualized SCSI controller, etc ...
81
82 It is highly recommended to use the virtio devices whenever you can, as they
83 provide a big performance improvement. Using the virtio generic disk controller
84 versus an emulated IDE controller will double the sequential write throughput,
85 as measured with `bonnie++(8)`. Using the virtio network interface can deliver
86 up to three times the throughput of an emulated Intel E1000 network card, as
87 measured with `iperf(1)`. footnote:[See this benchmark on the KVM wiki
88 http://www.linux-kvm.org/page/Using_VirtIO_NIC]
89
90
91 [[qm_virtual_machines_settings]]
92 Virtual Machines Settings
93 -------------------------
94
95 Generally speaking {pve} tries to choose sane defaults for virtual machines
96 (VM). Make sure you understand the meaning of the settings you change, as it
97 could incur a performance slowdown, or putting your data at risk.
98
99
100 [[qm_general_settings]]
101 General Settings
102 ~~~~~~~~~~~~~~~~
103
104 [thumbnail="screenshot/gui-create-vm-general.png"]
105
106 General settings of a VM include
107
108 * the *Node* : the physical server on which the VM will run
109 * the *VM ID*: a unique number in this {pve} installation used to identify your VM
110 * *Name*: a free form text string you can use to describe the VM
111 * *Resource Pool*: a logical group of VMs
112
113
114 [[qm_os_settings]]
115 OS Settings
116 ~~~~~~~~~~~
117
118 [thumbnail="screenshot/gui-create-vm-os.png"]
119
120 When creating a virtual machine (VM), setting the proper Operating System(OS)
121 allows {pve} to optimize some low level parameters. For instance Windows OS
122 expect the BIOS clock to use the local time, while Unix based OS expect the
123 BIOS clock to have the UTC time.
124
125 [[qm_system_settings]]
126 System Settings
127 ~~~~~~~~~~~~~~~
128
129 On VM creation you can change some basic system components of the new VM. You
130 can specify which xref:qm_display[display type] you want to use.
131 [thumbnail="screenshot/gui-create-vm-system.png"]
132 Additionally, the xref:qm_hard_disk[SCSI controller] can be changed.
133 If you plan to install the QEMU Guest Agent, or if your selected ISO image
134 already ships and installs it automatically, you may want to tick the 'Qemu
135 Agent' box, which lets {pve} know that it can use its features to show some
136 more information, and complete some actions (for example, shutdown or
137 snapshots) more intelligently.
138
139 {pve} allows to boot VMs with different firmware and machine types, namely
140 xref:qm_bios_and_uefi[SeaBIOS and OVMF]. In most cases you want to switch from
141 the default SeabBIOS to OVMF only if you plan to use
142 xref:qm_pci_passthrough[PCIe pass through]. A VMs 'Machine Type' defines the
143 hardware layout of the VM's virtual motherboard. You can choose between the
144 default https://en.wikipedia.org/wiki/Intel_440FX[Intel 440FX] or the
145 https://ark.intel.com/content/www/us/en/ark/products/31918/intel-82q35-graphics-and-memory-controller.html[Q35]
146 chipset, which also provides a virtual PCIe bus, and thus may be desired if
147 one wants to pass through PCIe hardware.
148
149 [[qm_hard_disk]]
150 Hard Disk
151 ~~~~~~~~~
152
153 [[qm_hard_disk_bus]]
154 Bus/Controller
155 ^^^^^^^^^^^^^^
156 Qemu can emulate a number of storage controllers:
157
158 * the *IDE* controller, has a design which goes back to the 1984 PC/AT disk
159 controller. Even if this controller has been superseded by recent designs,
160 each and every OS you can think of has support for it, making it a great choice
161 if you want to run an OS released before 2003. You can connect up to 4 devices
162 on this controller.
163
164 * the *SATA* (Serial ATA) controller, dating from 2003, has a more modern
165 design, allowing higher throughput and a greater number of devices to be
166 connected. You can connect up to 6 devices on this controller.
167
168 * the *SCSI* controller, designed in 1985, is commonly found on server grade
169 hardware, and can connect up to 14 storage devices. {pve} emulates by default a
170 LSI 53C895A controller.
171 +
172 A SCSI controller of type _VirtIO SCSI_ is the recommended setting if you aim for
173 performance and is automatically selected for newly created Linux VMs since
174 {pve} 4.3. Linux distributions have support for this controller since 2012, and
175 FreeBSD since 2014. For Windows OSes, you need to provide an extra iso
176 containing the drivers during the installation.
177 // https://pve.proxmox.com/wiki/Paravirtualized_Block_Drivers_for_Windows#During_windows_installation.
178 If you aim at maximum performance, you can select a SCSI controller of type
179 _VirtIO SCSI single_ which will allow you to select the *IO Thread* option.
180 When selecting _VirtIO SCSI single_ Qemu will create a new controller for
181 each disk, instead of adding all disks to the same controller.
182
183 * The *VirtIO Block* controller, often just called VirtIO or virtio-blk,
184 is an older type of paravirtualized controller. It has been superseded by the
185 VirtIO SCSI Controller, in terms of features.
186
187 [thumbnail="screenshot/gui-create-vm-hard-disk.png"]
188
189 [[qm_hard_disk_formats]]
190 Image Format
191 ^^^^^^^^^^^^
192 On each controller you attach a number of emulated hard disks, which are backed
193 by a file or a block device residing in the configured storage. The choice of
194 a storage type will determine the format of the hard disk image. Storages which
195 present block devices (LVM, ZFS, Ceph) will require the *raw disk image format*,
196 whereas files based storages (Ext4, NFS, CIFS, GlusterFS) will let you to choose
197 either the *raw disk image format* or the *QEMU image format*.
198
199 * the *QEMU image format* is a copy on write format which allows snapshots, and
200 thin provisioning of the disk image.
201 * the *raw disk image* is a bit-to-bit image of a hard disk, similar to what
202 you would get when executing the `dd` command on a block device in Linux. This
203 format does not support thin provisioning or snapshots by itself, requiring
204 cooperation from the storage layer for these tasks. It may, however, be up to
205 10% faster than the *QEMU image format*. footnote:[See this benchmark for details
206 https://events.static.linuxfound.org/sites/events/files/slides/CloudOpen2013_Khoa_Huynh_v3.pdf]
207 * the *VMware image format* only makes sense if you intend to import/export the
208 disk image to other hypervisors.
209
210 [[qm_hard_disk_cache]]
211 Cache Mode
212 ^^^^^^^^^^
213 Setting the *Cache* mode of the hard drive will impact how the host system will
214 notify the guest systems of block write completions. The *No cache* default
215 means that the guest system will be notified that a write is complete when each
216 block reaches the physical storage write queue, ignoring the host page cache.
217 This provides a good balance between safety and speed.
218
219 If you want the {pve} backup manager to skip a disk when doing a backup of a VM,
220 you can set the *No backup* option on that disk.
221
222 If you want the {pve} storage replication mechanism to skip a disk when starting
223 a replication job, you can set the *Skip replication* option on that disk.
224 As of {pve} 5.0, replication requires the disk images to be on a storage of type
225 `zfspool`, so adding a disk image to other storages when the VM has replication
226 configured requires to skip replication for this disk image.
227
228 [[qm_hard_disk_discard]]
229 Trim/Discard
230 ^^^^^^^^^^^^
231 If your storage supports _thin provisioning_ (see the storage chapter in the
232 {pve} guide), you can activate the *Discard* option on a drive. With *Discard*
233 set and a _TRIM_-enabled guest OS footnote:[TRIM, UNMAP, and discard
234 https://en.wikipedia.org/wiki/Trim_%28computing%29], when the VM's filesystem
235 marks blocks as unused after deleting files, the controller will relay this
236 information to the storage, which will then shrink the disk image accordingly.
237 For the guest to be able to issue _TRIM_ commands, you must enable the *Discard*
238 option on the drive. Some guest operating systems may also require the
239 *SSD Emulation* flag to be set. Note that *Discard* on *VirtIO Block* drives is
240 only supported on guests using Linux Kernel 5.0 or higher.
241
242 If you would like a drive to be presented to the guest as a solid-state drive
243 rather than a rotational hard disk, you can set the *SSD emulation* option on
244 that drive. There is no requirement that the underlying storage actually be
245 backed by SSDs; this feature can be used with physical media of any type.
246 Note that *SSD emulation* is not supported on *VirtIO Block* drives.
247
248
249 [[qm_hard_disk_iothread]]
250 IO Thread
251 ^^^^^^^^^
252 The option *IO Thread* can only be used when using a disk with the
253 *VirtIO* controller, or with the *SCSI* controller, when the emulated controller
254 type is *VirtIO SCSI single*.
255 With this enabled, Qemu creates one I/O thread per storage controller,
256 rather than a single thread for all I/O. This can increase performance when
257 multiple disks are used and each disk has its own storage controller.
258
259
260 [[qm_cpu]]
261 CPU
262 ~~~
263
264 [thumbnail="screenshot/gui-create-vm-cpu.png"]
265
266 A *CPU socket* is a physical slot on a PC motherboard where you can plug a CPU.
267 This CPU can then contain one or many *cores*, which are independent
268 processing units. Whether you have a single CPU socket with 4 cores, or two CPU
269 sockets with two cores is mostly irrelevant from a performance point of view.
270 However some software licenses depend on the number of sockets a machine has,
271 in that case it makes sense to set the number of sockets to what the license
272 allows you.
273
274 Increasing the number of virtual cpus (cores and sockets) will usually provide a
275 performance improvement though that is heavily dependent on the use of the VM.
276 Multithreaded applications will of course benefit from a large number of
277 virtual cpus, as for each virtual cpu you add, Qemu will create a new thread of
278 execution on the host system. If you're not sure about the workload of your VM,
279 it is usually a safe bet to set the number of *Total cores* to 2.
280
281 NOTE: It is perfectly safe if the _overall_ number of cores of all your VMs
282 is greater than the number of cores on the server (e.g., 4 VMs with each 4
283 cores on a machine with only 8 cores). In that case the host system will
284 balance the Qemu execution threads between your server cores, just like if you
285 were running a standard multithreaded application. However, {pve} will prevent
286 you from starting VMs with more virtual CPU cores than physically available, as
287 this will only bring the performance down due to the cost of context switches.
288
289 [[qm_cpu_resource_limits]]
290 Resource Limits
291 ^^^^^^^^^^^^^^^
292
293 In addition to the number of virtual cores, you can configure how much resources
294 a VM can get in relation to the host CPU time and also in relation to other
295 VMs.
296 With the *cpulimit* (``Host CPU Time'') option you can limit how much CPU time
297 the whole VM can use on the host. It is a floating point value representing CPU
298 time in percent, so `1.0` is equal to `100%`, `2.5` to `250%` and so on. If a
299 single process would fully use one single core it would have `100%` CPU Time
300 usage. If a VM with four cores utilizes all its cores fully it would
301 theoretically use `400%`. In reality the usage may be even a bit higher as Qemu
302 can have additional threads for VM peripherals besides the vCPU core ones.
303 This setting can be useful if a VM should have multiple vCPUs, as it runs a few
304 processes in parallel, but the VM as a whole should not be able to run all
305 vCPUs at 100% at the same time. Using a specific example: lets say we have a VM
306 which would profit from having 8 vCPUs, but at no time all of those 8 cores
307 should run at full load - as this would make the server so overloaded that
308 other VMs and CTs would get to less CPU. So, we set the *cpulimit* limit to
309 `4.0` (=400%). If all cores do the same heavy work they would all get 50% of a
310 real host cores CPU time. But, if only 4 would do work they could still get
311 almost 100% of a real core each.
312
313 NOTE: VMs can, depending on their configuration, use additional threads e.g.,
314 for networking or IO operations but also live migration. Thus a VM can show up
315 to use more CPU time than just its virtual CPUs could use. To ensure that a VM
316 never uses more CPU time than virtual CPUs assigned set the *cpulimit* setting
317 to the same value as the total core count.
318
319 The second CPU resource limiting setting, *cpuunits* (nowadays often called CPU
320 shares or CPU weight), controls how much CPU time a VM gets in regards to other
321 VMs running. It is a relative weight which defaults to `1024`, if you increase
322 this for a VM it will be prioritized by the scheduler in comparison to other
323 VMs with lower weight. E.g., if VM 100 has set the default 1024 and VM 200 was
324 changed to `2048`, the latter VM 200 would receive twice the CPU bandwidth than
325 the first VM 100.
326
327 For more information see `man systemd.resource-control`, here `CPUQuota`
328 corresponds to `cpulimit` and `CPUShares` corresponds to our `cpuunits`
329 setting, visit its Notes section for references and implementation details.
330
331 CPU Type
332 ^^^^^^^^
333
334 Qemu can emulate a number different of *CPU types* from 486 to the latest Xeon
335 processors. Each new processor generation adds new features, like hardware
336 assisted 3d rendering, random number generation, memory protection, etc ...
337 Usually you should select for your VM a processor type which closely matches the
338 CPU of the host system, as it means that the host CPU features (also called _CPU
339 flags_ ) will be available in your VMs. If you want an exact match, you can set
340 the CPU type to *host* in which case the VM will have exactly the same CPU flags
341 as your host system.
342
343 This has a downside though. If you want to do a live migration of VMs between
344 different hosts, your VM might end up on a new system with a different CPU type.
345 If the CPU flags passed to the guest are missing, the qemu process will stop. To
346 remedy this Qemu has also its own CPU type *kvm64*, that {pve} uses by defaults.
347 kvm64 is a Pentium 4 look a like CPU type, which has a reduced CPU flags set,
348 but is guaranteed to work everywhere.
349
350 In short, if you care about live migration and moving VMs between nodes, leave
351 the kvm64 default. If you don’t care about live migration or have a homogeneous
352 cluster where all nodes have the same CPU, set the CPU type to host, as in
353 theory this will give your guests maximum performance.
354
355 Custom CPU Types
356 ^^^^^^^^^^^^^^^^
357
358 You can specify custom CPU types with a configurable set of features. These are
359 maintained in the configuration file `/etc/pve/virtual-guest/cpu-models.conf` by
360 an administrator. See `man cpu-models.conf` for format details.
361
362 Specified custom types can be selected by any user with the `Sys.Audit`
363 privilege on `/nodes`. When configuring a custom CPU type for a VM via the CLI
364 or API, the name needs to be prefixed with 'custom-'.
365
366 Meltdown / Spectre related CPU flags
367 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
368
369 There are several CPU flags related to the Meltdown and Spectre vulnerabilities
370 footnote:[Meltdown Attack https://meltdownattack.com/] which need to be set
371 manually unless the selected CPU type of your VM already enables them by default.
372
373 There are two requirements that need to be fulfilled in order to use these
374 CPU flags:
375
376 * The host CPU(s) must support the feature and propagate it to the guest's virtual CPU(s)
377 * The guest operating system must be updated to a version which mitigates the
378 attacks and is able to utilize the CPU feature
379
380 Otherwise you need to set the desired CPU flag of the virtual CPU, either by
381 editing the CPU options in the WebUI, or by setting the 'flags' property of the
382 'cpu' option in the VM configuration file.
383
384 For Spectre v1,v2,v4 fixes, your CPU or system vendor also needs to provide a
385 so-called ``microcode update'' footnote:[You can use `intel-microcode' /
386 `amd-microcode' from Debian non-free if your vendor does not provide such an
387 update. Note that not all affected CPUs can be updated to support spec-ctrl.]
388 for your CPU.
389
390
391 To check if the {pve} host is vulnerable, execute the following command as root:
392
393 ----
394 for f in /sys/devices/system/cpu/vulnerabilities/*; do echo "${f##*/} -" $(cat "$f"); done
395 ----
396
397 A community script is also available to detect is the host is still vulnerable.
398 footnote:[spectre-meltdown-checker https://meltdown.ovh/]
399
400 Intel processors
401 ^^^^^^^^^^^^^^^^
402
403 * 'pcid'
404 +
405 This reduces the performance impact of the Meltdown (CVE-2017-5754) mitigation
406 called 'Kernel Page-Table Isolation (KPTI)', which effectively hides
407 the Kernel memory from the user space. Without PCID, KPTI is quite an expensive
408 mechanism footnote:[PCID is now a critical performance/security feature on x86
409 https://groups.google.com/forum/m/#!topic/mechanical-sympathy/L9mHTbeQLNU].
410 +
411 To check if the {pve} host supports PCID, execute the following command as root:
412 +
413 ----
414 # grep ' pcid ' /proc/cpuinfo
415 ----
416 +
417 If this does not return empty your host's CPU has support for 'pcid'.
418
419 * 'spec-ctrl'
420 +
421 Required to enable the Spectre v1 (CVE-2017-5753) and Spectre v2 (CVE-2017-5715) fix,
422 in cases where retpolines are not sufficient.
423 Included by default in Intel CPU models with -IBRS suffix.
424 Must be explicitly turned on for Intel CPU models without -IBRS suffix.
425 Requires an updated host CPU microcode (intel-microcode >= 20180425).
426 +
427 * 'ssbd'
428 +
429 Required to enable the Spectre V4 (CVE-2018-3639) fix. Not included by default in any Intel CPU model.
430 Must be explicitly turned on for all Intel CPU models.
431 Requires an updated host CPU microcode(intel-microcode >= 20180703).
432
433
434 AMD processors
435 ^^^^^^^^^^^^^^
436
437 * 'ibpb'
438 +
439 Required to enable the Spectre v1 (CVE-2017-5753) and Spectre v2 (CVE-2017-5715) fix,
440 in cases where retpolines are not sufficient.
441 Included by default in AMD CPU models with -IBPB suffix.
442 Must be explicitly turned on for AMD CPU models without -IBPB suffix.
443 Requires the host CPU microcode to support this feature before it can be used for guest CPUs.
444
445
446
447 * 'virt-ssbd'
448 +
449 Required to enable the Spectre v4 (CVE-2018-3639) fix.
450 Not included by default in any AMD CPU model.
451 Must be explicitly turned on for all AMD CPU models.
452 This should be provided to guests, even if amd-ssbd is also provided, for maximum guest compatibility.
453 Note that this must be explicitly enabled when when using the "host" cpu model,
454 because this is a virtual feature which does not exist in the physical CPUs.
455
456
457 * 'amd-ssbd'
458 +
459 Required to enable the Spectre v4 (CVE-2018-3639) fix.
460 Not included by default in any AMD CPU model. Must be explicitly turned on for all AMD CPU models.
461 This provides higher performance than virt-ssbd, therefore a host supporting this should always expose this to guests if possible.
462 virt-ssbd should none the less also be exposed for maximum guest compatibility as some kernels only know about virt-ssbd.
463
464
465 * 'amd-no-ssb'
466 +
467 Recommended to indicate the host is not vulnerable to Spectre V4 (CVE-2018-3639).
468 Not included by default in any AMD CPU model.
469 Future hardware generations of CPU will not be vulnerable to CVE-2018-3639,
470 and thus the guest should be told not to enable its mitigations, by exposing amd-no-ssb.
471 This is mutually exclusive with virt-ssbd and amd-ssbd.
472
473
474 NUMA
475 ^^^^
476 You can also optionally emulate a *NUMA*
477 footnote:[https://en.wikipedia.org/wiki/Non-uniform_memory_access] architecture
478 in your VMs. The basics of the NUMA architecture mean that instead of having a
479 global memory pool available to all your cores, the memory is spread into local
480 banks close to each socket.
481 This can bring speed improvements as the memory bus is not a bottleneck
482 anymore. If your system has a NUMA architecture footnote:[if the command
483 `numactl --hardware | grep available` returns more than one node, then your host
484 system has a NUMA architecture] we recommend to activate the option, as this
485 will allow proper distribution of the VM resources on the host system.
486 This option is also required to hot-plug cores or RAM in a VM.
487
488 If the NUMA option is used, it is recommended to set the number of sockets to
489 the number of nodes of the host system.
490
491 vCPU hot-plug
492 ^^^^^^^^^^^^^
493
494 Modern operating systems introduced the capability to hot-plug and, to a
495 certain extent, hot-unplug CPUs in a running systems. Virtualisation allows us
496 to avoid a lot of the (physical) problems real hardware can cause in such
497 scenarios.
498 Still, this is a rather new and complicated feature, so its use should be
499 restricted to cases where its absolutely needed. Most of the functionality can
500 be replicated with other, well tested and less complicated, features, see
501 xref:qm_cpu_resource_limits[Resource Limits].
502
503 In {pve} the maximal number of plugged CPUs is always `cores * sockets`.
504 To start a VM with less than this total core count of CPUs you may use the
505 *vpus* setting, it denotes how many vCPUs should be plugged in at VM start.
506
507 Currently only this feature is only supported on Linux, a kernel newer than 3.10
508 is needed, a kernel newer than 4.7 is recommended.
509
510 You can use a udev rule as follow to automatically set new CPUs as online in
511 the guest:
512
513 ----
514 SUBSYSTEM=="cpu", ACTION=="add", TEST=="online", ATTR{online}=="0", ATTR{online}="1"
515 ----
516
517 Save this under /etc/udev/rules.d/ as a file ending in `.rules`.
518
519 Note: CPU hot-remove is machine dependent and requires guest cooperation.
520 The deletion command does not guarantee CPU removal to actually happen,
521 typically it's a request forwarded to guest using target dependent mechanism,
522 e.g., ACPI on x86/amd64.
523
524
525 [[qm_memory]]
526 Memory
527 ~~~~~~
528
529 For each VM you have the option to set a fixed size memory or asking
530 {pve} to dynamically allocate memory based on the current RAM usage of the
531 host.
532
533 .Fixed Memory Allocation
534 [thumbnail="screenshot/gui-create-vm-memory.png"]
535
536 When setting memory and minimum memory to the same amount
537 {pve} will simply allocate what you specify to your VM.
538
539 Even when using a fixed memory size, the ballooning device gets added to the
540 VM, because it delivers useful information such as how much memory the guest
541 really uses.
542 In general, you should leave *ballooning* enabled, but if you want to disable
543 it (e.g. for debugging purposes), simply uncheck
544 *Ballooning Device* or set
545
546 balloon: 0
547
548 in the configuration.
549
550 .Automatic Memory Allocation
551
552 // see autoballoon() in pvestatd.pm
553 When setting the minimum memory lower than memory, {pve} will make sure that the
554 minimum amount you specified is always available to the VM, and if RAM usage on
555 the host is below 80%, will dynamically add memory to the guest up to the
556 maximum memory specified.
557
558 When the host is running low on RAM, the VM will then release some memory
559 back to the host, swapping running processes if needed and starting the oom
560 killer in last resort. The passing around of memory between host and guest is
561 done via a special `balloon` kernel driver running inside the guest, which will
562 grab or release memory pages from the host.
563 footnote:[A good explanation of the inner workings of the balloon driver can be found here https://rwmj.wordpress.com/2010/07/17/virtio-balloon/]
564
565 When multiple VMs use the autoallocate facility, it is possible to set a
566 *Shares* coefficient which indicates the relative amount of the free host memory
567 that each VM should take. Suppose for instance you have four VMs, three of them
568 running an HTTP server and the last one is a database server. To cache more
569 database blocks in the database server RAM, you would like to prioritize the
570 database VM when spare RAM is available. For this you assign a Shares property
571 of 3000 to the database VM, leaving the other VMs to the Shares default setting
572 of 1000. The host server has 32GB of RAM, and is currently using 16GB, leaving 32
573 * 80/100 - 16 = 9GB RAM to be allocated to the VMs. The database VM will get 9 *
574 3000 / (3000 + 1000 + 1000 + 1000) = 4.5 GB extra RAM and each HTTP server will
575 get 1.5 GB.
576
577 All Linux distributions released after 2010 have the balloon kernel driver
578 included. For Windows OSes, the balloon driver needs to be added manually and can
579 incur a slowdown of the guest, so we don't recommend using it on critical
580 systems.
581 // see https://forum.proxmox.com/threads/solved-hyper-threading-vs-no-hyper-threading-fixed-vs-variable-memory.20265/
582
583 When allocating RAM to your VMs, a good rule of thumb is always to leave 1GB
584 of RAM available to the host.
585
586
587 [[qm_network_device]]
588 Network Device
589 ~~~~~~~~~~~~~~
590
591 [thumbnail="screenshot/gui-create-vm-network.png"]
592
593 Each VM can have many _Network interface controllers_ (NIC), of four different
594 types:
595
596 * *Intel E1000* is the default, and emulates an Intel Gigabit network card.
597 * the *VirtIO* paravirtualized NIC should be used if you aim for maximum
598 performance. Like all VirtIO devices, the guest OS should have the proper driver
599 installed.
600 * the *Realtek 8139* emulates an older 100 MB/s network card, and should
601 only be used when emulating older operating systems ( released before 2002 )
602 * the *vmxnet3* is another paravirtualized device, which should only be used
603 when importing a VM from another hypervisor.
604
605 {pve} will generate for each NIC a random *MAC address*, so that your VM is
606 addressable on Ethernet networks.
607
608 The NIC you added to the VM can follow one of two different models:
609
610 * in the default *Bridged mode* each virtual NIC is backed on the host by a
611 _tap device_, ( a software loopback device simulating an Ethernet NIC ). This
612 tap device is added to a bridge, by default vmbr0 in {pve}. In this mode, VMs
613 have direct access to the Ethernet LAN on which the host is located.
614 * in the alternative *NAT mode*, each virtual NIC will only communicate with
615 the Qemu user networking stack, where a built-in router and DHCP server can
616 provide network access. This built-in DHCP will serve addresses in the private
617 10.0.2.0/24 range. The NAT mode is much slower than the bridged mode, and
618 should only be used for testing. This mode is only available via CLI or the API,
619 but not via the WebUI.
620
621 You can also skip adding a network device when creating a VM by selecting *No
622 network device*.
623
624 .Multiqueue
625 If you are using the VirtIO driver, you can optionally activate the
626 *Multiqueue* option. This option allows the guest OS to process networking
627 packets using multiple virtual CPUs, providing an increase in the total number
628 of packets transferred.
629
630 //http://blog.vmsplice.net/2011/09/qemu-internals-vhost-architecture.html
631 When using the VirtIO driver with {pve}, each NIC network queue is passed to the
632 host kernel, where the queue will be processed by a kernel thread spawned by the
633 vhost driver. With this option activated, it is possible to pass _multiple_
634 network queues to the host kernel for each NIC.
635
636 //https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Tuning_and_Optimization_Guide/sect-Virtualization_Tuning_Optimization_Guide-Networking-Techniques.html#sect-Virtualization_Tuning_Optimization_Guide-Networking-Multi-queue_virtio-net
637 When using Multiqueue, it is recommended to set it to a value equal
638 to the number of Total Cores of your guest. You also need to set in
639 the VM the number of multi-purpose channels on each VirtIO NIC with the ethtool
640 command:
641
642 `ethtool -L ens1 combined X`
643
644 where X is the number of the number of vcpus of the VM.
645
646 You should note that setting the Multiqueue parameter to a value greater
647 than one will increase the CPU load on the host and guest systems as the
648 traffic increases. We recommend to set this option only when the VM has to
649 process a great number of incoming connections, such as when the VM is running
650 as a router, reverse proxy or a busy HTTP server doing long polling.
651
652 [[qm_display]]
653 Display
654 ~~~~~~~
655
656 QEMU can virtualize a few types of VGA hardware. Some examples are:
657
658 * *std*, the default, emulates a card with Bochs VBE extensions.
659 * *cirrus*, this was once the default, it emulates a very old hardware module
660 with all its problems. This display type should only be used if really
661 necessary footnote:[https://www.kraxel.org/blog/2014/10/qemu-using-cirrus-considered-harmful/
662 qemu: using cirrus considered harmful], e.g., if using Windows XP or earlier
663 * *vmware*, is a VMWare SVGA-II compatible adapter.
664 * *qxl*, is the QXL paravirtualized graphics card. Selecting this also
665 enables https://www.spice-space.org/[SPICE] (a remote viewer protocol) for the
666 VM.
667
668 You can edit the amount of memory given to the virtual GPU, by setting
669 the 'memory' option. This can enable higher resolutions inside the VM,
670 especially with SPICE/QXL.
671
672 As the memory is reserved by display device, selecting Multi-Monitor mode
673 for SPICE (e.g., `qxl2` for dual monitors) has some implications:
674
675 * Windows needs a device for each monitor, so if your 'ostype' is some
676 version of Windows, {pve} gives the VM an extra device per monitor.
677 Each device gets the specified amount of memory.
678
679 * Linux VMs, can always enable more virtual monitors, but selecting
680 a Multi-Monitor mode multiplies the memory given to the device with
681 the number of monitors.
682
683 Selecting `serialX` as display 'type' disables the VGA output, and redirects
684 the Web Console to the selected serial port. A configured display 'memory'
685 setting will be ignored in that case.
686
687 [[qm_usb_passthrough]]
688 USB Passthrough
689 ~~~~~~~~~~~~~~~
690
691 There are two different types of USB passthrough devices:
692
693 * Host USB passthrough
694 * SPICE USB passthrough
695
696 Host USB passthrough works by giving a VM a USB device of the host.
697 This can either be done via the vendor- and product-id, or
698 via the host bus and port.
699
700 The vendor/product-id looks like this: *0123:abcd*,
701 where *0123* is the id of the vendor, and *abcd* is the id
702 of the product, meaning two pieces of the same usb device
703 have the same id.
704
705 The bus/port looks like this: *1-2.3.4*, where *1* is the bus
706 and *2.3.4* is the port path. This represents the physical
707 ports of your host (depending of the internal order of the
708 usb controllers).
709
710 If a device is present in a VM configuration when the VM starts up,
711 but the device is not present in the host, the VM can boot without problems.
712 As soon as the device/port is available in the host, it gets passed through.
713
714 WARNING: Using this kind of USB passthrough means that you cannot move
715 a VM online to another host, since the hardware is only available
716 on the host the VM is currently residing.
717
718 The second type of passthrough is SPICE USB passthrough. This is useful
719 if you use a SPICE client which supports it. If you add a SPICE USB port
720 to your VM, you can passthrough a USB device from where your SPICE client is,
721 directly to the VM (for example an input device or hardware dongle).
722
723
724 [[qm_bios_and_uefi]]
725 BIOS and UEFI
726 ~~~~~~~~~~~~~
727
728 In order to properly emulate a computer, QEMU needs to use a firmware.
729 Which, on common PCs often known as BIOS or (U)EFI, is executed as one of the
730 first steps when booting a VM. It is responsible for doing basic hardware
731 initialization and for providing an interface to the firmware and hardware for
732 the operating system. By default QEMU uses *SeaBIOS* for this, which is an
733 open-source, x86 BIOS implementation. SeaBIOS is a good choice for most
734 standard setups.
735
736 There are, however, some scenarios in which a BIOS is not a good firmware
737 to boot from, e.g. if you want to do VGA passthrough. footnote:[Alex Williamson has a very good blog entry about this.
738 http://vfio.blogspot.co.at/2014/08/primary-graphics-assignment-without-vga.html]
739 In such cases, you should rather use *OVMF*, which is an open-source UEFI implementation. footnote:[See the OVMF Project http://www.tianocore.org/ovmf/]
740
741 If you want to use OVMF, there are several things to consider:
742
743 In order to save things like the *boot order*, there needs to be an EFI Disk.
744 This disk will be included in backups and snapshots, and there can only be one.
745
746 You can create such a disk with the following command:
747
748 qm set <vmid> -efidisk0 <storage>:1,format=<format>
749
750 Where *<storage>* is the storage where you want to have the disk, and
751 *<format>* is a format which the storage supports. Alternatively, you can
752 create such a disk through the web interface with 'Add' -> 'EFI Disk' in the
753 hardware section of a VM.
754
755 When using OVMF with a virtual display (without VGA passthrough),
756 you need to set the client resolution in the OVMF menu(which you can reach
757 with a press of the ESC button during boot), or you have to choose
758 SPICE as the display type.
759
760 [[qm_ivshmem]]
761 Inter-VM shared memory
762 ~~~~~~~~~~~~~~~~~~~~~~
763
764 You can add an Inter-VM shared memory device (`ivshmem`), which allows one to
765 share memory between the host and a guest, or also between multiple guests.
766
767 To add such a device, you can use `qm`:
768
769 qm set <vmid> -ivshmem size=32,name=foo
770
771 Where the size is in MiB. The file will be located under
772 `/dev/shm/pve-shm-$name` (the default name is the vmid).
773
774 NOTE: Currently the device will get deleted as soon as any VM using it got
775 shutdown or stopped. Open connections will still persist, but new connections
776 to the exact same device cannot be made anymore.
777
778 A use case for such a device is the Looking Glass
779 footnote:[Looking Glass: https://looking-glass.io/] project, which enables high
780 performance, low-latency display mirroring between host and guest.
781
782 [[qm_audio_device]]
783 Audio Device
784 ~~~~~~~~~~~~
785
786 To add an audio device run the following command:
787
788 ----
789 qm set <vmid> -audio0 device=<device>
790 ----
791
792 Supported audio devices are:
793
794 * `ich9-intel-hda`: Intel HD Audio Controller, emulates ICH9
795 * `intel-hda`: Intel HD Audio Controller, emulates ICH6
796 * `AC97`: Audio Codec '97, useful for older operating systems like Windows XP
797
798 There are two backends available:
799
800 * 'spice'
801 * 'none'
802
803 The 'spice' backend can be used in combination with xref:qm_display[SPICE] while
804 the 'none' backend can be useful if an audio device is needed in the VM for some
805 software to work. To use the physical audio device of the host use device
806 passthrough (see xref:qm_pci_passthrough[PCI Passthrough] and
807 xref:qm_usb_passthrough[USB Passthrough]). Remote protocols like Microsoft’s RDP
808 have options to play sound.
809
810
811 [[qm_virtio_rng]]
812 VirtIO RNG
813 ~~~~~~~~~~
814
815 A RNG (Random Number Generator) is a device providing entropy ('randomness') to
816 a system. A virtual hardware-RNG can be used to provide such entropy from the
817 host system to a guest VM. This helps to avoid entropy starvation problems in
818 the guest (a situation where not enough entropy is available and the system may
819 slow down or run into problems), especially during the guests boot process.
820
821 To add a VirtIO-based emulated RNG, run the following command:
822
823 ----
824 qm set <vmid> -rng0 source=<source>[,max_bytes=X,period=Y]
825 ----
826
827 `source` specifies where entropy is read from on the host and has to be one of
828 the following:
829
830 * `/dev/urandom`: Non-blocking kernel entropy pool (preferred)
831 * `/dev/random`: Blocking kernel pool (not recommended, can lead to entropy
832 starvation on the host system)
833 * `/dev/hwrng`: To pass through a hardware RNG attached to the host (if multiple
834 are available, the one selected in
835 `/sys/devices/virtual/misc/hw_random/rng_current` will be used)
836
837 A limit can be specified via the `max_bytes` and `period` parameters, they are
838 read as `max_bytes` per `period` in milliseconds. However, it does not represent
839 a linear relationship: 1024B/1000ms would mean that up to 1 KiB of data becomes
840 available on a 1 second timer, not that 1 KiB is streamed to the guest over the
841 course of one second. Reducing the `period` can thus be used to inject entropy
842 into the guest at a faster rate.
843
844 By default, the limit is set to 1024 bytes per 1000 ms (1 KiB/s). It is
845 recommended to always use a limiter to avoid guests using too many host
846 resources. If desired, a value of '0' for `max_bytes` can be used to disable
847 all limits.
848
849 [[qm_bootorder]]
850 Device Boot Order
851 ~~~~~~~~~~~~~~~~~
852
853 QEMU can tell the guest which devices it should boot from, and in which order.
854 This can be specified in the config via the `boot` property, e.g.:
855
856 ----
857 boot: order=scsi0;net0;hostpci0
858 ----
859
860 [thumbnail="screenshot/gui-qemu-edit-bootorder.png"]
861
862 This way, the guest would first attempt to boot from the disk `scsi0`, if that
863 fails, it would go on to attempt network boot from `net0`, and in case that
864 fails too, finally attempt to boot from a passed through PCIe device (seen as
865 disk in case of NVMe, otherwise tries to launch into an option ROM).
866
867 On the GUI you can use a drag-and-drop editor to specify the boot order, and use
868 the checkbox to enable or disable certain devices for booting altogether.
869
870 NOTE: If your guest uses multiple disks to boot the OS or load the bootloader,
871 all of them must be marked as 'bootable' (that is, they must have the checkbox
872 enabled or appear in the list in the config) for the guest to be able to boot.
873 This is because recent SeaBIOS and OVMF versions only initialize disks if they
874 are marked 'bootable'.
875
876 In any case, even devices not appearing in the list or having the checkmark
877 disabled will still be available to the guest, once it's operating system has
878 booted and initialized them. The 'bootable' flag only affects the guest BIOS and
879 bootloader.
880
881
882 [[qm_startup_and_shutdown]]
883 Automatic Start and Shutdown of Virtual Machines
884 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
885
886 After creating your VMs, you probably want them to start automatically
887 when the host system boots. For this you need to select the option 'Start at
888 boot' from the 'Options' Tab of your VM in the web interface, or set it with
889 the following command:
890
891 qm set <vmid> -onboot 1
892
893 .Start and Shutdown Order
894
895 [thumbnail="screenshot/gui-qemu-edit-start-order.png"]
896
897 In some case you want to be able to fine tune the boot order of your
898 VMs, for instance if one of your VM is providing firewalling or DHCP
899 to other guest systems. For this you can use the following
900 parameters:
901
902 * *Start/Shutdown order*: Defines the start order priority. E.g. set it to 1 if
903 you want the VM to be the first to be started. (We use the reverse startup
904 order for shutdown, so a machine with a start order of 1 would be the last to
905 be shut down). If multiple VMs have the same order defined on a host, they will
906 additionally be ordered by 'VMID' in ascending order.
907 * *Startup delay*: Defines the interval between this VM start and subsequent
908 VMs starts . E.g. set it to 240 if you want to wait 240 seconds before starting
909 other VMs.
910 * *Shutdown timeout*: Defines the duration in seconds {pve} should wait
911 for the VM to be offline after issuing a shutdown command.
912 By default this value is set to 180, which means that {pve} will issue a
913 shutdown request and wait 180 seconds for the machine to be offline. If
914 the machine is still online after the timeout it will be stopped forcefully.
915
916 NOTE: VMs managed by the HA stack do not follow the 'start on boot' and
917 'boot order' options currently. Those VMs will be skipped by the startup and
918 shutdown algorithm as the HA manager itself ensures that VMs get started and
919 stopped.
920
921 Please note that machines without a Start/Shutdown order parameter will always
922 start after those where the parameter is set. Further, this parameter can only
923 be enforced between virtual machines running on the same host, not
924 cluster-wide.
925
926
927 [[qm_qemu_agent]]
928 Qemu Guest Agent
929 ~~~~~~~~~~~~~~~~
930
931 The Qemu Guest Agent is a service which runs inside the VM, providing a
932 communication channel between the host and the guest. It is used to exchange
933 information and allows the host to issue commands to the guest.
934
935 For example, the IP addresses in the VM summary panel are fetched via the guest
936 agent.
937
938 Or when starting a backup, the guest is told via the guest agent to sync
939 outstanding writes via the 'fs-freeze' and 'fs-thaw' commands.
940
941 For the guest agent to work properly the following steps must be taken:
942
943 * install the agent in the guest and make sure it is running
944 * enable the communication via the agent in {pve}
945
946 Install Guest Agent
947 ^^^^^^^^^^^^^^^^^^^
948
949 For most Linux distributions, the guest agent is available. The package is
950 usually named `qemu-guest-agent`.
951
952 For Windows, it can be installed from the
953 https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso[Fedora
954 VirtIO driver ISO].
955
956 Enable Guest Agent Communication
957 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
958
959 Communication from {pve} with the guest agent can be enabled in the VM's
960 *Options* panel. A fresh start of the VM is necessary for the changes to take
961 effect.
962
963 It is possible to enable the 'Run guest-trim' option. With this enabled,
964 {pve} will issue a trim command to the guest after the following
965 operations that have the potential to write out zeros to the storage:
966
967 * moving a disk to another storage
968 * live migrating a VM to another node with local storage
969
970 On a thin provisioned storage, this can help to free up unused space.
971
972 Troubleshooting
973 ^^^^^^^^^^^^^^^
974
975 .VM does not shut down
976
977 Make sure the guest agent is installed and running.
978
979 Once the guest agent is enabled, {pve} will send power commands like
980 'shutdown' via the guest agent. If the guest agent is not running, commands
981 cannot get executed properly and the shutdown command will run into a timeout.
982
983 [[qm_spice_enhancements]]
984 SPICE Enhancements
985 ~~~~~~~~~~~~~~~~~~
986
987 SPICE Enhancements are optional features that can improve the remote viewer
988 experience.
989
990 To enable them via the GUI go to the *Options* panel of the virtual machine. Run
991 the following command to enable them via the CLI:
992
993 ----
994 qm set <vmid> -spice_enhancements foldersharing=1,videostreaming=all
995 ----
996
997 NOTE: To use these features the <<qm_display,*Display*>> of the virtual machine
998 must be set to SPICE (qxl).
999
1000 Folder Sharing
1001 ^^^^^^^^^^^^^^
1002
1003 Share a local folder with the guest. The `spice-webdavd` daemon needs to be
1004 installed in the guest. It makes the shared folder available through a local
1005 WebDAV server located at http://localhost:9843.
1006
1007 For Windows guests the installer for the 'Spice WebDAV daemon' can be downloaded
1008 from the
1009 https://www.spice-space.org/download.html#windows-binaries[official SPICE website].
1010
1011 Most Linux distributions have a package called `spice-webdavd` that can be
1012 installed.
1013
1014 To share a folder in Virt-Viewer (Remote Viewer) go to 'File -> Preferences'.
1015 Select the folder to share and then enable the checkbox.
1016
1017 NOTE: Folder sharing currently only works in the Linux version of Virt-Viewer.
1018
1019 CAUTION: Experimental! Currently this feature does not work reliably.
1020
1021 Video Streaming
1022 ^^^^^^^^^^^^^^^
1023
1024 Fast refreshing areas are encoded into a video stream. Two options exist:
1025
1026 * *all*: Any fast refreshing area will be encoded into a video stream.
1027 * *filter*: Additional filters are used to decide if video streaming should be
1028 used (currently only small window surfaces are skipped).
1029
1030 A general recommendation if video streaming should be enabled and which option
1031 to choose from cannot be given. Your mileage may vary depending on the specific
1032 circumstances.
1033
1034 Troubleshooting
1035 ^^^^^^^^^^^^^^^
1036
1037 .Shared folder does not show up
1038
1039 Make sure the WebDAV service is enabled and running in the guest. On Windows it
1040 is called 'Spice webdav proxy'. In Linux the name is 'spice-webdavd' but can be
1041 different depending on the distribution.
1042
1043 If the service is running, check the WebDAV server by opening
1044 http://localhost:9843 in a browser in the guest.
1045
1046 It can help to restart the SPICE session.
1047
1048 [[qm_migration]]
1049 Migration
1050 ---------
1051
1052 [thumbnail="screenshot/gui-qemu-migrate.png"]
1053
1054 If you have a cluster, you can migrate your VM to another host with
1055
1056 qm migrate <vmid> <target>
1057
1058 There are generally two mechanisms for this
1059
1060 * Online Migration (aka Live Migration)
1061 * Offline Migration
1062
1063 Online Migration
1064 ~~~~~~~~~~~~~~~~
1065
1066 When your VM is running and it has no local resources defined (such as disks
1067 on local storage, passed through devices, etc.) you can initiate a live
1068 migration with the -online flag.
1069
1070 How it works
1071 ^^^^^^^^^^^^
1072
1073 This starts a Qemu Process on the target host with the 'incoming' flag, which
1074 means that the process starts and waits for the memory data and device states
1075 from the source Virtual Machine (since all other resources, e.g. disks,
1076 are shared, the memory content and device state are the only things left
1077 to transmit).
1078
1079 Once this connection is established, the source begins to send the memory
1080 content asynchronously to the target. If the memory on the source changes,
1081 those sections are marked dirty and there will be another pass of sending data.
1082 This happens until the amount of data to send is so small that it can
1083 pause the VM on the source, send the remaining data to the target and start
1084 the VM on the target in under a second.
1085
1086 Requirements
1087 ^^^^^^^^^^^^
1088
1089 For Live Migration to work, there are some things required:
1090
1091 * The VM has no local resources (e.g. passed through devices, local disks, etc.)
1092 * The hosts are in the same {pve} cluster.
1093 * The hosts have a working (and reliable) network connection.
1094 * The target host must have the same or higher versions of the
1095 {pve} packages. (It *might* work the other way, but this is never guaranteed)
1096
1097 Offline Migration
1098 ~~~~~~~~~~~~~~~~~
1099
1100 If you have local resources, you can still offline migrate your VMs,
1101 as long as all disk are on storages, which are defined on both hosts.
1102 Then the migration will copy the disk over the network to the target host.
1103
1104 [[qm_copy_and_clone]]
1105 Copies and Clones
1106 -----------------
1107
1108 [thumbnail="screenshot/gui-qemu-full-clone.png"]
1109
1110 VM installation is usually done using an installation media (CD-ROM)
1111 from the operation system vendor. Depending on the OS, this can be a
1112 time consuming task one might want to avoid.
1113
1114 An easy way to deploy many VMs of the same type is to copy an existing
1115 VM. We use the term 'clone' for such copies, and distinguish between
1116 'linked' and 'full' clones.
1117
1118 Full Clone::
1119
1120 The result of such copy is an independent VM. The
1121 new VM does not share any storage resources with the original.
1122 +
1123
1124 It is possible to select a *Target Storage*, so one can use this to
1125 migrate a VM to a totally different storage. You can also change the
1126 disk image *Format* if the storage driver supports several formats.
1127 +
1128
1129 NOTE: A full clone needs to read and copy all VM image data. This is
1130 usually much slower than creating a linked clone.
1131 +
1132
1133 Some storage types allows to copy a specific *Snapshot*, which
1134 defaults to the 'current' VM data. This also means that the final copy
1135 never includes any additional snapshots from the original VM.
1136
1137
1138 Linked Clone::
1139
1140 Modern storage drivers support a way to generate fast linked
1141 clones. Such a clone is a writable copy whose initial contents are the
1142 same as the original data. Creating a linked clone is nearly
1143 instantaneous, and initially consumes no additional space.
1144 +
1145
1146 They are called 'linked' because the new image still refers to the
1147 original. Unmodified data blocks are read from the original image, but
1148 modification are written (and afterwards read) from a new
1149 location. This technique is called 'Copy-on-write'.
1150 +
1151
1152 This requires that the original volume is read-only. With {pve} one
1153 can convert any VM into a read-only <<qm_templates, Template>>). Such
1154 templates can later be used to create linked clones efficiently.
1155 +
1156
1157 NOTE: You cannot delete an original template while linked clones
1158 exist.
1159 +
1160
1161 It is not possible to change the *Target storage* for linked clones,
1162 because this is a storage internal feature.
1163
1164
1165 The *Target node* option allows you to create the new VM on a
1166 different node. The only restriction is that the VM is on shared
1167 storage, and that storage is also available on the target node.
1168
1169 To avoid resource conflicts, all network interface MAC addresses get
1170 randomized, and we generate a new 'UUID' for the VM BIOS (smbios1)
1171 setting.
1172
1173
1174 [[qm_templates]]
1175 Virtual Machine Templates
1176 -------------------------
1177
1178 One can convert a VM into a Template. Such templates are read-only,
1179 and you can use them to create linked clones.
1180
1181 NOTE: It is not possible to start templates, because this would modify
1182 the disk images. If you want to change the template, create a linked
1183 clone and modify that.
1184
1185 VM Generation ID
1186 ----------------
1187
1188 {pve} supports Virtual Machine Generation ID ('vmgenid') footnote:[Official
1189 'vmgenid' Specification
1190 https://docs.microsoft.com/en-us/windows/desktop/hyperv_v2/virtual-machine-generation-identifier]
1191 for virtual machines.
1192 This can be used by the guest operating system to detect any event resulting
1193 in a time shift event, for example, restoring a backup or a snapshot rollback.
1194
1195 When creating new VMs, a 'vmgenid' will be automatically generated and saved
1196 in its configuration file.
1197
1198 To create and add a 'vmgenid' to an already existing VM one can pass the
1199 special value `1' to let {pve} autogenerate one or manually set the 'UUID'
1200 footnote:[Online GUID generator http://guid.one/] by using it as value,
1201 e.g.:
1202
1203 ----
1204 qm set VMID -vmgenid 1
1205 qm set VMID -vmgenid 00000000-0000-0000-0000-000000000000
1206 ----
1207
1208 NOTE: The initial addition of a 'vmgenid' device to an existing VM, may result
1209 in the same effects as a change on snapshot rollback, backup restore, etc., has
1210 as the VM can interpret this as generation change.
1211
1212 In the rare case the 'vmgenid' mechanism is not wanted one can pass `0' for
1213 its value on VM creation, or retroactively delete the property in the
1214 configuration with:
1215
1216 ----
1217 qm set VMID -delete vmgenid
1218 ----
1219
1220 The most prominent use case for 'vmgenid' are newer Microsoft Windows
1221 operating systems, which use it to avoid problems in time sensitive or
1222 replicate services (e.g., databases, domain controller
1223 footnote:[https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/get-started/virtual-dc/virtualized-domain-controller-architecture])
1224 on snapshot rollback, backup restore or a whole VM clone operation.
1225
1226 Importing Virtual Machines and disk images
1227 ------------------------------------------
1228
1229 A VM export from a foreign hypervisor takes usually the form of one or more disk
1230 images, with a configuration file describing the settings of the VM (RAM,
1231 number of cores). +
1232 The disk images can be in the vmdk format, if the disks come from
1233 VMware or VirtualBox, or qcow2 if the disks come from a KVM hypervisor.
1234 The most popular configuration format for VM exports is the OVF standard, but in
1235 practice interoperation is limited because many settings are not implemented in
1236 the standard itself, and hypervisors export the supplementary information
1237 in non-standard extensions.
1238
1239 Besides the problem of format, importing disk images from other hypervisors
1240 may fail if the emulated hardware changes too much from one hypervisor to
1241 another. Windows VMs are particularly concerned by this, as the OS is very
1242 picky about any changes of hardware. This problem may be solved by
1243 installing the MergeIDE.zip utility available from the Internet before exporting
1244 and choosing a hard disk type of *IDE* before booting the imported Windows VM.
1245
1246 Finally there is the question of paravirtualized drivers, which improve the
1247 speed of the emulated system and are specific to the hypervisor.
1248 GNU/Linux and other free Unix OSes have all the necessary drivers installed by
1249 default and you can switch to the paravirtualized drivers right after importing
1250 the VM. For Windows VMs, you need to install the Windows paravirtualized
1251 drivers by yourself.
1252
1253 GNU/Linux and other free Unix can usually be imported without hassle. Note
1254 that we cannot guarantee a successful import/export of Windows VMs in all
1255 cases due to the problems above.
1256
1257 Step-by-step example of a Windows OVF import
1258 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1259
1260 Microsoft provides
1261 https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/[Virtual Machines downloads]
1262 to get started with Windows development.We are going to use one of these
1263 to demonstrate the OVF import feature.
1264
1265 Download the Virtual Machine zip
1266 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1267
1268 After getting informed about the user agreement, choose the _Windows 10
1269 Enterprise (Evaluation - Build)_ for the VMware platform, and download the zip.
1270
1271 Extract the disk image from the zip
1272 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1273
1274 Using the `unzip` utility or any archiver of your choice, unpack the zip,
1275 and copy via ssh/scp the ovf and vmdk files to your {pve} host.
1276
1277 Import the Virtual Machine
1278 ^^^^^^^^^^^^^^^^^^^^^^^^^^
1279
1280 This will create a new virtual machine, using cores, memory and
1281 VM name as read from the OVF manifest, and import the disks to the +local-lvm+
1282 storage. You have to configure the network manually.
1283
1284 qm importovf 999 WinDev1709Eval.ovf local-lvm
1285
1286 The VM is ready to be started.
1287
1288 Adding an external disk image to a Virtual Machine
1289 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1290
1291 You can also add an existing disk image to a VM, either coming from a
1292 foreign hypervisor, or one that you created yourself.
1293
1294 Suppose you created a Debian/Ubuntu disk image with the 'vmdebootstrap' tool:
1295
1296 vmdebootstrap --verbose \
1297 --size 10GiB --serial-console \
1298 --grub --no-extlinux \
1299 --package openssh-server \
1300 --package avahi-daemon \
1301 --package qemu-guest-agent \
1302 --hostname vm600 --enable-dhcp \
1303 --customize=./copy_pub_ssh.sh \
1304 --sparse --image vm600.raw
1305
1306 You can now create a new target VM for this image.
1307
1308 qm create 600 --net0 virtio,bridge=vmbr0 --name vm600 --serial0 socket \
1309 --bootdisk scsi0 --scsihw virtio-scsi-pci --ostype l26
1310
1311 Add the disk image as +unused0+ to the VM, using the storage +pvedir+:
1312
1313 qm importdisk 600 vm600.raw pvedir
1314
1315 Finally attach the unused disk to the SCSI controller of the VM:
1316
1317 qm set 600 --scsi0 pvedir:600/vm-600-disk-1.raw
1318
1319 The VM is ready to be started.
1320
1321
1322 ifndef::wiki[]
1323 include::qm-cloud-init.adoc[]
1324 endif::wiki[]
1325
1326 ifndef::wiki[]
1327 include::qm-pci-passthrough.adoc[]
1328 endif::wiki[]
1329
1330 Hookscripts
1331 -----------
1332
1333 You can add a hook script to VMs with the config property `hookscript`.
1334
1335 qm set 100 -hookscript local:snippets/hookscript.pl
1336
1337 It will be called during various phases of the guests lifetime.
1338 For an example and documentation see the example script under
1339 `/usr/share/pve-docs/examples/guest-example-hookscript.pl`.
1340
1341 [[qm_hibernate]]
1342 Hibernation
1343 -----------
1344
1345 You can suspend a VM to disk with the GUI option `Hibernate` or with
1346
1347 qm suspend ID --todisk
1348
1349 That means that the current content of the memory will be saved onto disk
1350 and the VM gets stopped. On the next start, the memory content will be
1351 loaded and the VM can continue where it was left off.
1352
1353 [[qm_vmstatestorage]]
1354 .State storage selection
1355 If no target storage for the memory is given, it will be automatically
1356 chosen, the first of:
1357
1358 1. The storage `vmstatestorage` from the VM config.
1359 2. The first shared storage from any VM disk.
1360 3. The first non-shared storage from any VM disk.
1361 4. The storage `local` as a fallback.
1362
1363 Managing Virtual Machines with `qm`
1364 ------------------------------------
1365
1366 qm is the tool to manage Qemu/Kvm virtual machines on {pve}. You can
1367 create and destroy virtual machines, and control execution
1368 (start/stop/suspend/resume). Besides that, you can use qm to set
1369 parameters in the associated config file. It is also possible to
1370 create and delete virtual disks.
1371
1372 CLI Usage Examples
1373 ~~~~~~~~~~~~~~~~~~
1374
1375 Using an iso file uploaded on the 'local' storage, create a VM
1376 with a 4 GB IDE disk on the 'local-lvm' storage
1377
1378 qm create 300 -ide0 local-lvm:4 -net0 e1000 -cdrom local:iso/proxmox-mailgateway_2.1.iso
1379
1380 Start the new VM
1381
1382 qm start 300
1383
1384 Send a shutdown request, then wait until the VM is stopped.
1385
1386 qm shutdown 300 && qm wait 300
1387
1388 Same as above, but only wait for 40 seconds.
1389
1390 qm shutdown 300 && qm wait 300 -timeout 40
1391
1392 Destroying a VM always removes it from Access Control Lists and it always
1393 removes the firewall configuration of the VM. You have to activate
1394 '--purge', if you want to additionally remove the VM from replication jobs,
1395 backup jobs and HA resource configurations.
1396
1397 qm destroy 300 --purge
1398
1399
1400
1401 [[qm_configuration]]
1402 Configuration
1403 -------------
1404
1405 VM configuration files are stored inside the Proxmox cluster file
1406 system, and can be accessed at `/etc/pve/qemu-server/<VMID>.conf`.
1407 Like other files stored inside `/etc/pve/`, they get automatically
1408 replicated to all other cluster nodes.
1409
1410 NOTE: VMIDs < 100 are reserved for internal purposes, and VMIDs need to be
1411 unique cluster wide.
1412
1413 .Example VM Configuration
1414 ----
1415 boot: order=virtio0;net0
1416 cores: 1
1417 sockets: 1
1418 memory: 512
1419 name: webmail
1420 ostype: l26
1421 net0: e1000=EE:D2:28:5F:B6:3E,bridge=vmbr0
1422 virtio0: local:vm-100-disk-1,size=32G
1423 ----
1424
1425 Those configuration files are simple text files, and you can edit them
1426 using a normal text editor (`vi`, `nano`, ...). This is sometimes
1427 useful to do small corrections, but keep in mind that you need to
1428 restart the VM to apply such changes.
1429
1430 For that reason, it is usually better to use the `qm` command to
1431 generate and modify those files, or do the whole thing using the GUI.
1432 Our toolkit is smart enough to instantaneously apply most changes to
1433 running VM. This feature is called "hot plug", and there is no
1434 need to restart the VM in that case.
1435
1436
1437 File Format
1438 ~~~~~~~~~~~
1439
1440 VM configuration files use a simple colon separated key/value
1441 format. Each line has the following format:
1442
1443 -----
1444 # this is a comment
1445 OPTION: value
1446 -----
1447
1448 Blank lines in those files are ignored, and lines starting with a `#`
1449 character are treated as comments and are also ignored.
1450
1451
1452 [[qm_snapshots]]
1453 Snapshots
1454 ~~~~~~~~~
1455
1456 When you create a snapshot, `qm` stores the configuration at snapshot
1457 time into a separate snapshot section within the same configuration
1458 file. For example, after creating a snapshot called ``testsnapshot'',
1459 your configuration file will look like this:
1460
1461 .VM configuration with snapshot
1462 ----
1463 memory: 512
1464 swap: 512
1465 parent: testsnaphot
1466 ...
1467
1468 [testsnaphot]
1469 memory: 512
1470 swap: 512
1471 snaptime: 1457170803
1472 ...
1473 ----
1474
1475 There are a few snapshot related properties like `parent` and
1476 `snaptime`. The `parent` property is used to store the parent/child
1477 relationship between snapshots. `snaptime` is the snapshot creation
1478 time stamp (Unix epoch).
1479
1480 You can optionally save the memory of a running VM with the option `vmstate`.
1481 For details about how the target storage gets chosen for the VM state, see
1482 xref:qm_vmstatestorage[State storage selection] in the chapter
1483 xref:qm_hibernate[Hibernation].
1484
1485 [[qm_options]]
1486 Options
1487 ~~~~~~~
1488
1489 include::qm.conf.5-opts.adoc[]
1490
1491
1492 Locks
1493 -----
1494
1495 Online migrations, snapshots and backups (`vzdump`) set a lock to
1496 prevent incompatible concurrent actions on the affected VMs. Sometimes
1497 you need to remove such a lock manually (e.g., after a power failure).
1498
1499 qm unlock <vmid>
1500
1501 CAUTION: Only do that if you are sure the action which set the lock is
1502 no longer running.
1503
1504
1505 ifdef::wiki[]
1506
1507 See Also
1508 ~~~~~~~~
1509
1510 * link:/wiki/Cloud-Init_Support[Cloud-Init Support]
1511
1512 endif::wiki[]
1513
1514
1515 ifdef::manvolnum[]
1516
1517 Files
1518 ------
1519
1520 `/etc/pve/qemu-server/<VMID>.conf`::
1521
1522 Configuration file for the VM '<VMID>'.
1523
1524
1525 include::pve-copyright.adoc[]
1526 endif::manvolnum[]