]>
Commit | Line | Data |
---|---|---|
80c0adcb | 1 | [[chapter_virtual_machines]] |
f69cfd23 | 2 | ifdef::manvolnum[] |
b2f242ab DM |
3 | qm(1) |
4 | ===== | |
5f09af76 DM |
5 | :pve-toplevel: |
6 | ||
f69cfd23 DM |
7 | NAME |
8 | ---- | |
9 | ||
10 | qm - Qemu/KVM Virtual Machine Manager | |
11 | ||
12 | ||
49a5e11c | 13 | SYNOPSIS |
f69cfd23 DM |
14 | -------- |
15 | ||
16 | include::qm.1-synopsis.adoc[] | |
17 | ||
18 | DESCRIPTION | |
19 | ----------- | |
20 | endif::manvolnum[] | |
f69cfd23 DM |
21 | ifndef::manvolnum[] |
22 | Qemu/KVM Virtual Machines | |
23 | ========================= | |
5f09af76 | 24 | :pve-toplevel: |
194d2f29 | 25 | endif::manvolnum[] |
5f09af76 | 26 | |
c4cba5d7 EK |
27 | // deprecates |
28 | // http://pve.proxmox.com/wiki/Container_and_Full_Virtualization | |
29 | // http://pve.proxmox.com/wiki/KVM | |
30 | // http://pve.proxmox.com/wiki/Qemu_Server | |
31 | ||
5eba0743 | 32 | Qemu (short form for Quick Emulator) is an open source hypervisor that emulates a |
c4cba5d7 EK |
33 | physical computer. From the perspective of the host system where Qemu is |
34 | running, Qemu is a user program which has access to a number of local resources | |
35 | like partitions, files, network cards which are then passed to an | |
189d3661 | 36 | emulated computer which sees them as if they were real devices. |
c4cba5d7 EK |
37 | |
38 | A guest operating system running in the emulated computer accesses these | |
39 | devices, and runs as it were running on real hardware. For instance you can pass | |
40 | an iso image as a parameter to Qemu, and the OS running in the emulated computer | |
189d3661 | 41 | will see a real CDROM inserted in a CD drive. |
c4cba5d7 | 42 | |
6fb50457 | 43 | Qemu can emulate a great variety of hardware from ARM to Sparc, but {pve} is |
c4cba5d7 EK |
44 | only concerned with 32 and 64 bits PC clone emulation, since it represents the |
45 | overwhelming majority of server hardware. The emulation of PC clones is also one | |
46 | of the fastest due to the availability of processor extensions which greatly | |
47 | speed up Qemu when the emulated architecture is the same as the host | |
9c63b5d9 EK |
48 | architecture. |
49 | ||
50 | NOTE: You may sometimes encounter the term _KVM_ (Kernel-based Virtual Machine). | |
51 | It means that Qemu is running with the support of the virtualization processor | |
52 | extensions, via the Linux kvm module. In the context of {pve} _Qemu_ and | |
6fb50457 | 53 | _KVM_ can be used interchangeably as Qemu in {pve} will always try to load the kvm |
9c63b5d9 EK |
54 | module. |
55 | ||
c4cba5d7 EK |
56 | Qemu inside {pve} runs as a root process, since this is required to access block |
57 | and PCI devices. | |
58 | ||
5eba0743 | 59 | |
c4cba5d7 EK |
60 | Emulated devices and paravirtualized devices |
61 | -------------------------------------------- | |
62 | ||
189d3661 DC |
63 | The PC hardware emulated by Qemu includes a mainboard, network controllers, |
64 | scsi, ide and sata controllers, serial ports (the complete list can be seen in | |
65 | the `kvm(1)` man page) all of them emulated in software. All these devices | |
66 | are the exact software equivalent of existing hardware devices, and if the OS | |
67 | running in the guest has the proper drivers it will use the devices as if it | |
c4cba5d7 EK |
68 | were running on real hardware. This allows Qemu to runs _unmodified_ operating |
69 | systems. | |
70 | ||
71 | This however has a performance cost, as running in software what was meant to | |
72 | run in hardware involves a lot of extra work for the host CPU. To mitigate this, | |
73 | Qemu can present to the guest operating system _paravirtualized devices_, where | |
74 | the guest OS recognizes it is running inside Qemu and cooperates with the | |
75 | hypervisor. | |
76 | ||
470d4313 | 77 | Qemu relies on the virtio virtualization standard, and is thus able to present |
189d3661 DC |
78 | paravirtualized virtio devices, which includes a paravirtualized generic disk |
79 | controller, a paravirtualized network card, a paravirtualized serial port, | |
c4cba5d7 EK |
80 | a paravirtualized SCSI controller, etc ... |
81 | ||
189d3661 DC |
82 | It is highly recommended to use the virtio devices whenever you can, as they |
83 | provide a big performance improvement. Using the virtio generic disk controller | |
84 | versus an emulated IDE controller will double the sequential write throughput, | |
85 | as measured with `bonnie++(8)`. Using the virtio network interface can deliver | |
c4cba5d7 | 86 | up to three times the throughput of an emulated Intel E1000 network card, as |
189d3661 | 87 | measured with `iperf(1)`. footnote:[See this benchmark on the KVM wiki |
c4cba5d7 EK |
88 | http://www.linux-kvm.org/page/Using_VirtIO_NIC] |
89 | ||
5eba0743 | 90 | |
80c0adcb | 91 | [[qm_virtual_machines_settings]] |
5274ad28 | 92 | Virtual Machines Settings |
c4cba5d7 | 93 | ------------------------- |
80c0adcb | 94 | |
c4cba5d7 EK |
95 | Generally speaking {pve} tries to choose sane defaults for virtual machines |
96 | (VM). Make sure you understand the meaning of the settings you change, as it | |
97 | could incur a performance slowdown, or putting your data at risk. | |
98 | ||
5eba0743 | 99 | |
80c0adcb | 100 | [[qm_general_settings]] |
c4cba5d7 EK |
101 | General Settings |
102 | ~~~~~~~~~~~~~~~~ | |
80c0adcb | 103 | |
1ff5e4e8 | 104 | [thumbnail="screenshot/gui-create-vm-general.png"] |
b16d767f | 105 | |
c4cba5d7 EK |
106 | General settings of a VM include |
107 | ||
108 | * the *Node* : the physical server on which the VM will run | |
109 | * the *VM ID*: a unique number in this {pve} installation used to identify your VM | |
110 | * *Name*: a free form text string you can use to describe the VM | |
111 | * *Resource Pool*: a logical group of VMs | |
112 | ||
5eba0743 | 113 | |
80c0adcb | 114 | [[qm_os_settings]] |
c4cba5d7 EK |
115 | OS Settings |
116 | ~~~~~~~~~~~ | |
80c0adcb | 117 | |
1ff5e4e8 | 118 | [thumbnail="screenshot/gui-create-vm-os.png"] |
200114a7 | 119 | |
c4cba5d7 EK |
120 | When creating a VM, setting the proper Operating System(OS) allows {pve} to |
121 | optimize some low level parameters. For instance Windows OS expect the BIOS | |
122 | clock to use the local time, while Unix based OS expect the BIOS clock to have | |
123 | the UTC time. | |
124 | ||
5eba0743 | 125 | |
80c0adcb | 126 | [[qm_hard_disk]] |
c4cba5d7 EK |
127 | Hard Disk |
128 | ~~~~~~~~~ | |
80c0adcb | 129 | |
2ec49380 | 130 | Qemu can emulate a number of storage controllers: |
c4cba5d7 EK |
131 | |
132 | * the *IDE* controller, has a design which goes back to the 1984 PC/AT disk | |
44f38275 | 133 | controller. Even if this controller has been superseded by recent designs, |
6fb50457 | 134 | each and every OS you can think of has support for it, making it a great choice |
c4cba5d7 EK |
135 | if you want to run an OS released before 2003. You can connect up to 4 devices |
136 | on this controller. | |
137 | ||
138 | * the *SATA* (Serial ATA) controller, dating from 2003, has a more modern | |
139 | design, allowing higher throughput and a greater number of devices to be | |
140 | connected. You can connect up to 6 devices on this controller. | |
141 | ||
b0b6802b EK |
142 | * the *SCSI* controller, designed in 1985, is commonly found on server grade |
143 | hardware, and can connect up to 14 storage devices. {pve} emulates by default a | |
f4bfd701 DM |
144 | LSI 53C895A controller. |
145 | + | |
81868c7e | 146 | A SCSI controller of type _VirtIO SCSI_ is the recommended setting if you aim for |
b0b6802b EK |
147 | performance and is automatically selected for newly created Linux VMs since |
148 | {pve} 4.3. Linux distributions have support for this controller since 2012, and | |
c4cba5d7 | 149 | FreeBSD since 2014. For Windows OSes, you need to provide an extra iso |
b0b6802b EK |
150 | containing the drivers during the installation. |
151 | // https://pve.proxmox.com/wiki/Paravirtualized_Block_Drivers_for_Windows#During_windows_installation. | |
81868c7e EK |
152 | If you aim at maximum performance, you can select a SCSI controller of type |
153 | _VirtIO SCSI single_ which will allow you to select the *IO Thread* option. | |
154 | When selecting _VirtIO SCSI single_ Qemu will create a new controller for | |
155 | each disk, instead of adding all disks to the same controller. | |
b0b6802b | 156 | |
30e6fe00 TL |
157 | * The *VirtIO Block* controller, often just called VirtIO or virtio-blk, |
158 | is an older type of paravirtualized controller. It has been superseded by the | |
159 | VirtIO SCSI Controller, in terms of features. | |
c4cba5d7 | 160 | |
1ff5e4e8 | 161 | [thumbnail="screenshot/gui-create-vm-hard-disk.png"] |
c4cba5d7 EK |
162 | On each controller you attach a number of emulated hard disks, which are backed |
163 | by a file or a block device residing in the configured storage. The choice of | |
164 | a storage type will determine the format of the hard disk image. Storages which | |
165 | present block devices (LVM, ZFS, Ceph) will require the *raw disk image format*, | |
de14ebff | 166 | whereas files based storages (Ext4, NFS, CIFS, GlusterFS) will let you to choose |
c4cba5d7 EK |
167 | either the *raw disk image format* or the *QEMU image format*. |
168 | ||
169 | * the *QEMU image format* is a copy on write format which allows snapshots, and | |
170 | thin provisioning of the disk image. | |
189d3661 DC |
171 | * the *raw disk image* is a bit-to-bit image of a hard disk, similar to what |
172 | you would get when executing the `dd` command on a block device in Linux. This | |
4371b2fe | 173 | format does not support thin provisioning or snapshots by itself, requiring |
30e6fe00 TL |
174 | cooperation from the storage layer for these tasks. It may, however, be up to |
175 | 10% faster than the *QEMU image format*. footnote:[See this benchmark for details | |
c4cba5d7 | 176 | http://events.linuxfoundation.org/sites/events/files/slides/CloudOpen2013_Khoa_Huynh_v3.pdf] |
189d3661 | 177 | * the *VMware image format* only makes sense if you intend to import/export the |
c4cba5d7 EK |
178 | disk image to other hypervisors. |
179 | ||
180 | Setting the *Cache* mode of the hard drive will impact how the host system will | |
181 | notify the guest systems of block write completions. The *No cache* default | |
182 | means that the guest system will be notified that a write is complete when each | |
183 | block reaches the physical storage write queue, ignoring the host page cache. | |
184 | This provides a good balance between safety and speed. | |
185 | ||
186 | If you want the {pve} backup manager to skip a disk when doing a backup of a VM, | |
187 | you can set the *No backup* option on that disk. | |
188 | ||
3205ac49 EK |
189 | If you want the {pve} storage replication mechanism to skip a disk when starting |
190 | a replication job, you can set the *Skip replication* option on that disk. | |
6fb50457 | 191 | As of {pve} 5.0, replication requires the disk images to be on a storage of type |
3205ac49 | 192 | `zfspool`, so adding a disk image to other storages when the VM has replication |
6fb50457 | 193 | configured requires to skip replication for this disk image. |
3205ac49 | 194 | |
c4cba5d7 | 195 | If your storage supports _thin provisioning_ (see the storage chapter in the |
53cbac40 NC |
196 | {pve} guide), you can activate the *Discard* option on a drive. With *Discard* |
197 | set and a _TRIM_-enabled guest OS footnote:[TRIM, UNMAP, and discard | |
198 | https://en.wikipedia.org/wiki/Trim_%28computing%29], when the VM's filesystem | |
199 | marks blocks as unused after deleting files, the controller will relay this | |
200 | information to the storage, which will then shrink the disk image accordingly. | |
201 | For the guest to be able to issue _TRIM_ commands, you must either use a | |
202 | *VirtIO SCSI* (or *VirtIO SCSI Single*) controller or set the *SSD emulation* | |
203 | option on the drive. Note that *Discard* is not supported on *VirtIO Block* | |
204 | drives. | |
c4cba5d7 | 205 | |
25203dc1 NC |
206 | If you would like a drive to be presented to the guest as a solid-state drive |
207 | rather than a rotational hard disk, you can set the *SSD emulation* option on | |
208 | that drive. There is no requirement that the underlying storage actually be | |
209 | backed by SSDs; this feature can be used with physical media of any type. | |
53cbac40 | 210 | Note that *SSD emulation* is not supported on *VirtIO Block* drives. |
25203dc1 | 211 | |
af9c6de1 | 212 | .IO Thread |
59552707 | 213 | The option *IO Thread* can only be used when using a disk with the |
81868c7e EK |
214 | *VirtIO* controller, or with the *SCSI* controller, when the emulated controller |
215 | type is *VirtIO SCSI single*. | |
216 | With this enabled, Qemu creates one I/O thread per storage controller, | |
59552707 | 217 | instead of a single thread for all I/O, so it increases performance when |
81868c7e | 218 | multiple disks are used and each disk has its own storage controller. |
c564fc52 DC |
219 | Note that backups do not currently work with *IO Thread* enabled. |
220 | ||
80c0adcb DM |
221 | |
222 | [[qm_cpu]] | |
34e541c5 EK |
223 | CPU |
224 | ~~~ | |
80c0adcb | 225 | |
1ff5e4e8 | 226 | [thumbnail="screenshot/gui-create-vm-cpu.png"] |
397c74c3 | 227 | |
34e541c5 EK |
228 | A *CPU socket* is a physical slot on a PC motherboard where you can plug a CPU. |
229 | This CPU can then contain one or many *cores*, which are independent | |
230 | processing units. Whether you have a single CPU socket with 4 cores, or two CPU | |
231 | sockets with two cores is mostly irrelevant from a performance point of view. | |
44f38275 TL |
232 | However some software licenses depend on the number of sockets a machine has, |
233 | in that case it makes sense to set the number of sockets to what the license | |
234 | allows you. | |
f4bfd701 | 235 | |
34e541c5 EK |
236 | Increasing the number of virtual cpus (cores and sockets) will usually provide a |
237 | performance improvement though that is heavily dependent on the use of the VM. | |
238 | Multithreaded applications will of course benefit from a large number of | |
239 | virtual cpus, as for each virtual cpu you add, Qemu will create a new thread of | |
240 | execution on the host system. If you're not sure about the workload of your VM, | |
241 | it is usually a safe bet to set the number of *Total cores* to 2. | |
242 | ||
fb29acdd | 243 | NOTE: It is perfectly safe if the _overall_ number of cores of all your VMs |
7dd7a0b7 TL |
244 | is greater than the number of cores on the server (e.g., 4 VMs with each 4 |
245 | cores on a machine with only 8 cores). In that case the host system will | |
246 | balance the Qemu execution threads between your server cores, just like if you | |
247 | were running a standard multithreaded application. However, {pve} will prevent | |
fb29acdd | 248 | you from assigning more virtual CPU cores than physically available, as this will |
7dd7a0b7 | 249 | only bring the performance down due to the cost of context switches. |
34e541c5 | 250 | |
af54f54d TL |
251 | [[qm_cpu_resource_limits]] |
252 | Resource Limits | |
253 | ^^^^^^^^^^^^^^^ | |
254 | ||
4371b2fe | 255 | In addition to the number of virtual cores, you can configure how much resources |
af54f54d TL |
256 | a VM can get in relation to the host CPU time and also in relation to other |
257 | VMs. | |
046643ec FG |
258 | With the *cpulimit* (``Host CPU Time'') option you can limit how much CPU time |
259 | the whole VM can use on the host. It is a floating point value representing CPU | |
af54f54d | 260 | time in percent, so `1.0` is equal to `100%`, `2.5` to `250%` and so on. If a |
4371b2fe | 261 | single process would fully use one single core it would have `100%` CPU Time |
af54f54d TL |
262 | usage. If a VM with four cores utilizes all its cores fully it would |
263 | theoretically use `400%`. In reality the usage may be even a bit higher as Qemu | |
264 | can have additional threads for VM peripherals besides the vCPU core ones. | |
265 | This setting can be useful if a VM should have multiple vCPUs, as it runs a few | |
266 | processes in parallel, but the VM as a whole should not be able to run all | |
267 | vCPUs at 100% at the same time. Using a specific example: lets say we have a VM | |
268 | which would profit from having 8 vCPUs, but at no time all of those 8 cores | |
269 | should run at full load - as this would make the server so overloaded that | |
270 | other VMs and CTs would get to less CPU. So, we set the *cpulimit* limit to | |
271 | `4.0` (=400%). If all cores do the same heavy work they would all get 50% of a | |
272 | real host cores CPU time. But, if only 4 would do work they could still get | |
273 | almost 100% of a real core each. | |
274 | ||
275 | NOTE: VMs can, depending on their configuration, use additional threads e.g., | |
276 | for networking or IO operations but also live migration. Thus a VM can show up | |
277 | to use more CPU time than just its virtual CPUs could use. To ensure that a VM | |
278 | never uses more CPU time than virtual CPUs assigned set the *cpulimit* setting | |
279 | to the same value as the total core count. | |
280 | ||
281 | The second CPU resource limiting setting, *cpuunits* (nowadays often called CPU | |
282 | shares or CPU weight), controls how much CPU time a VM gets in regards to other | |
283 | VMs running. It is a relative weight which defaults to `1024`, if you increase | |
284 | this for a VM it will be prioritized by the scheduler in comparison to other | |
285 | VMs with lower weight. E.g., if VM 100 has set the default 1024 and VM 200 was | |
286 | changed to `2048`, the latter VM 200 would receive twice the CPU bandwidth than | |
287 | the first VM 100. | |
288 | ||
289 | For more information see `man systemd.resource-control`, here `CPUQuota` | |
290 | corresponds to `cpulimit` and `CPUShares` corresponds to our `cpuunits` | |
291 | setting, visit its Notes section for references and implementation details. | |
292 | ||
293 | CPU Type | |
294 | ^^^^^^^^ | |
295 | ||
34e541c5 EK |
296 | Qemu can emulate a number different of *CPU types* from 486 to the latest Xeon |
297 | processors. Each new processor generation adds new features, like hardware | |
298 | assisted 3d rendering, random number generation, memory protection, etc ... | |
299 | Usually you should select for your VM a processor type which closely matches the | |
300 | CPU of the host system, as it means that the host CPU features (also called _CPU | |
301 | flags_ ) will be available in your VMs. If you want an exact match, you can set | |
302 | the CPU type to *host* in which case the VM will have exactly the same CPU flags | |
f4bfd701 DM |
303 | as your host system. |
304 | ||
34e541c5 EK |
305 | This has a downside though. If you want to do a live migration of VMs between |
306 | different hosts, your VM might end up on a new system with a different CPU type. | |
307 | If the CPU flags passed to the guest are missing, the qemu process will stop. To | |
308 | remedy this Qemu has also its own CPU type *kvm64*, that {pve} uses by defaults. | |
309 | kvm64 is a Pentium 4 look a like CPU type, which has a reduced CPU flags set, | |
f4bfd701 DM |
310 | but is guaranteed to work everywhere. |
311 | ||
312 | In short, if you care about live migration and moving VMs between nodes, leave | |
af54f54d TL |
313 | the kvm64 default. If you don’t care about live migration or have a homogeneous |
314 | cluster where all nodes have the same CPU, set the CPU type to host, as in | |
315 | theory this will give your guests maximum performance. | |
316 | ||
72ae8aa2 FG |
317 | Meltdown / Spectre related CPU flags |
318 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
319 | ||
2975cb7a | 320 | There are several CPU flags related to the Meltdown and Spectre vulnerabilities |
72ae8aa2 FG |
321 | footnote:[Meltdown Attack https://meltdownattack.com/] which need to be set |
322 | manually unless the selected CPU type of your VM already enables them by default. | |
323 | ||
2975cb7a | 324 | There are two requirements that need to be fulfilled in order to use these |
72ae8aa2 | 325 | CPU flags: |
5dba2677 | 326 | |
72ae8aa2 FG |
327 | * The host CPU(s) must support the feature and propagate it to the guest's virtual CPU(s) |
328 | * The guest operating system must be updated to a version which mitigates the | |
329 | attacks and is able to utilize the CPU feature | |
330 | ||
2975cb7a AD |
331 | Otherwise you need to set the desired CPU flag of the virtual CPU, either by |
332 | editing the CPU options in the WebUI, or by setting the 'flags' property of the | |
333 | 'cpu' option in the VM configuration file. | |
334 | ||
335 | For Spectre v1,v2,v4 fixes, your CPU or system vendor also needs to provide a | |
72ae8aa2 FG |
336 | so-called ``microcode update'' footnote:[You can use `intel-microcode' / |
337 | `amd-microcode' from Debian non-free if your vendor does not provide such an | |
338 | update. Note that not all affected CPUs can be updated to support spec-ctrl.] | |
339 | for your CPU. | |
5dba2677 | 340 | |
2975cb7a AD |
341 | |
342 | To check if the {pve} host is vulnerable, execute the following command as root: | |
5dba2677 TL |
343 | |
344 | ---- | |
2975cb7a | 345 | for f in /sys/devices/system/cpu/vulnerabilities/*; do echo "${f##*/} -" $(cat "$f"); done |
5dba2677 TL |
346 | ---- |
347 | ||
144d5ede | 348 | A community script is also available to detect is the host is still vulnerable. |
2975cb7a | 349 | footnote:[spectre-meltdown-checker https://meltdown.ovh/] |
72ae8aa2 | 350 | |
2975cb7a AD |
351 | Intel processors |
352 | ^^^^^^^^^^^^^^^^ | |
72ae8aa2 | 353 | |
2975cb7a AD |
354 | * 'pcid' |
355 | + | |
144d5ede | 356 | This reduces the performance impact of the Meltdown (CVE-2017-5754) mitigation |
2975cb7a AD |
357 | called 'Kernel Page-Table Isolation (KPTI)', which effectively hides |
358 | the Kernel memory from the user space. Without PCID, KPTI is quite an expensive | |
359 | mechanism footnote:[PCID is now a critical performance/security feature on x86 | |
360 | https://groups.google.com/forum/m/#!topic/mechanical-sympathy/L9mHTbeQLNU]. | |
361 | + | |
362 | To check if the {pve} host supports PCID, execute the following command as root: | |
363 | + | |
72ae8aa2 | 364 | ---- |
2975cb7a | 365 | # grep ' pcid ' /proc/cpuinfo |
72ae8aa2 | 366 | ---- |
2975cb7a AD |
367 | + |
368 | If this does not return empty your host's CPU has support for 'pcid'. | |
72ae8aa2 | 369 | |
2975cb7a AD |
370 | * 'spec-ctrl' |
371 | + | |
144d5ede WB |
372 | Required to enable the Spectre v1 (CVE-2017-5753) and Spectre v2 (CVE-2017-5715) fix, |
373 | in cases where retpolines are not sufficient. | |
374 | Included by default in Intel CPU models with -IBRS suffix. | |
375 | Must be explicitly turned on for Intel CPU models without -IBRS suffix. | |
376 | Requires an updated host CPU microcode (intel-microcode >= 20180425). | |
2975cb7a AD |
377 | + |
378 | * 'ssbd' | |
379 | + | |
144d5ede WB |
380 | Required to enable the Spectre V4 (CVE-2018-3639) fix. Not included by default in any Intel CPU model. |
381 | Must be explicitly turned on for all Intel CPU models. | |
382 | Requires an updated host CPU microcode(intel-microcode >= 20180703). | |
72ae8aa2 | 383 | |
72ae8aa2 | 384 | |
2975cb7a AD |
385 | AMD processors |
386 | ^^^^^^^^^^^^^^ | |
387 | ||
388 | * 'ibpb' | |
389 | + | |
144d5ede WB |
390 | Required to enable the Spectre v1 (CVE-2017-5753) and Spectre v2 (CVE-2017-5715) fix, |
391 | in cases where retpolines are not sufficient. | |
392 | Included by default in AMD CPU models with -IBPB suffix. | |
393 | Must be explicitly turned on for AMD CPU models without -IBPB suffix. | |
2975cb7a AD |
394 | Requires the host CPU microcode to support this feature before it can be used for guest CPUs. |
395 | ||
396 | ||
397 | ||
398 | * 'virt-ssbd' | |
399 | + | |
400 | Required to enable the Spectre v4 (CVE-2018-3639) fix. | |
144d5ede WB |
401 | Not included by default in any AMD CPU model. |
402 | Must be explicitly turned on for all AMD CPU models. | |
403 | This should be provided to guests, even if amd-ssbd is also provided, for maximum guest compatibility. | |
404 | Note that this must be explicitly enabled when when using the "host" cpu model, | |
405 | because this is a virtual feature which does not exist in the physical CPUs. | |
2975cb7a AD |
406 | |
407 | ||
408 | * 'amd-ssbd' | |
409 | + | |
144d5ede WB |
410 | Required to enable the Spectre v4 (CVE-2018-3639) fix. |
411 | Not included by default in any AMD CPU model. Must be explicitly turned on for all AMD CPU models. | |
412 | This provides higher performance than virt-ssbd, therefore a host supporting this should always expose this to guests if possible. | |
2975cb7a AD |
413 | virt-ssbd should none the less also be exposed for maximum guest compatibility as some kernels only know about virt-ssbd. |
414 | ||
415 | ||
416 | * 'amd-no-ssb' | |
417 | + | |
418 | Recommended to indicate the host is not vulnerable to Spectre V4 (CVE-2018-3639). | |
144d5ede WB |
419 | Not included by default in any AMD CPU model. |
420 | Future hardware generations of CPU will not be vulnerable to CVE-2018-3639, | |
421 | and thus the guest should be told not to enable its mitigations, by exposing amd-no-ssb. | |
2975cb7a AD |
422 | This is mutually exclusive with virt-ssbd and amd-ssbd. |
423 | ||
5dba2677 | 424 | |
af54f54d TL |
425 | NUMA |
426 | ^^^^ | |
427 | You can also optionally emulate a *NUMA* | |
428 | footnote:[https://en.wikipedia.org/wiki/Non-uniform_memory_access] architecture | |
429 | in your VMs. The basics of the NUMA architecture mean that instead of having a | |
430 | global memory pool available to all your cores, the memory is spread into local | |
431 | banks close to each socket. | |
34e541c5 EK |
432 | This can bring speed improvements as the memory bus is not a bottleneck |
433 | anymore. If your system has a NUMA architecture footnote:[if the command | |
434 | `numactl --hardware | grep available` returns more than one node, then your host | |
435 | system has a NUMA architecture] we recommend to activate the option, as this | |
af54f54d TL |
436 | will allow proper distribution of the VM resources on the host system. |
437 | This option is also required to hot-plug cores or RAM in a VM. | |
34e541c5 EK |
438 | |
439 | If the NUMA option is used, it is recommended to set the number of sockets to | |
4ccb911c | 440 | the number of nodes of the host system. |
34e541c5 | 441 | |
af54f54d TL |
442 | vCPU hot-plug |
443 | ^^^^^^^^^^^^^ | |
444 | ||
445 | Modern operating systems introduced the capability to hot-plug and, to a | |
4371b2fe FG |
446 | certain extent, hot-unplug CPUs in a running systems. Virtualisation allows us |
447 | to avoid a lot of the (physical) problems real hardware can cause in such | |
448 | scenarios. | |
449 | Still, this is a rather new and complicated feature, so its use should be | |
450 | restricted to cases where its absolutely needed. Most of the functionality can | |
451 | be replicated with other, well tested and less complicated, features, see | |
af54f54d TL |
452 | xref:qm_cpu_resource_limits[Resource Limits]. |
453 | ||
454 | In {pve} the maximal number of plugged CPUs is always `cores * sockets`. | |
455 | To start a VM with less than this total core count of CPUs you may use the | |
4371b2fe | 456 | *vpus* setting, it denotes how many vCPUs should be plugged in at VM start. |
af54f54d | 457 | |
4371b2fe | 458 | Currently only this feature is only supported on Linux, a kernel newer than 3.10 |
af54f54d TL |
459 | is needed, a kernel newer than 4.7 is recommended. |
460 | ||
461 | You can use a udev rule as follow to automatically set new CPUs as online in | |
462 | the guest: | |
463 | ||
464 | ---- | |
465 | SUBSYSTEM=="cpu", ACTION=="add", TEST=="online", ATTR{online}=="0", ATTR{online}="1" | |
466 | ---- | |
467 | ||
468 | Save this under /etc/udev/rules.d/ as a file ending in `.rules`. | |
469 | ||
470 | Note: CPU hot-remove is machine dependent and requires guest cooperation. | |
471 | The deletion command does not guarantee CPU removal to actually happen, | |
472 | typically it's a request forwarded to guest using target dependent mechanism, | |
473 | e.g., ACPI on x86/amd64. | |
474 | ||
80c0adcb DM |
475 | |
476 | [[qm_memory]] | |
34e541c5 EK |
477 | Memory |
478 | ~~~~~~ | |
80c0adcb | 479 | |
34e541c5 EK |
480 | For each VM you have the option to set a fixed size memory or asking |
481 | {pve} to dynamically allocate memory based on the current RAM usage of the | |
59552707 | 482 | host. |
34e541c5 | 483 | |
96124d0f | 484 | .Fixed Memory Allocation |
1ff5e4e8 | 485 | [thumbnail="screenshot/gui-create-vm-memory.png"] |
96124d0f | 486 | |
9fb002e6 DC |
487 | When setting memory and minimum memory to the same amount |
488 | {pve} will simply allocate what you specify to your VM. | |
34e541c5 | 489 | |
9abfec65 DC |
490 | Even when using a fixed memory size, the ballooning device gets added to the |
491 | VM, because it delivers useful information such as how much memory the guest | |
492 | really uses. | |
493 | In general, you should leave *ballooning* enabled, but if you want to disable | |
e60ce90c | 494 | it (e.g. for debugging purposes), simply uncheck |
9fb002e6 | 495 | *Ballooning Device* or set |
9abfec65 DC |
496 | |
497 | balloon: 0 | |
498 | ||
499 | in the configuration. | |
500 | ||
96124d0f | 501 | .Automatic Memory Allocation |
96124d0f | 502 | |
34e541c5 | 503 | // see autoballoon() in pvestatd.pm |
58e04593 | 504 | When setting the minimum memory lower than memory, {pve} will make sure that the |
34e541c5 EK |
505 | minimum amount you specified is always available to the VM, and if RAM usage on |
506 | the host is below 80%, will dynamically add memory to the guest up to the | |
f4bfd701 DM |
507 | maximum memory specified. |
508 | ||
a35aad4a | 509 | When the host is running low on RAM, the VM will then release some memory |
34e541c5 EK |
510 | back to the host, swapping running processes if needed and starting the oom |
511 | killer in last resort. The passing around of memory between host and guest is | |
512 | done via a special `balloon` kernel driver running inside the guest, which will | |
513 | grab or release memory pages from the host. | |
514 | footnote:[A good explanation of the inner workings of the balloon driver can be found here https://rwmj.wordpress.com/2010/07/17/virtio-balloon/] | |
515 | ||
c9f6e1a4 EK |
516 | When multiple VMs use the autoallocate facility, it is possible to set a |
517 | *Shares* coefficient which indicates the relative amount of the free host memory | |
470d4313 | 518 | that each VM should take. Suppose for instance you have four VMs, three of them |
a35aad4a | 519 | running an HTTP server and the last one is a database server. To cache more |
c9f6e1a4 EK |
520 | database blocks in the database server RAM, you would like to prioritize the |
521 | database VM when spare RAM is available. For this you assign a Shares property | |
522 | of 3000 to the database VM, leaving the other VMs to the Shares default setting | |
470d4313 | 523 | of 1000. The host server has 32GB of RAM, and is currently using 16GB, leaving 32 |
c9f6e1a4 EK |
524 | * 80/100 - 16 = 9GB RAM to be allocated to the VMs. The database VM will get 9 * |
525 | 3000 / (3000 + 1000 + 1000 + 1000) = 4.5 GB extra RAM and each HTTP server will | |
a35aad4a | 526 | get 1.5 GB. |
c9f6e1a4 | 527 | |
34e541c5 EK |
528 | All Linux distributions released after 2010 have the balloon kernel driver |
529 | included. For Windows OSes, the balloon driver needs to be added manually and can | |
530 | incur a slowdown of the guest, so we don't recommend using it on critical | |
59552707 | 531 | systems. |
34e541c5 EK |
532 | // see https://forum.proxmox.com/threads/solved-hyper-threading-vs-no-hyper-threading-fixed-vs-variable-memory.20265/ |
533 | ||
470d4313 | 534 | When allocating RAM to your VMs, a good rule of thumb is always to leave 1GB |
34e541c5 EK |
535 | of RAM available to the host. |
536 | ||
80c0adcb DM |
537 | |
538 | [[qm_network_device]] | |
1ff7835b EK |
539 | Network Device |
540 | ~~~~~~~~~~~~~~ | |
80c0adcb | 541 | |
1ff5e4e8 | 542 | [thumbnail="screenshot/gui-create-vm-network.png"] |
c24ddb0a | 543 | |
1ff7835b EK |
544 | Each VM can have many _Network interface controllers_ (NIC), of four different |
545 | types: | |
546 | ||
547 | * *Intel E1000* is the default, and emulates an Intel Gigabit network card. | |
548 | * the *VirtIO* paravirtualized NIC should be used if you aim for maximum | |
549 | performance. Like all VirtIO devices, the guest OS should have the proper driver | |
550 | installed. | |
551 | * the *Realtek 8139* emulates an older 100 MB/s network card, and should | |
59552707 | 552 | only be used when emulating older operating systems ( released before 2002 ) |
1ff7835b EK |
553 | * the *vmxnet3* is another paravirtualized device, which should only be used |
554 | when importing a VM from another hypervisor. | |
555 | ||
556 | {pve} will generate for each NIC a random *MAC address*, so that your VM is | |
557 | addressable on Ethernet networks. | |
558 | ||
470d4313 | 559 | The NIC you added to the VM can follow one of two different models: |
af9c6de1 EK |
560 | |
561 | * in the default *Bridged mode* each virtual NIC is backed on the host by a | |
562 | _tap device_, ( a software loopback device simulating an Ethernet NIC ). This | |
563 | tap device is added to a bridge, by default vmbr0 in {pve}. In this mode, VMs | |
564 | have direct access to the Ethernet LAN on which the host is located. | |
565 | * in the alternative *NAT mode*, each virtual NIC will only communicate with | |
470d4313 DC |
566 | the Qemu user networking stack, where a built-in router and DHCP server can |
567 | provide network access. This built-in DHCP will serve addresses in the private | |
af9c6de1 | 568 | 10.0.2.0/24 range. The NAT mode is much slower than the bridged mode, and |
f5041150 DC |
569 | should only be used for testing. This mode is only available via CLI or the API, |
570 | but not via the WebUI. | |
af9c6de1 EK |
571 | |
572 | You can also skip adding a network device when creating a VM by selecting *No | |
573 | network device*. | |
574 | ||
575 | .Multiqueue | |
1ff7835b | 576 | If you are using the VirtIO driver, you can optionally activate the |
af9c6de1 | 577 | *Multiqueue* option. This option allows the guest OS to process networking |
1ff7835b | 578 | packets using multiple virtual CPUs, providing an increase in the total number |
470d4313 | 579 | of packets transferred. |
1ff7835b EK |
580 | |
581 | //http://blog.vmsplice.net/2011/09/qemu-internals-vhost-architecture.html | |
582 | When using the VirtIO driver with {pve}, each NIC network queue is passed to the | |
a35aad4a | 583 | host kernel, where the queue will be processed by a kernel thread spawned by the |
1ff7835b EK |
584 | vhost driver. With this option activated, it is possible to pass _multiple_ |
585 | network queues to the host kernel for each NIC. | |
586 | ||
587 | //https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Tuning_and_Optimization_Guide/sect-Virtualization_Tuning_Optimization_Guide-Networking-Techniques.html#sect-Virtualization_Tuning_Optimization_Guide-Networking-Multi-queue_virtio-net | |
af9c6de1 | 588 | When using Multiqueue, it is recommended to set it to a value equal |
1ff7835b EK |
589 | to the number of Total Cores of your guest. You also need to set in |
590 | the VM the number of multi-purpose channels on each VirtIO NIC with the ethtool | |
59552707 | 591 | command: |
1ff7835b | 592 | |
7a0d4784 | 593 | `ethtool -L ens1 combined X` |
1ff7835b EK |
594 | |
595 | where X is the number of the number of vcpus of the VM. | |
596 | ||
af9c6de1 | 597 | You should note that setting the Multiqueue parameter to a value greater |
1ff7835b EK |
598 | than one will increase the CPU load on the host and guest systems as the |
599 | traffic increases. We recommend to set this option only when the VM has to | |
600 | process a great number of incoming connections, such as when the VM is running | |
601 | as a router, reverse proxy or a busy HTTP server doing long polling. | |
602 | ||
6cb67d7f DC |
603 | [[qm_display]] |
604 | Display | |
605 | ~~~~~~~ | |
606 | ||
607 | QEMU can virtualize a few types of VGA hardware. Some examples are: | |
608 | ||
609 | * *std*, the default, emulates a card with Bochs VBE extensions. | |
1368dc02 TL |
610 | * *cirrus*, this was once the default, it emulates a very old hardware module |
611 | with all its problems. This display type should only be used if really | |
612 | necessary footnote:[https://www.kraxel.org/blog/2014/10/qemu-using-cirrus-considered-harmful/ | |
613 | qemu: using cirrus considered harmful], e.g., if using Windows XP or earlier | |
6cb67d7f DC |
614 | * *vmware*, is a VMWare SVGA-II compatible adapter. |
615 | * *qxl*, is the QXL paravirtualized graphics card. Selecting this also | |
616 | enables SPICE for the VM. | |
617 | ||
618 | You can edit the amount of memory given to the virtual GPU, by setting | |
1368dc02 | 619 | the 'memory' option. This can enable higher resolutions inside the VM, |
6cb67d7f DC |
620 | especially with SPICE/QXL. |
621 | ||
1368dc02 TL |
622 | As the memory is reserved by display device, selecting Multi-Monitor mode |
623 | for SPICE (e.g., `qxl2` for dual monitors) has some implications: | |
6cb67d7f | 624 | |
1368dc02 TL |
625 | * Windows needs a device for each monitor, so if your 'ostype' is some |
626 | version of Windows, {pve} gives the VM an extra device per monitor. | |
6cb67d7f | 627 | Each device gets the specified amount of memory. |
1368dc02 | 628 | |
6cb67d7f DC |
629 | * Linux VMs, can always enable more virtual monitors, but selecting |
630 | a Multi-Monitor mode multiplies the memory given to the device with | |
631 | the number of monitors. | |
632 | ||
1368dc02 TL |
633 | Selecting `serialX` as display 'type' disables the VGA output, and redirects |
634 | the Web Console to the selected serial port. A configured display 'memory' | |
635 | setting will be ignored in that case. | |
80c0adcb | 636 | |
dbb44ef0 | 637 | [[qm_usb_passthrough]] |
685cc8e0 DC |
638 | USB Passthrough |
639 | ~~~~~~~~~~~~~~~ | |
80c0adcb | 640 | |
685cc8e0 DC |
641 | There are two different types of USB passthrough devices: |
642 | ||
470d4313 | 643 | * Host USB passthrough |
685cc8e0 DC |
644 | * SPICE USB passthrough |
645 | ||
646 | Host USB passthrough works by giving a VM a USB device of the host. | |
647 | This can either be done via the vendor- and product-id, or | |
648 | via the host bus and port. | |
649 | ||
650 | The vendor/product-id looks like this: *0123:abcd*, | |
651 | where *0123* is the id of the vendor, and *abcd* is the id | |
652 | of the product, meaning two pieces of the same usb device | |
653 | have the same id. | |
654 | ||
655 | The bus/port looks like this: *1-2.3.4*, where *1* is the bus | |
656 | and *2.3.4* is the port path. This represents the physical | |
657 | ports of your host (depending of the internal order of the | |
658 | usb controllers). | |
659 | ||
660 | If a device is present in a VM configuration when the VM starts up, | |
661 | but the device is not present in the host, the VM can boot without problems. | |
470d4313 | 662 | As soon as the device/port is available in the host, it gets passed through. |
685cc8e0 | 663 | |
e60ce90c | 664 | WARNING: Using this kind of USB passthrough means that you cannot move |
685cc8e0 DC |
665 | a VM online to another host, since the hardware is only available |
666 | on the host the VM is currently residing. | |
667 | ||
668 | The second type of passthrough is SPICE USB passthrough. This is useful | |
669 | if you use a SPICE client which supports it. If you add a SPICE USB port | |
670 | to your VM, you can passthrough a USB device from where your SPICE client is, | |
671 | directly to the VM (for example an input device or hardware dongle). | |
672 | ||
80c0adcb DM |
673 | |
674 | [[qm_bios_and_uefi]] | |
076d60ae DC |
675 | BIOS and UEFI |
676 | ~~~~~~~~~~~~~ | |
677 | ||
678 | In order to properly emulate a computer, QEMU needs to use a firmware. | |
679 | By default QEMU uses *SeaBIOS* for this, which is an open-source, x86 BIOS | |
680 | implementation. SeaBIOS is a good choice for most standard setups. | |
681 | ||
682 | There are, however, some scenarios in which a BIOS is not a good firmware | |
683 | to boot from, e.g. if you want to do VGA passthrough. footnote:[Alex Williamson has a very good blog entry about this. | |
684 | http://vfio.blogspot.co.at/2014/08/primary-graphics-assignment-without-vga.html] | |
470d4313 | 685 | In such cases, you should rather use *OVMF*, which is an open-source UEFI implementation. footnote:[See the OVMF Project http://www.tianocore.org/ovmf/] |
076d60ae DC |
686 | |
687 | If you want to use OVMF, there are several things to consider: | |
688 | ||
689 | In order to save things like the *boot order*, there needs to be an EFI Disk. | |
690 | This disk will be included in backups and snapshots, and there can only be one. | |
691 | ||
692 | You can create such a disk with the following command: | |
693 | ||
694 | qm set <vmid> -efidisk0 <storage>:1,format=<format> | |
695 | ||
696 | Where *<storage>* is the storage where you want to have the disk, and | |
697 | *<format>* is a format which the storage supports. Alternatively, you can | |
698 | create such a disk through the web interface with 'Add' -> 'EFI Disk' in the | |
699 | hardware section of a VM. | |
700 | ||
701 | When using OVMF with a virtual display (without VGA passthrough), | |
702 | you need to set the client resolution in the OVMF menu(which you can reach | |
703 | with a press of the ESC button during boot), or you have to choose | |
704 | SPICE as the display type. | |
705 | ||
288e3f46 EK |
706 | [[qm_startup_and_shutdown]] |
707 | Automatic Start and Shutdown of Virtual Machines | |
708 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
709 | ||
710 | After creating your VMs, you probably want them to start automatically | |
711 | when the host system boots. For this you need to select the option 'Start at | |
712 | boot' from the 'Options' Tab of your VM in the web interface, or set it with | |
713 | the following command: | |
714 | ||
715 | qm set <vmid> -onboot 1 | |
716 | ||
4dbeb548 DM |
717 | .Start and Shutdown Order |
718 | ||
1ff5e4e8 | 719 | [thumbnail="screenshot/gui-qemu-edit-start-order.png"] |
4dbeb548 DM |
720 | |
721 | In some case you want to be able to fine tune the boot order of your | |
722 | VMs, for instance if one of your VM is providing firewalling or DHCP | |
723 | to other guest systems. For this you can use the following | |
724 | parameters: | |
288e3f46 EK |
725 | |
726 | * *Start/Shutdown order*: Defines the start order priority. E.g. set it to 1 if | |
727 | you want the VM to be the first to be started. (We use the reverse startup | |
728 | order for shutdown, so a machine with a start order of 1 would be the last to | |
7eed72d8 | 729 | be shut down). If multiple VMs have the same order defined on a host, they will |
d750c851 | 730 | additionally be ordered by 'VMID' in ascending order. |
288e3f46 EK |
731 | * *Startup delay*: Defines the interval between this VM start and subsequent |
732 | VMs starts . E.g. set it to 240 if you want to wait 240 seconds before starting | |
733 | other VMs. | |
734 | * *Shutdown timeout*: Defines the duration in seconds {pve} should wait | |
735 | for the VM to be offline after issuing a shutdown command. | |
7eed72d8 | 736 | By default this value is set to 180, which means that {pve} will issue a |
d750c851 WB |
737 | shutdown request and wait 180 seconds for the machine to be offline. If |
738 | the machine is still online after the timeout it will be stopped forcefully. | |
288e3f46 | 739 | |
2b2c6286 TL |
740 | NOTE: VMs managed by the HA stack do not follow the 'start on boot' and |
741 | 'boot order' options currently. Those VMs will be skipped by the startup and | |
742 | shutdown algorithm as the HA manager itself ensures that VMs get started and | |
743 | stopped. | |
744 | ||
288e3f46 | 745 | Please note that machines without a Start/Shutdown order parameter will always |
7eed72d8 | 746 | start after those where the parameter is set. Further, this parameter can only |
d750c851 | 747 | be enforced between virtual machines running on the same host, not |
288e3f46 | 748 | cluster-wide. |
076d60ae | 749 | |
c73c190f DM |
750 | |
751 | [[qm_migration]] | |
752 | Migration | |
753 | --------- | |
754 | ||
1ff5e4e8 | 755 | [thumbnail="screenshot/gui-qemu-migrate.png"] |
e4bcef0a | 756 | |
c73c190f DM |
757 | If you have a cluster, you can migrate your VM to another host with |
758 | ||
759 | qm migrate <vmid> <target> | |
760 | ||
8df8cfb7 DC |
761 | There are generally two mechanisms for this |
762 | ||
763 | * Online Migration (aka Live Migration) | |
764 | * Offline Migration | |
765 | ||
766 | Online Migration | |
767 | ~~~~~~~~~~~~~~~~ | |
768 | ||
c73c190f DM |
769 | When your VM is running and it has no local resources defined (such as disks |
770 | on local storage, passed through devices, etc.) you can initiate a live | |
771 | migration with the -online flag. | |
772 | ||
8df8cfb7 DC |
773 | How it works |
774 | ^^^^^^^^^^^^ | |
775 | ||
776 | This starts a Qemu Process on the target host with the 'incoming' flag, which | |
777 | means that the process starts and waits for the memory data and device states | |
778 | from the source Virtual Machine (since all other resources, e.g. disks, | |
779 | are shared, the memory content and device state are the only things left | |
780 | to transmit). | |
781 | ||
782 | Once this connection is established, the source begins to send the memory | |
783 | content asynchronously to the target. If the memory on the source changes, | |
784 | those sections are marked dirty and there will be another pass of sending data. | |
785 | This happens until the amount of data to send is so small that it can | |
786 | pause the VM on the source, send the remaining data to the target and start | |
787 | the VM on the target in under a second. | |
788 | ||
789 | Requirements | |
790 | ^^^^^^^^^^^^ | |
791 | ||
792 | For Live Migration to work, there are some things required: | |
793 | ||
794 | * The VM has no local resources (e.g. passed through devices, local disks, etc.) | |
795 | * The hosts are in the same {pve} cluster. | |
796 | * The hosts have a working (and reliable) network connection. | |
797 | * The target host must have the same or higher versions of the | |
798 | {pve} packages. (It *might* work the other way, but this is never guaranteed) | |
799 | ||
800 | Offline Migration | |
801 | ~~~~~~~~~~~~~~~~~ | |
802 | ||
c73c190f DM |
803 | If you have local resources, you can still offline migrate your VMs, |
804 | as long as all disk are on storages, which are defined on both hosts. | |
805 | Then the migration will copy the disk over the network to the target host. | |
806 | ||
eeb87f95 DM |
807 | [[qm_copy_and_clone]] |
808 | Copies and Clones | |
809 | ----------------- | |
9e55c76d | 810 | |
1ff5e4e8 | 811 | [thumbnail="screenshot/gui-qemu-full-clone.png"] |
9e55c76d DM |
812 | |
813 | VM installation is usually done using an installation media (CD-ROM) | |
814 | from the operation system vendor. Depending on the OS, this can be a | |
815 | time consuming task one might want to avoid. | |
816 | ||
817 | An easy way to deploy many VMs of the same type is to copy an existing | |
818 | VM. We use the term 'clone' for such copies, and distinguish between | |
819 | 'linked' and 'full' clones. | |
820 | ||
821 | Full Clone:: | |
822 | ||
823 | The result of such copy is an independent VM. The | |
824 | new VM does not share any storage resources with the original. | |
825 | + | |
707e37a2 | 826 | |
9e55c76d DM |
827 | It is possible to select a *Target Storage*, so one can use this to |
828 | migrate a VM to a totally different storage. You can also change the | |
829 | disk image *Format* if the storage driver supports several formats. | |
830 | + | |
707e37a2 | 831 | |
730fbca4 | 832 | NOTE: A full clone needs to read and copy all VM image data. This is |
9e55c76d | 833 | usually much slower than creating a linked clone. |
707e37a2 DM |
834 | + |
835 | ||
836 | Some storage types allows to copy a specific *Snapshot*, which | |
837 | defaults to the 'current' VM data. This also means that the final copy | |
838 | never includes any additional snapshots from the original VM. | |
839 | ||
9e55c76d DM |
840 | |
841 | Linked Clone:: | |
842 | ||
730fbca4 | 843 | Modern storage drivers support a way to generate fast linked |
9e55c76d DM |
844 | clones. Such a clone is a writable copy whose initial contents are the |
845 | same as the original data. Creating a linked clone is nearly | |
846 | instantaneous, and initially consumes no additional space. | |
847 | + | |
707e37a2 | 848 | |
9e55c76d DM |
849 | They are called 'linked' because the new image still refers to the |
850 | original. Unmodified data blocks are read from the original image, but | |
851 | modification are written (and afterwards read) from a new | |
852 | location. This technique is called 'Copy-on-write'. | |
853 | + | |
707e37a2 DM |
854 | |
855 | This requires that the original volume is read-only. With {pve} one | |
856 | can convert any VM into a read-only <<qm_templates, Template>>). Such | |
857 | templates can later be used to create linked clones efficiently. | |
858 | + | |
859 | ||
730fbca4 OB |
860 | NOTE: You cannot delete an original template while linked clones |
861 | exist. | |
9e55c76d | 862 | + |
707e37a2 DM |
863 | |
864 | It is not possible to change the *Target storage* for linked clones, | |
865 | because this is a storage internal feature. | |
9e55c76d DM |
866 | |
867 | ||
868 | The *Target node* option allows you to create the new VM on a | |
869 | different node. The only restriction is that the VM is on shared | |
870 | storage, and that storage is also available on the target node. | |
871 | ||
730fbca4 | 872 | To avoid resource conflicts, all network interface MAC addresses get |
9e55c76d DM |
873 | randomized, and we generate a new 'UUID' for the VM BIOS (smbios1) |
874 | setting. | |
875 | ||
876 | ||
707e37a2 DM |
877 | [[qm_templates]] |
878 | Virtual Machine Templates | |
879 | ------------------------- | |
880 | ||
881 | One can convert a VM into a Template. Such templates are read-only, | |
882 | and you can use them to create linked clones. | |
883 | ||
884 | NOTE: It is not possible to start templates, because this would modify | |
885 | the disk images. If you want to change the template, create a linked | |
886 | clone and modify that. | |
887 | ||
319d5325 DC |
888 | VM Generation ID |
889 | ---------------- | |
890 | ||
941ff8d3 | 891 | {pve} supports Virtual Machine Generation ID ('vmgenid') footnote:[Official |
effa4818 TL |
892 | 'vmgenid' Specification |
893 | https://docs.microsoft.com/en-us/windows/desktop/hyperv_v2/virtual-machine-generation-identifier] | |
894 | for virtual machines. | |
895 | This can be used by the guest operating system to detect any event resulting | |
896 | in a time shift event, for example, restoring a backup or a snapshot rollback. | |
319d5325 | 897 | |
effa4818 TL |
898 | When creating new VMs, a 'vmgenid' will be automatically generated and saved |
899 | in its configuration file. | |
319d5325 | 900 | |
effa4818 TL |
901 | To create and add a 'vmgenid' to an already existing VM one can pass the |
902 | special value `1' to let {pve} autogenerate one or manually set the 'UUID' | |
903 | footnote:[Online GUID generator http://guid.one/] by using it as value, | |
904 | e.g.: | |
319d5325 | 905 | |
effa4818 TL |
906 | ---- |
907 | qm set VMID -vmgenid 1 | |
908 | qm set VMID -vmgenid 00000000-0000-0000-0000-000000000000 | |
909 | ---- | |
319d5325 | 910 | |
cfd48f55 TL |
911 | NOTE: The initial addition of a 'vmgenid' device to an existing VM, may result |
912 | in the same effects as a change on snapshot rollback, backup restore, etc., has | |
913 | as the VM can interpret this as generation change. | |
914 | ||
effa4818 TL |
915 | In the rare case the 'vmgenid' mechanism is not wanted one can pass `0' for |
916 | its value on VM creation, or retroactively delete the property in the | |
917 | configuration with: | |
319d5325 | 918 | |
effa4818 TL |
919 | ---- |
920 | qm set VMID -delete vmgenid | |
921 | ---- | |
319d5325 | 922 | |
effa4818 TL |
923 | The most prominent use case for 'vmgenid' are newer Microsoft Windows |
924 | operating systems, which use it to avoid problems in time sensitive or | |
cfd48f55 TL |
925 | replicate services (e.g., databases, domain controller |
926 | footnote:[https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/get-started/virtual-dc/virtualized-domain-controller-architecture]) | |
927 | on snapshot rollback, backup restore or a whole VM clone operation. | |
319d5325 | 928 | |
c069256d EK |
929 | Importing Virtual Machines and disk images |
930 | ------------------------------------------ | |
56368da8 EK |
931 | |
932 | A VM export from a foreign hypervisor takes usually the form of one or more disk | |
59552707 | 933 | images, with a configuration file describing the settings of the VM (RAM, |
56368da8 EK |
934 | number of cores). + |
935 | The disk images can be in the vmdk format, if the disks come from | |
59552707 DM |
936 | VMware or VirtualBox, or qcow2 if the disks come from a KVM hypervisor. |
937 | The most popular configuration format for VM exports is the OVF standard, but in | |
938 | practice interoperation is limited because many settings are not implemented in | |
939 | the standard itself, and hypervisors export the supplementary information | |
56368da8 EK |
940 | in non-standard extensions. |
941 | ||
942 | Besides the problem of format, importing disk images from other hypervisors | |
943 | may fail if the emulated hardware changes too much from one hypervisor to | |
944 | another. Windows VMs are particularly concerned by this, as the OS is very | |
945 | picky about any changes of hardware. This problem may be solved by | |
946 | installing the MergeIDE.zip utility available from the Internet before exporting | |
947 | and choosing a hard disk type of *IDE* before booting the imported Windows VM. | |
948 | ||
59552707 | 949 | Finally there is the question of paravirtualized drivers, which improve the |
56368da8 EK |
950 | speed of the emulated system and are specific to the hypervisor. |
951 | GNU/Linux and other free Unix OSes have all the necessary drivers installed by | |
952 | default and you can switch to the paravirtualized drivers right after importing | |
59552707 | 953 | the VM. For Windows VMs, you need to install the Windows paravirtualized |
56368da8 EK |
954 | drivers by yourself. |
955 | ||
956 | GNU/Linux and other free Unix can usually be imported without hassle. Note | |
eb01c5cf | 957 | that we cannot guarantee a successful import/export of Windows VMs in all |
56368da8 EK |
958 | cases due to the problems above. |
959 | ||
c069256d EK |
960 | Step-by-step example of a Windows OVF import |
961 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
56368da8 | 962 | |
59552707 | 963 | Microsoft provides |
c069256d | 964 | https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/[Virtual Machines downloads] |
144d5ede | 965 | to get started with Windows development.We are going to use one of these |
c069256d | 966 | to demonstrate the OVF import feature. |
56368da8 | 967 | |
c069256d EK |
968 | Download the Virtual Machine zip |
969 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
56368da8 | 970 | |
144d5ede | 971 | After getting informed about the user agreement, choose the _Windows 10 |
c069256d | 972 | Enterprise (Evaluation - Build)_ for the VMware platform, and download the zip. |
56368da8 EK |
973 | |
974 | Extract the disk image from the zip | |
975 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
976 | ||
c069256d EK |
977 | Using the `unzip` utility or any archiver of your choice, unpack the zip, |
978 | and copy via ssh/scp the ovf and vmdk files to your {pve} host. | |
56368da8 | 979 | |
c069256d EK |
980 | Import the Virtual Machine |
981 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
56368da8 | 982 | |
c069256d EK |
983 | This will create a new virtual machine, using cores, memory and |
984 | VM name as read from the OVF manifest, and import the disks to the +local-lvm+ | |
985 | storage. You have to configure the network manually. | |
56368da8 | 986 | |
c069256d | 987 | qm importovf 999 WinDev1709Eval.ovf local-lvm |
56368da8 | 988 | |
c069256d | 989 | The VM is ready to be started. |
56368da8 | 990 | |
c069256d EK |
991 | Adding an external disk image to a Virtual Machine |
992 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
56368da8 | 993 | |
144d5ede | 994 | You can also add an existing disk image to a VM, either coming from a |
c069256d EK |
995 | foreign hypervisor, or one that you created yourself. |
996 | ||
997 | Suppose you created a Debian/Ubuntu disk image with the 'vmdebootstrap' tool: | |
998 | ||
999 | vmdebootstrap --verbose \ | |
67d59a35 | 1000 | --size 10GiB --serial-console \ |
c069256d EK |
1001 | --grub --no-extlinux \ |
1002 | --package openssh-server \ | |
1003 | --package avahi-daemon \ | |
1004 | --package qemu-guest-agent \ | |
1005 | --hostname vm600 --enable-dhcp \ | |
1006 | --customize=./copy_pub_ssh.sh \ | |
1007 | --sparse --image vm600.raw | |
1008 | ||
1009 | You can now create a new target VM for this image. | |
1010 | ||
1011 | qm create 600 --net0 virtio,bridge=vmbr0 --name vm600 --serial0 socket \ | |
1012 | --bootdisk scsi0 --scsihw virtio-scsi-pci --ostype l26 | |
56368da8 | 1013 | |
c069256d EK |
1014 | Add the disk image as +unused0+ to the VM, using the storage +pvedir+: |
1015 | ||
1016 | qm importdisk 600 vm600.raw pvedir | |
1017 | ||
1018 | Finally attach the unused disk to the SCSI controller of the VM: | |
1019 | ||
1020 | qm set 600 --scsi0 pvedir:600/vm-600-disk-1.raw | |
1021 | ||
1022 | The VM is ready to be started. | |
707e37a2 | 1023 | |
7eb69fd2 | 1024 | |
16b4185a | 1025 | ifndef::wiki[] |
7eb69fd2 | 1026 | include::qm-cloud-init.adoc[] |
16b4185a DM |
1027 | endif::wiki[] |
1028 | ||
6e4c46c4 DC |
1029 | ifndef::wiki[] |
1030 | include::qm-pci-passthrough.adoc[] | |
1031 | endif::wiki[] | |
16b4185a | 1032 | |
c2c8eb89 DC |
1033 | Hookscripts |
1034 | ~~~~~~~~~~~ | |
1035 | ||
1036 | You can add a hook script to VMs with the config property `hookscript`. | |
1037 | ||
1038 | qm set 100 -hookscript local:snippets/hookscript.pl | |
1039 | ||
1040 | It will be called during various phases of the guests lifetime. | |
1041 | For an example and documentation see the example script under | |
1042 | `/usr/share/pve-docs/examples/guest-example-hookscript.pl`. | |
7eb69fd2 | 1043 | |
8c1189b6 | 1044 | Managing Virtual Machines with `qm` |
dd042288 | 1045 | ------------------------------------ |
f69cfd23 | 1046 | |
dd042288 | 1047 | qm is the tool to manage Qemu/Kvm virtual machines on {pve}. You can |
f69cfd23 DM |
1048 | create and destroy virtual machines, and control execution |
1049 | (start/stop/suspend/resume). Besides that, you can use qm to set | |
1050 | parameters in the associated config file. It is also possible to | |
1051 | create and delete virtual disks. | |
1052 | ||
dd042288 EK |
1053 | CLI Usage Examples |
1054 | ~~~~~~~~~~~~~~~~~~ | |
1055 | ||
b01b1f2c EK |
1056 | Using an iso file uploaded on the 'local' storage, create a VM |
1057 | with a 4 GB IDE disk on the 'local-lvm' storage | |
dd042288 | 1058 | |
b01b1f2c | 1059 | qm create 300 -ide0 local-lvm:4 -net0 e1000 -cdrom local:iso/proxmox-mailgateway_2.1.iso |
dd042288 EK |
1060 | |
1061 | Start the new VM | |
1062 | ||
1063 | qm start 300 | |
1064 | ||
1065 | Send a shutdown request, then wait until the VM is stopped. | |
1066 | ||
1067 | qm shutdown 300 && qm wait 300 | |
1068 | ||
1069 | Same as above, but only wait for 40 seconds. | |
1070 | ||
1071 | qm shutdown 300 && qm wait 300 -timeout 40 | |
1072 | ||
f0a8ab95 DM |
1073 | |
1074 | [[qm_configuration]] | |
f69cfd23 DM |
1075 | Configuration |
1076 | ------------- | |
1077 | ||
f0a8ab95 DM |
1078 | VM configuration files are stored inside the Proxmox cluster file |
1079 | system, and can be accessed at `/etc/pve/qemu-server/<VMID>.conf`. | |
1080 | Like other files stored inside `/etc/pve/`, they get automatically | |
1081 | replicated to all other cluster nodes. | |
f69cfd23 | 1082 | |
f0a8ab95 DM |
1083 | NOTE: VMIDs < 100 are reserved for internal purposes, and VMIDs need to be |
1084 | unique cluster wide. | |
1085 | ||
1086 | .Example VM Configuration | |
1087 | ---- | |
1088 | cores: 1 | |
1089 | sockets: 1 | |
1090 | memory: 512 | |
1091 | name: webmail | |
1092 | ostype: l26 | |
1093 | bootdisk: virtio0 | |
1094 | net0: e1000=EE:D2:28:5F:B6:3E,bridge=vmbr0 | |
1095 | virtio0: local:vm-100-disk-1,size=32G | |
1096 | ---- | |
1097 | ||
1098 | Those configuration files are simple text files, and you can edit them | |
1099 | using a normal text editor (`vi`, `nano`, ...). This is sometimes | |
1100 | useful to do small corrections, but keep in mind that you need to | |
1101 | restart the VM to apply such changes. | |
1102 | ||
1103 | For that reason, it is usually better to use the `qm` command to | |
1104 | generate and modify those files, or do the whole thing using the GUI. | |
1105 | Our toolkit is smart enough to instantaneously apply most changes to | |
1106 | running VM. This feature is called "hot plug", and there is no | |
1107 | need to restart the VM in that case. | |
1108 | ||
1109 | ||
1110 | File Format | |
1111 | ~~~~~~~~~~~ | |
1112 | ||
1113 | VM configuration files use a simple colon separated key/value | |
1114 | format. Each line has the following format: | |
1115 | ||
1116 | ----- | |
1117 | # this is a comment | |
1118 | OPTION: value | |
1119 | ----- | |
1120 | ||
1121 | Blank lines in those files are ignored, and lines starting with a `#` | |
1122 | character are treated as comments and are also ignored. | |
1123 | ||
1124 | ||
1125 | [[qm_snapshots]] | |
1126 | Snapshots | |
1127 | ~~~~~~~~~ | |
1128 | ||
1129 | When you create a snapshot, `qm` stores the configuration at snapshot | |
1130 | time into a separate snapshot section within the same configuration | |
1131 | file. For example, after creating a snapshot called ``testsnapshot'', | |
1132 | your configuration file will look like this: | |
1133 | ||
1134 | .VM configuration with snapshot | |
1135 | ---- | |
1136 | memory: 512 | |
1137 | swap: 512 | |
1138 | parent: testsnaphot | |
1139 | ... | |
1140 | ||
1141 | [testsnaphot] | |
1142 | memory: 512 | |
1143 | swap: 512 | |
1144 | snaptime: 1457170803 | |
1145 | ... | |
1146 | ---- | |
1147 | ||
1148 | There are a few snapshot related properties like `parent` and | |
1149 | `snaptime`. The `parent` property is used to store the parent/child | |
1150 | relationship between snapshots. `snaptime` is the snapshot creation | |
1151 | time stamp (Unix epoch). | |
f69cfd23 | 1152 | |
f69cfd23 | 1153 | |
80c0adcb | 1154 | [[qm_options]] |
a7f36905 DM |
1155 | Options |
1156 | ~~~~~~~ | |
1157 | ||
1158 | include::qm.conf.5-opts.adoc[] | |
1159 | ||
f69cfd23 DM |
1160 | |
1161 | Locks | |
1162 | ----- | |
1163 | ||
0bcc62dd DM |
1164 | Online migrations, snapshots and backups (`vzdump`) set a lock to |
1165 | prevent incompatible concurrent actions on the affected VMs. Sometimes | |
1166 | you need to remove such a lock manually (e.g., after a power failure). | |
f69cfd23 DM |
1167 | |
1168 | qm unlock <vmid> | |
1169 | ||
0bcc62dd DM |
1170 | CAUTION: Only do that if you are sure the action which set the lock is |
1171 | no longer running. | |
1172 | ||
f69cfd23 | 1173 | |
16b4185a DM |
1174 | ifdef::wiki[] |
1175 | ||
1176 | See Also | |
1177 | ~~~~~~~~ | |
1178 | ||
1179 | * link:/wiki/Cloud-Init_Support[Cloud-Init Support] | |
1180 | ||
1181 | endif::wiki[] | |
1182 | ||
1183 | ||
f69cfd23 | 1184 | ifdef::manvolnum[] |
704f19fb DM |
1185 | |
1186 | Files | |
1187 | ------ | |
1188 | ||
1189 | `/etc/pve/qemu-server/<VMID>.conf`:: | |
1190 | ||
1191 | Configuration file for the VM '<VMID>'. | |
1192 | ||
1193 | ||
f69cfd23 DM |
1194 | include::pve-copyright.adoc[] |
1195 | endif::manvolnum[] |