]>
Commit | Line | Data |
---|---|---|
80c0adcb | 1 | [[chapter_virtual_machines]] |
f69cfd23 | 2 | ifdef::manvolnum[] |
b2f242ab DM |
3 | qm(1) |
4 | ===== | |
5f09af76 DM |
5 | :pve-toplevel: |
6 | ||
f69cfd23 DM |
7 | NAME |
8 | ---- | |
9 | ||
10 | qm - Qemu/KVM Virtual Machine Manager | |
11 | ||
12 | ||
49a5e11c | 13 | SYNOPSIS |
f69cfd23 DM |
14 | -------- |
15 | ||
16 | include::qm.1-synopsis.adoc[] | |
17 | ||
18 | DESCRIPTION | |
19 | ----------- | |
20 | endif::manvolnum[] | |
f69cfd23 DM |
21 | ifndef::manvolnum[] |
22 | Qemu/KVM Virtual Machines | |
23 | ========================= | |
5f09af76 | 24 | :pve-toplevel: |
194d2f29 | 25 | endif::manvolnum[] |
5f09af76 | 26 | |
c4cba5d7 EK |
27 | // deprecates |
28 | // http://pve.proxmox.com/wiki/Container_and_Full_Virtualization | |
29 | // http://pve.proxmox.com/wiki/KVM | |
30 | // http://pve.proxmox.com/wiki/Qemu_Server | |
31 | ||
5eba0743 | 32 | Qemu (short form for Quick Emulator) is an open source hypervisor that emulates a |
c4cba5d7 EK |
33 | physical computer. From the perspective of the host system where Qemu is |
34 | running, Qemu is a user program which has access to a number of local resources | |
35 | like partitions, files, network cards which are then passed to an | |
189d3661 | 36 | emulated computer which sees them as if they were real devices. |
c4cba5d7 EK |
37 | |
38 | A guest operating system running in the emulated computer accesses these | |
39 | devices, and runs as it were running on real hardware. For instance you can pass | |
40 | an iso image as a parameter to Qemu, and the OS running in the emulated computer | |
189d3661 | 41 | will see a real CDROM inserted in a CD drive. |
c4cba5d7 | 42 | |
189d3661 | 43 | Qemu can emulates a great variety of hardware from ARM to Sparc, but {pve} is |
c4cba5d7 EK |
44 | only concerned with 32 and 64 bits PC clone emulation, since it represents the |
45 | overwhelming majority of server hardware. The emulation of PC clones is also one | |
46 | of the fastest due to the availability of processor extensions which greatly | |
47 | speed up Qemu when the emulated architecture is the same as the host | |
9c63b5d9 EK |
48 | architecture. |
49 | ||
50 | NOTE: You may sometimes encounter the term _KVM_ (Kernel-based Virtual Machine). | |
51 | It means that Qemu is running with the support of the virtualization processor | |
52 | extensions, via the Linux kvm module. In the context of {pve} _Qemu_ and | |
53 | _KVM_ can be use interchangeably as Qemu in {pve} will always try to load the kvm | |
54 | module. | |
55 | ||
c4cba5d7 EK |
56 | Qemu inside {pve} runs as a root process, since this is required to access block |
57 | and PCI devices. | |
58 | ||
5eba0743 | 59 | |
c4cba5d7 EK |
60 | Emulated devices and paravirtualized devices |
61 | -------------------------------------------- | |
62 | ||
189d3661 DC |
63 | The PC hardware emulated by Qemu includes a mainboard, network controllers, |
64 | scsi, ide and sata controllers, serial ports (the complete list can be seen in | |
65 | the `kvm(1)` man page) all of them emulated in software. All these devices | |
66 | are the exact software equivalent of existing hardware devices, and if the OS | |
67 | running in the guest has the proper drivers it will use the devices as if it | |
c4cba5d7 EK |
68 | were running on real hardware. This allows Qemu to runs _unmodified_ operating |
69 | systems. | |
70 | ||
71 | This however has a performance cost, as running in software what was meant to | |
72 | run in hardware involves a lot of extra work for the host CPU. To mitigate this, | |
73 | Qemu can present to the guest operating system _paravirtualized devices_, where | |
74 | the guest OS recognizes it is running inside Qemu and cooperates with the | |
75 | hypervisor. | |
76 | ||
77 | Qemu relies on the virtio virtualization standard, and is thus able to presente | |
189d3661 DC |
78 | paravirtualized virtio devices, which includes a paravirtualized generic disk |
79 | controller, a paravirtualized network card, a paravirtualized serial port, | |
c4cba5d7 EK |
80 | a paravirtualized SCSI controller, etc ... |
81 | ||
189d3661 DC |
82 | It is highly recommended to use the virtio devices whenever you can, as they |
83 | provide a big performance improvement. Using the virtio generic disk controller | |
84 | versus an emulated IDE controller will double the sequential write throughput, | |
85 | as measured with `bonnie++(8)`. Using the virtio network interface can deliver | |
c4cba5d7 | 86 | up to three times the throughput of an emulated Intel E1000 network card, as |
189d3661 | 87 | measured with `iperf(1)`. footnote:[See this benchmark on the KVM wiki |
c4cba5d7 EK |
88 | http://www.linux-kvm.org/page/Using_VirtIO_NIC] |
89 | ||
5eba0743 | 90 | |
80c0adcb | 91 | [[qm_virtual_machines_settings]] |
5274ad28 | 92 | Virtual Machines Settings |
c4cba5d7 | 93 | ------------------------- |
80c0adcb | 94 | |
c4cba5d7 EK |
95 | Generally speaking {pve} tries to choose sane defaults for virtual machines |
96 | (VM). Make sure you understand the meaning of the settings you change, as it | |
97 | could incur a performance slowdown, or putting your data at risk. | |
98 | ||
5eba0743 | 99 | |
80c0adcb | 100 | [[qm_general_settings]] |
c4cba5d7 EK |
101 | General Settings |
102 | ~~~~~~~~~~~~~~~~ | |
80c0adcb | 103 | |
b473f999 | 104 | [thumbnail="gui-create-vm-general.png"] |
b16d767f | 105 | |
c4cba5d7 EK |
106 | General settings of a VM include |
107 | ||
108 | * the *Node* : the physical server on which the VM will run | |
109 | * the *VM ID*: a unique number in this {pve} installation used to identify your VM | |
110 | * *Name*: a free form text string you can use to describe the VM | |
111 | * *Resource Pool*: a logical group of VMs | |
112 | ||
5eba0743 | 113 | |
80c0adcb | 114 | [[qm_os_settings]] |
c4cba5d7 EK |
115 | OS Settings |
116 | ~~~~~~~~~~~ | |
80c0adcb | 117 | |
b473f999 | 118 | [thumbnail="gui-create-vm-os.png"] |
200114a7 | 119 | |
c4cba5d7 EK |
120 | When creating a VM, setting the proper Operating System(OS) allows {pve} to |
121 | optimize some low level parameters. For instance Windows OS expect the BIOS | |
122 | clock to use the local time, while Unix based OS expect the BIOS clock to have | |
123 | the UTC time. | |
124 | ||
5eba0743 | 125 | |
80c0adcb | 126 | [[qm_hard_disk]] |
c4cba5d7 EK |
127 | Hard Disk |
128 | ~~~~~~~~~ | |
80c0adcb | 129 | |
2ec49380 | 130 | Qemu can emulate a number of storage controllers: |
c4cba5d7 EK |
131 | |
132 | * the *IDE* controller, has a design which goes back to the 1984 PC/AT disk | |
133 | controller. Even if this controller has been superseded by more more designs, | |
134 | each and every OS you can think has support for it, making it a great choice | |
135 | if you want to run an OS released before 2003. You can connect up to 4 devices | |
136 | on this controller. | |
137 | ||
138 | * the *SATA* (Serial ATA) controller, dating from 2003, has a more modern | |
139 | design, allowing higher throughput and a greater number of devices to be | |
140 | connected. You can connect up to 6 devices on this controller. | |
141 | ||
b0b6802b EK |
142 | * the *SCSI* controller, designed in 1985, is commonly found on server grade |
143 | hardware, and can connect up to 14 storage devices. {pve} emulates by default a | |
f4bfd701 DM |
144 | LSI 53C895A controller. |
145 | + | |
81868c7e | 146 | A SCSI controller of type _VirtIO SCSI_ is the recommended setting if you aim for |
b0b6802b EK |
147 | performance and is automatically selected for newly created Linux VMs since |
148 | {pve} 4.3. Linux distributions have support for this controller since 2012, and | |
c4cba5d7 | 149 | FreeBSD since 2014. For Windows OSes, you need to provide an extra iso |
b0b6802b EK |
150 | containing the drivers during the installation. |
151 | // https://pve.proxmox.com/wiki/Paravirtualized_Block_Drivers_for_Windows#During_windows_installation. | |
81868c7e EK |
152 | If you aim at maximum performance, you can select a SCSI controller of type |
153 | _VirtIO SCSI single_ which will allow you to select the *IO Thread* option. | |
154 | When selecting _VirtIO SCSI single_ Qemu will create a new controller for | |
155 | each disk, instead of adding all disks to the same controller. | |
b0b6802b EK |
156 | |
157 | * The *Virtio* controller, also called virtio-blk to distinguish from | |
81868c7e | 158 | the VirtIO SCSI controller, is an older type of paravirtualized controller |
b0b6802b | 159 | which has been superseded in features by the Virtio SCSI Controller. |
c4cba5d7 | 160 | |
b473f999 | 161 | [thumbnail="gui-create-vm-hard-disk.png"] |
c4cba5d7 EK |
162 | On each controller you attach a number of emulated hard disks, which are backed |
163 | by a file or a block device residing in the configured storage. The choice of | |
164 | a storage type will determine the format of the hard disk image. Storages which | |
165 | present block devices (LVM, ZFS, Ceph) will require the *raw disk image format*, | |
166 | whereas files based storages (Ext4, NFS, GlusterFS) will let you to choose | |
167 | either the *raw disk image format* or the *QEMU image format*. | |
168 | ||
169 | * the *QEMU image format* is a copy on write format which allows snapshots, and | |
170 | thin provisioning of the disk image. | |
189d3661 DC |
171 | * the *raw disk image* is a bit-to-bit image of a hard disk, similar to what |
172 | you would get when executing the `dd` command on a block device in Linux. This | |
173 | format do not support thin provisioning or snapshotting by itself, requiring | |
174 | cooperation from the storage layer for these tasks. It is however 10% faster | |
175 | than the *QEMU image format*. footnote:[See this benchmark for details | |
c4cba5d7 | 176 | http://events.linuxfoundation.org/sites/events/files/slides/CloudOpen2013_Khoa_Huynh_v3.pdf] |
189d3661 | 177 | * the *VMware image format* only makes sense if you intend to import/export the |
c4cba5d7 EK |
178 | disk image to other hypervisors. |
179 | ||
180 | Setting the *Cache* mode of the hard drive will impact how the host system will | |
181 | notify the guest systems of block write completions. The *No cache* default | |
182 | means that the guest system will be notified that a write is complete when each | |
183 | block reaches the physical storage write queue, ignoring the host page cache. | |
184 | This provides a good balance between safety and speed. | |
185 | ||
186 | If you want the {pve} backup manager to skip a disk when doing a backup of a VM, | |
187 | you can set the *No backup* option on that disk. | |
188 | ||
189 | If your storage supports _thin provisioning_ (see the storage chapter in the | |
190 | {pve} guide), and your VM has a *SCSI* controller you can activate the *Discard* | |
191 | option on the hard disks connected to that controller. With *Discard* enabled, | |
192 | when the filesystem of a VM marks blocks as unused after removing files, the | |
193 | emulated SCSI controller will relay this information to the storage, which will | |
194 | then shrink the disk image accordingly. | |
195 | ||
af9c6de1 | 196 | .IO Thread |
81868c7e EK |
197 | The option *IO Thread* can only be used when using a disk with the |
198 | *VirtIO* controller, or with the *SCSI* controller, when the emulated controller | |
199 | type is *VirtIO SCSI single*. | |
200 | With this enabled, Qemu creates one I/O thread per storage controller, | |
201 | instead of a single thread for all I/O, so it increases performance when | |
202 | multiple disks are used and each disk has its own storage controller. | |
c564fc52 DC |
203 | Note that backups do not currently work with *IO Thread* enabled. |
204 | ||
80c0adcb DM |
205 | |
206 | [[qm_cpu]] | |
34e541c5 EK |
207 | CPU |
208 | ~~~ | |
80c0adcb | 209 | |
b473f999 | 210 | [thumbnail="gui-create-vm-cpu.png"] |
397c74c3 | 211 | |
34e541c5 EK |
212 | A *CPU socket* is a physical slot on a PC motherboard where you can plug a CPU. |
213 | This CPU can then contain one or many *cores*, which are independent | |
214 | processing units. Whether you have a single CPU socket with 4 cores, or two CPU | |
215 | sockets with two cores is mostly irrelevant from a performance point of view. | |
216 | However some software is licensed depending on the number of sockets you have in | |
217 | your machine, in that case it makes sense to set the number of of sockets to | |
f4bfd701 DM |
218 | what the license allows you, and increase the number of cores. |
219 | ||
34e541c5 EK |
220 | Increasing the number of virtual cpus (cores and sockets) will usually provide a |
221 | performance improvement though that is heavily dependent on the use of the VM. | |
222 | Multithreaded applications will of course benefit from a large number of | |
223 | virtual cpus, as for each virtual cpu you add, Qemu will create a new thread of | |
224 | execution on the host system. If you're not sure about the workload of your VM, | |
225 | it is usually a safe bet to set the number of *Total cores* to 2. | |
226 | ||
227 | NOTE: It is perfectly safe to set the _overall_ number of total cores in all | |
228 | your VMs to be greater than the number of of cores you have on your server (ie. | |
229 | 4 VMs with each 4 Total cores running in a 8 core machine is OK) In that case | |
230 | the host system will balance the Qemu execution threads between your server | |
231 | cores just like if you were running a standard multithreaded application. | |
232 | However {pve} will prevent you to allocate on a _single_ machine more vcpus than | |
233 | physically available, as this will only bring the performance down due to the | |
234 | cost of context switches. | |
235 | ||
236 | Qemu can emulate a number different of *CPU types* from 486 to the latest Xeon | |
237 | processors. Each new processor generation adds new features, like hardware | |
238 | assisted 3d rendering, random number generation, memory protection, etc ... | |
239 | Usually you should select for your VM a processor type which closely matches the | |
240 | CPU of the host system, as it means that the host CPU features (also called _CPU | |
241 | flags_ ) will be available in your VMs. If you want an exact match, you can set | |
242 | the CPU type to *host* in which case the VM will have exactly the same CPU flags | |
f4bfd701 DM |
243 | as your host system. |
244 | ||
34e541c5 EK |
245 | This has a downside though. If you want to do a live migration of VMs between |
246 | different hosts, your VM might end up on a new system with a different CPU type. | |
247 | If the CPU flags passed to the guest are missing, the qemu process will stop. To | |
248 | remedy this Qemu has also its own CPU type *kvm64*, that {pve} uses by defaults. | |
249 | kvm64 is a Pentium 4 look a like CPU type, which has a reduced CPU flags set, | |
f4bfd701 DM |
250 | but is guaranteed to work everywhere. |
251 | ||
252 | In short, if you care about live migration and moving VMs between nodes, leave | |
34e541c5 EK |
253 | the kvm64 default. If you don’t care about live migration, set the CPU type to |
254 | host, as in theory this will give your guests maximum performance. | |
255 | ||
256 | You can also optionally emulate a *NUMA* architecture in your VMs. The basics of | |
257 | the NUMA architecture mean that instead of having a global memory pool available | |
258 | to all your cores, the memory is spread into local banks close to each socket. | |
259 | This can bring speed improvements as the memory bus is not a bottleneck | |
260 | anymore. If your system has a NUMA architecture footnote:[if the command | |
261 | `numactl --hardware | grep available` returns more than one node, then your host | |
262 | system has a NUMA architecture] we recommend to activate the option, as this | |
263 | will allow proper distribution of the VM resources on the host system. This | |
264 | option is also required in {pve} to allow hotplugging of cores and RAM to a VM. | |
265 | ||
266 | If the NUMA option is used, it is recommended to set the number of sockets to | |
267 | the number of sockets of the host system. | |
268 | ||
80c0adcb DM |
269 | |
270 | [[qm_memory]] | |
34e541c5 EK |
271 | Memory |
272 | ~~~~~~ | |
80c0adcb | 273 | |
34e541c5 EK |
274 | For each VM you have the option to set a fixed size memory or asking |
275 | {pve} to dynamically allocate memory based on the current RAM usage of the | |
276 | host. | |
277 | ||
96124d0f | 278 | .Fixed Memory Allocation |
b473f999 | 279 | [thumbnail="gui-create-vm-memory-fixed.png"] |
96124d0f | 280 | |
34e541c5 EK |
281 | When choosing a *fixed size memory* {pve} will simply allocate what you |
282 | specify to your VM. | |
283 | ||
9abfec65 DC |
284 | Even when using a fixed memory size, the ballooning device gets added to the |
285 | VM, because it delivers useful information such as how much memory the guest | |
286 | really uses. | |
287 | In general, you should leave *ballooning* enabled, but if you want to disable | |
e60ce90c | 288 | it (e.g. for debugging purposes), simply uncheck |
9abfec65 DC |
289 | *Ballooning* or set |
290 | ||
291 | balloon: 0 | |
292 | ||
293 | in the configuration. | |
294 | ||
96124d0f | 295 | .Automatic Memory Allocation |
b473f999 | 296 | [thumbnail="gui-create-vm-memory-dynamic.png", float="left"] |
96124d0f | 297 | |
34e541c5 EK |
298 | // see autoballoon() in pvestatd.pm |
299 | When choosing to *automatically allocate memory*, {pve} will make sure that the | |
300 | minimum amount you specified is always available to the VM, and if RAM usage on | |
301 | the host is below 80%, will dynamically add memory to the guest up to the | |
f4bfd701 DM |
302 | maximum memory specified. |
303 | ||
34e541c5 EK |
304 | When the host is becoming short on RAM, the VM will then release some memory |
305 | back to the host, swapping running processes if needed and starting the oom | |
306 | killer in last resort. The passing around of memory between host and guest is | |
307 | done via a special `balloon` kernel driver running inside the guest, which will | |
308 | grab or release memory pages from the host. | |
309 | footnote:[A good explanation of the inner workings of the balloon driver can be found here https://rwmj.wordpress.com/2010/07/17/virtio-balloon/] | |
310 | ||
c9f6e1a4 EK |
311 | When multiple VMs use the autoallocate facility, it is possible to set a |
312 | *Shares* coefficient which indicates the relative amount of the free host memory | |
313 | that each VM shoud take. Suppose for instance you have four VMs, three of them | |
314 | running a HTTP server and the last one is a database server. To cache more | |
315 | database blocks in the database server RAM, you would like to prioritize the | |
316 | database VM when spare RAM is available. For this you assign a Shares property | |
317 | of 3000 to the database VM, leaving the other VMs to the Shares default setting | |
318 | of 1000. The host server has 32GB of RAM, and is curring using 16GB, leaving 32 | |
319 | * 80/100 - 16 = 9GB RAM to be allocated to the VMs. The database VM will get 9 * | |
320 | 3000 / (3000 + 1000 + 1000 + 1000) = 4.5 GB extra RAM and each HTTP server will | |
321 | get 1/5 GB. | |
322 | ||
34e541c5 EK |
323 | All Linux distributions released after 2010 have the balloon kernel driver |
324 | included. For Windows OSes, the balloon driver needs to be added manually and can | |
325 | incur a slowdown of the guest, so we don't recommend using it on critical | |
326 | systems. | |
327 | // see https://forum.proxmox.com/threads/solved-hyper-threading-vs-no-hyper-threading-fixed-vs-variable-memory.20265/ | |
328 | ||
329 | When allocating RAMs to your VMs, a good rule of thumb is always to leave 1GB | |
330 | of RAM available to the host. | |
331 | ||
80c0adcb DM |
332 | |
333 | [[qm_network_device]] | |
1ff7835b EK |
334 | Network Device |
335 | ~~~~~~~~~~~~~~ | |
80c0adcb | 336 | |
b473f999 | 337 | [thumbnail="gui-create-vm-network.png"] |
c24ddb0a | 338 | |
1ff7835b EK |
339 | Each VM can have many _Network interface controllers_ (NIC), of four different |
340 | types: | |
341 | ||
342 | * *Intel E1000* is the default, and emulates an Intel Gigabit network card. | |
343 | * the *VirtIO* paravirtualized NIC should be used if you aim for maximum | |
344 | performance. Like all VirtIO devices, the guest OS should have the proper driver | |
345 | installed. | |
346 | * the *Realtek 8139* emulates an older 100 MB/s network card, and should | |
347 | only be used when emulating older operating systems ( released before 2002 ) | |
348 | * the *vmxnet3* is another paravirtualized device, which should only be used | |
349 | when importing a VM from another hypervisor. | |
350 | ||
351 | {pve} will generate for each NIC a random *MAC address*, so that your VM is | |
352 | addressable on Ethernet networks. | |
353 | ||
af9c6de1 EK |
354 | The NIC you added to the VM can follow one of two differents models: |
355 | ||
356 | * in the default *Bridged mode* each virtual NIC is backed on the host by a | |
357 | _tap device_, ( a software loopback device simulating an Ethernet NIC ). This | |
358 | tap device is added to a bridge, by default vmbr0 in {pve}. In this mode, VMs | |
359 | have direct access to the Ethernet LAN on which the host is located. | |
360 | * in the alternative *NAT mode*, each virtual NIC will only communicate with | |
361 | the Qemu user networking stack, where a builting router and DHCP server can | |
362 | provide network access. This built-in DHCP will serve adresses in the private | |
363 | 10.0.2.0/24 range. The NAT mode is much slower than the bridged mode, and | |
364 | should only be used for testing. | |
365 | ||
366 | You can also skip adding a network device when creating a VM by selecting *No | |
367 | network device*. | |
368 | ||
369 | .Multiqueue | |
1ff7835b | 370 | If you are using the VirtIO driver, you can optionally activate the |
af9c6de1 | 371 | *Multiqueue* option. This option allows the guest OS to process networking |
1ff7835b EK |
372 | packets using multiple virtual CPUs, providing an increase in the total number |
373 | of packets transfered. | |
374 | ||
375 | //http://blog.vmsplice.net/2011/09/qemu-internals-vhost-architecture.html | |
376 | When using the VirtIO driver with {pve}, each NIC network queue is passed to the | |
377 | host kernel, where the queue will be processed by a kernel thread spawn by the | |
378 | vhost driver. With this option activated, it is possible to pass _multiple_ | |
379 | network queues to the host kernel for each NIC. | |
380 | ||
381 | //https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Tuning_and_Optimization_Guide/sect-Virtualization_Tuning_Optimization_Guide-Networking-Techniques.html#sect-Virtualization_Tuning_Optimization_Guide-Networking-Multi-queue_virtio-net | |
af9c6de1 | 382 | When using Multiqueue, it is recommended to set it to a value equal |
1ff7835b EK |
383 | to the number of Total Cores of your guest. You also need to set in |
384 | the VM the number of multi-purpose channels on each VirtIO NIC with the ethtool | |
385 | command: | |
386 | ||
387 | `ethtool -L eth0 combined X` | |
388 | ||
389 | where X is the number of the number of vcpus of the VM. | |
390 | ||
af9c6de1 | 391 | You should note that setting the Multiqueue parameter to a value greater |
1ff7835b EK |
392 | than one will increase the CPU load on the host and guest systems as the |
393 | traffic increases. We recommend to set this option only when the VM has to | |
394 | process a great number of incoming connections, such as when the VM is running | |
395 | as a router, reverse proxy or a busy HTTP server doing long polling. | |
396 | ||
80c0adcb | 397 | |
685cc8e0 DC |
398 | USB Passthrough |
399 | ~~~~~~~~~~~~~~~ | |
80c0adcb | 400 | |
685cc8e0 DC |
401 | There are two different types of USB passthrough devices: |
402 | ||
403 | * Host USB passtrough | |
404 | * SPICE USB passthrough | |
405 | ||
406 | Host USB passthrough works by giving a VM a USB device of the host. | |
407 | This can either be done via the vendor- and product-id, or | |
408 | via the host bus and port. | |
409 | ||
410 | The vendor/product-id looks like this: *0123:abcd*, | |
411 | where *0123* is the id of the vendor, and *abcd* is the id | |
412 | of the product, meaning two pieces of the same usb device | |
413 | have the same id. | |
414 | ||
415 | The bus/port looks like this: *1-2.3.4*, where *1* is the bus | |
416 | and *2.3.4* is the port path. This represents the physical | |
417 | ports of your host (depending of the internal order of the | |
418 | usb controllers). | |
419 | ||
420 | If a device is present in a VM configuration when the VM starts up, | |
421 | but the device is not present in the host, the VM can boot without problems. | |
422 | As soon as the device/port ist available in the host, it gets passed through. | |
423 | ||
e60ce90c | 424 | WARNING: Using this kind of USB passthrough means that you cannot move |
685cc8e0 DC |
425 | a VM online to another host, since the hardware is only available |
426 | on the host the VM is currently residing. | |
427 | ||
428 | The second type of passthrough is SPICE USB passthrough. This is useful | |
429 | if you use a SPICE client which supports it. If you add a SPICE USB port | |
430 | to your VM, you can passthrough a USB device from where your SPICE client is, | |
431 | directly to the VM (for example an input device or hardware dongle). | |
432 | ||
80c0adcb DM |
433 | |
434 | [[qm_bios_and_uefi]] | |
076d60ae DC |
435 | BIOS and UEFI |
436 | ~~~~~~~~~~~~~ | |
437 | ||
438 | In order to properly emulate a computer, QEMU needs to use a firmware. | |
439 | By default QEMU uses *SeaBIOS* for this, which is an open-source, x86 BIOS | |
440 | implementation. SeaBIOS is a good choice for most standard setups. | |
441 | ||
442 | There are, however, some scenarios in which a BIOS is not a good firmware | |
443 | to boot from, e.g. if you want to do VGA passthrough. footnote:[Alex Williamson has a very good blog entry about this. | |
444 | http://vfio.blogspot.co.at/2014/08/primary-graphics-assignment-without-vga.html] | |
445 | In such cases, you should rather use *OVMF*, which is an open-source UEFI implemenation. footnote:[See the OVMF Project http://www.tianocore.org/ovmf/] | |
446 | ||
447 | If you want to use OVMF, there are several things to consider: | |
448 | ||
449 | In order to save things like the *boot order*, there needs to be an EFI Disk. | |
450 | This disk will be included in backups and snapshots, and there can only be one. | |
451 | ||
452 | You can create such a disk with the following command: | |
453 | ||
454 | qm set <vmid> -efidisk0 <storage>:1,format=<format> | |
455 | ||
456 | Where *<storage>* is the storage where you want to have the disk, and | |
457 | *<format>* is a format which the storage supports. Alternatively, you can | |
458 | create such a disk through the web interface with 'Add' -> 'EFI Disk' in the | |
459 | hardware section of a VM. | |
460 | ||
461 | When using OVMF with a virtual display (without VGA passthrough), | |
462 | you need to set the client resolution in the OVMF menu(which you can reach | |
463 | with a press of the ESC button during boot), or you have to choose | |
464 | SPICE as the display type. | |
465 | ||
288e3f46 EK |
466 | [[qm_startup_and_shutdown]] |
467 | Automatic Start and Shutdown of Virtual Machines | |
468 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
469 | ||
470 | After creating your VMs, you probably want them to start automatically | |
471 | when the host system boots. For this you need to select the option 'Start at | |
472 | boot' from the 'Options' Tab of your VM in the web interface, or set it with | |
473 | the following command: | |
474 | ||
475 | qm set <vmid> -onboot 1 | |
476 | ||
4dbeb548 DM |
477 | .Start and Shutdown Order |
478 | ||
479 | [thumbnail="gui-qemu-edit-start-order.png"] | |
480 | ||
481 | In some case you want to be able to fine tune the boot order of your | |
482 | VMs, for instance if one of your VM is providing firewalling or DHCP | |
483 | to other guest systems. For this you can use the following | |
484 | parameters: | |
288e3f46 EK |
485 | |
486 | * *Start/Shutdown order*: Defines the start order priority. E.g. set it to 1 if | |
487 | you want the VM to be the first to be started. (We use the reverse startup | |
488 | order for shutdown, so a machine with a start order of 1 would be the last to | |
489 | be shut down) | |
490 | * *Startup delay*: Defines the interval between this VM start and subsequent | |
491 | VMs starts . E.g. set it to 240 if you want to wait 240 seconds before starting | |
492 | other VMs. | |
493 | * *Shutdown timeout*: Defines the duration in seconds {pve} should wait | |
494 | for the VM to be offline after issuing a shutdown command. | |
495 | By default this value is set to 60, which means that {pve} will issue a | |
496 | shutdown request, wait 60s for the machine to be offline, and if after 60s | |
497 | the machine is still online will notify that the shutdown action failed. | |
498 | ||
499 | Please note that machines without a Start/Shutdown order parameter will always | |
500 | start after those where the parameter is set, and this parameter only | |
501 | makes sense between the machines running locally on a host, and not | |
502 | cluster-wide. | |
076d60ae | 503 | |
c73c190f DM |
504 | |
505 | [[qm_migration]] | |
506 | Migration | |
507 | --------- | |
508 | ||
e4bcef0a DM |
509 | [thumbnail="gui-qemu-migrate.png"] |
510 | ||
c73c190f DM |
511 | If you have a cluster, you can migrate your VM to another host with |
512 | ||
513 | qm migrate <vmid> <target> | |
514 | ||
8df8cfb7 DC |
515 | There are generally two mechanisms for this |
516 | ||
517 | * Online Migration (aka Live Migration) | |
518 | * Offline Migration | |
519 | ||
520 | Online Migration | |
521 | ~~~~~~~~~~~~~~~~ | |
522 | ||
c73c190f DM |
523 | When your VM is running and it has no local resources defined (such as disks |
524 | on local storage, passed through devices, etc.) you can initiate a live | |
525 | migration with the -online flag. | |
526 | ||
8df8cfb7 DC |
527 | How it works |
528 | ^^^^^^^^^^^^ | |
529 | ||
530 | This starts a Qemu Process on the target host with the 'incoming' flag, which | |
531 | means that the process starts and waits for the memory data and device states | |
532 | from the source Virtual Machine (since all other resources, e.g. disks, | |
533 | are shared, the memory content and device state are the only things left | |
534 | to transmit). | |
535 | ||
536 | Once this connection is established, the source begins to send the memory | |
537 | content asynchronously to the target. If the memory on the source changes, | |
538 | those sections are marked dirty and there will be another pass of sending data. | |
539 | This happens until the amount of data to send is so small that it can | |
540 | pause the VM on the source, send the remaining data to the target and start | |
541 | the VM on the target in under a second. | |
542 | ||
543 | Requirements | |
544 | ^^^^^^^^^^^^ | |
545 | ||
546 | For Live Migration to work, there are some things required: | |
547 | ||
548 | * The VM has no local resources (e.g. passed through devices, local disks, etc.) | |
549 | * The hosts are in the same {pve} cluster. | |
550 | * The hosts have a working (and reliable) network connection. | |
551 | * The target host must have the same or higher versions of the | |
552 | {pve} packages. (It *might* work the other way, but this is never guaranteed) | |
553 | ||
554 | Offline Migration | |
555 | ~~~~~~~~~~~~~~~~~ | |
556 | ||
c73c190f DM |
557 | If you have local resources, you can still offline migrate your VMs, |
558 | as long as all disk are on storages, which are defined on both hosts. | |
559 | Then the migration will copy the disk over the network to the target host. | |
560 | ||
eeb87f95 DM |
561 | [[qm_copy_and_clone]] |
562 | Copies and Clones | |
563 | ----------------- | |
9e55c76d DM |
564 | |
565 | [thumbnail="gui-qemu-full-clone.png"] | |
566 | ||
567 | VM installation is usually done using an installation media (CD-ROM) | |
568 | from the operation system vendor. Depending on the OS, this can be a | |
569 | time consuming task one might want to avoid. | |
570 | ||
571 | An easy way to deploy many VMs of the same type is to copy an existing | |
572 | VM. We use the term 'clone' for such copies, and distinguish between | |
573 | 'linked' and 'full' clones. | |
574 | ||
575 | Full Clone:: | |
576 | ||
577 | The result of such copy is an independent VM. The | |
578 | new VM does not share any storage resources with the original. | |
579 | + | |
707e37a2 | 580 | |
9e55c76d DM |
581 | It is possible to select a *Target Storage*, so one can use this to |
582 | migrate a VM to a totally different storage. You can also change the | |
583 | disk image *Format* if the storage driver supports several formats. | |
584 | + | |
707e37a2 | 585 | |
9e55c76d DM |
586 | NOTE: A full clone need to read and copy all VM image data. This is |
587 | usually much slower than creating a linked clone. | |
707e37a2 DM |
588 | + |
589 | ||
590 | Some storage types allows to copy a specific *Snapshot*, which | |
591 | defaults to the 'current' VM data. This also means that the final copy | |
592 | never includes any additional snapshots from the original VM. | |
593 | ||
9e55c76d DM |
594 | |
595 | Linked Clone:: | |
596 | ||
597 | Modern storage drivers supports a way to generate fast linked | |
598 | clones. Such a clone is a writable copy whose initial contents are the | |
599 | same as the original data. Creating a linked clone is nearly | |
600 | instantaneous, and initially consumes no additional space. | |
601 | + | |
707e37a2 | 602 | |
9e55c76d DM |
603 | They are called 'linked' because the new image still refers to the |
604 | original. Unmodified data blocks are read from the original image, but | |
605 | modification are written (and afterwards read) from a new | |
606 | location. This technique is called 'Copy-on-write'. | |
607 | + | |
707e37a2 DM |
608 | |
609 | This requires that the original volume is read-only. With {pve} one | |
610 | can convert any VM into a read-only <<qm_templates, Template>>). Such | |
611 | templates can later be used to create linked clones efficiently. | |
612 | + | |
613 | ||
614 | NOTE: You cannot delete the original template while linked clones | |
615 | exists. | |
9e55c76d | 616 | + |
707e37a2 DM |
617 | |
618 | It is not possible to change the *Target storage* for linked clones, | |
619 | because this is a storage internal feature. | |
9e55c76d DM |
620 | |
621 | ||
622 | The *Target node* option allows you to create the new VM on a | |
623 | different node. The only restriction is that the VM is on shared | |
624 | storage, and that storage is also available on the target node. | |
625 | ||
9e55c76d DM |
626 | To avoid resource conflicts, all network interface MAC addresses gets |
627 | randomized, and we generate a new 'UUID' for the VM BIOS (smbios1) | |
628 | setting. | |
629 | ||
630 | ||
707e37a2 DM |
631 | [[qm_templates]] |
632 | Virtual Machine Templates | |
633 | ------------------------- | |
634 | ||
635 | One can convert a VM into a Template. Such templates are read-only, | |
636 | and you can use them to create linked clones. | |
637 | ||
638 | NOTE: It is not possible to start templates, because this would modify | |
639 | the disk images. If you want to change the template, create a linked | |
640 | clone and modify that. | |
641 | ||
642 | ||
8c1189b6 | 643 | Managing Virtual Machines with `qm` |
dd042288 | 644 | ------------------------------------ |
f69cfd23 | 645 | |
dd042288 | 646 | qm is the tool to manage Qemu/Kvm virtual machines on {pve}. You can |
f69cfd23 DM |
647 | create and destroy virtual machines, and control execution |
648 | (start/stop/suspend/resume). Besides that, you can use qm to set | |
649 | parameters in the associated config file. It is also possible to | |
650 | create and delete virtual disks. | |
651 | ||
dd042288 EK |
652 | CLI Usage Examples |
653 | ~~~~~~~~~~~~~~~~~~ | |
654 | ||
655 | Create a new VM with 4 GB IDE disk. | |
656 | ||
657 | qm create 300 -ide0 4 -net0 e1000 -cdrom proxmox-mailgateway_2.1.iso | |
658 | ||
659 | Start the new VM | |
660 | ||
661 | qm start 300 | |
662 | ||
663 | Send a shutdown request, then wait until the VM is stopped. | |
664 | ||
665 | qm shutdown 300 && qm wait 300 | |
666 | ||
667 | Same as above, but only wait for 40 seconds. | |
668 | ||
669 | qm shutdown 300 && qm wait 300 -timeout 40 | |
670 | ||
f0a8ab95 DM |
671 | |
672 | [[qm_configuration]] | |
f69cfd23 DM |
673 | Configuration |
674 | ------------- | |
675 | ||
f0a8ab95 DM |
676 | VM configuration files are stored inside the Proxmox cluster file |
677 | system, and can be accessed at `/etc/pve/qemu-server/<VMID>.conf`. | |
678 | Like other files stored inside `/etc/pve/`, they get automatically | |
679 | replicated to all other cluster nodes. | |
f69cfd23 | 680 | |
f0a8ab95 DM |
681 | NOTE: VMIDs < 100 are reserved for internal purposes, and VMIDs need to be |
682 | unique cluster wide. | |
683 | ||
684 | .Example VM Configuration | |
685 | ---- | |
686 | cores: 1 | |
687 | sockets: 1 | |
688 | memory: 512 | |
689 | name: webmail | |
690 | ostype: l26 | |
691 | bootdisk: virtio0 | |
692 | net0: e1000=EE:D2:28:5F:B6:3E,bridge=vmbr0 | |
693 | virtio0: local:vm-100-disk-1,size=32G | |
694 | ---- | |
695 | ||
696 | Those configuration files are simple text files, and you can edit them | |
697 | using a normal text editor (`vi`, `nano`, ...). This is sometimes | |
698 | useful to do small corrections, but keep in mind that you need to | |
699 | restart the VM to apply such changes. | |
700 | ||
701 | For that reason, it is usually better to use the `qm` command to | |
702 | generate and modify those files, or do the whole thing using the GUI. | |
703 | Our toolkit is smart enough to instantaneously apply most changes to | |
704 | running VM. This feature is called "hot plug", and there is no | |
705 | need to restart the VM in that case. | |
706 | ||
707 | ||
708 | File Format | |
709 | ~~~~~~~~~~~ | |
710 | ||
711 | VM configuration files use a simple colon separated key/value | |
712 | format. Each line has the following format: | |
713 | ||
714 | ----- | |
715 | # this is a comment | |
716 | OPTION: value | |
717 | ----- | |
718 | ||
719 | Blank lines in those files are ignored, and lines starting with a `#` | |
720 | character are treated as comments and are also ignored. | |
721 | ||
722 | ||
723 | [[qm_snapshots]] | |
724 | Snapshots | |
725 | ~~~~~~~~~ | |
726 | ||
727 | When you create a snapshot, `qm` stores the configuration at snapshot | |
728 | time into a separate snapshot section within the same configuration | |
729 | file. For example, after creating a snapshot called ``testsnapshot'', | |
730 | your configuration file will look like this: | |
731 | ||
732 | .VM configuration with snapshot | |
733 | ---- | |
734 | memory: 512 | |
735 | swap: 512 | |
736 | parent: testsnaphot | |
737 | ... | |
738 | ||
739 | [testsnaphot] | |
740 | memory: 512 | |
741 | swap: 512 | |
742 | snaptime: 1457170803 | |
743 | ... | |
744 | ---- | |
745 | ||
746 | There are a few snapshot related properties like `parent` and | |
747 | `snaptime`. The `parent` property is used to store the parent/child | |
748 | relationship between snapshots. `snaptime` is the snapshot creation | |
749 | time stamp (Unix epoch). | |
f69cfd23 | 750 | |
f69cfd23 | 751 | |
80c0adcb | 752 | [[qm_options]] |
a7f36905 DM |
753 | Options |
754 | ~~~~~~~ | |
755 | ||
756 | include::qm.conf.5-opts.adoc[] | |
757 | ||
f69cfd23 DM |
758 | |
759 | Locks | |
760 | ----- | |
761 | ||
0bcc62dd DM |
762 | Online migrations, snapshots and backups (`vzdump`) set a lock to |
763 | prevent incompatible concurrent actions on the affected VMs. Sometimes | |
764 | you need to remove such a lock manually (e.g., after a power failure). | |
f69cfd23 DM |
765 | |
766 | qm unlock <vmid> | |
767 | ||
0bcc62dd DM |
768 | CAUTION: Only do that if you are sure the action which set the lock is |
769 | no longer running. | |
770 | ||
f69cfd23 DM |
771 | |
772 | ifdef::manvolnum[] | |
704f19fb DM |
773 | |
774 | Files | |
775 | ------ | |
776 | ||
777 | `/etc/pve/qemu-server/<VMID>.conf`:: | |
778 | ||
779 | Configuration file for the VM '<VMID>'. | |
780 | ||
781 | ||
f69cfd23 DM |
782 | include::pve-copyright.adoc[] |
783 | endif::manvolnum[] |