sysadmin: add section about NTP configuration
[pve-docs.git] / qm.adoc
CommitLineData
f69cfd23 1ifdef::manvolnum[]
b2f242ab
DM
2qm(1)
3=====
38fd0958 4include::attributes.txt[]
5f09af76
DM
5:pve-toplevel:
6
f69cfd23
DM
7NAME
8----
9
10qm - Qemu/KVM Virtual Machine Manager
11
12
49a5e11c 13SYNOPSIS
f69cfd23
DM
14--------
15
16include::qm.1-synopsis.adoc[]
17
18DESCRIPTION
19-----------
20endif::manvolnum[]
21
22ifndef::manvolnum[]
23Qemu/KVM Virtual Machines
24=========================
38fd0958 25include::attributes.txt[]
f69cfd23 26endif::manvolnum[]
5f09af76
DM
27ifdef::wiki[]
28:pve-toplevel:
29endif::wiki[]
30
c4cba5d7
EK
31// deprecates
32// http://pve.proxmox.com/wiki/Container_and_Full_Virtualization
33// http://pve.proxmox.com/wiki/KVM
34// http://pve.proxmox.com/wiki/Qemu_Server
35
5eba0743 36Qemu (short form for Quick Emulator) is an open source hypervisor that emulates a
c4cba5d7
EK
37physical computer. From the perspective of the host system where Qemu is
38running, Qemu is a user program which has access to a number of local resources
39like partitions, files, network cards which are then passed to an
189d3661 40emulated computer which sees them as if they were real devices.
c4cba5d7
EK
41
42A guest operating system running in the emulated computer accesses these
43devices, and runs as it were running on real hardware. For instance you can pass
44an iso image as a parameter to Qemu, and the OS running in the emulated computer
189d3661 45will see a real CDROM inserted in a CD drive.
c4cba5d7 46
189d3661 47Qemu can emulates a great variety of hardware from ARM to Sparc, but {pve} is
c4cba5d7
EK
48only concerned with 32 and 64 bits PC clone emulation, since it represents the
49overwhelming majority of server hardware. The emulation of PC clones is also one
50of the fastest due to the availability of processor extensions which greatly
51speed up Qemu when the emulated architecture is the same as the host
9c63b5d9
EK
52architecture.
53
54NOTE: You may sometimes encounter the term _KVM_ (Kernel-based Virtual Machine).
55It means that Qemu is running with the support of the virtualization processor
56extensions, via the Linux kvm module. In the context of {pve} _Qemu_ and
57_KVM_ can be use interchangeably as Qemu in {pve} will always try to load the kvm
58module.
59
c4cba5d7
EK
60Qemu inside {pve} runs as a root process, since this is required to access block
61and PCI devices.
62
5eba0743 63
c4cba5d7
EK
64Emulated devices and paravirtualized devices
65--------------------------------------------
66
189d3661
DC
67The PC hardware emulated by Qemu includes a mainboard, network controllers,
68scsi, ide and sata controllers, serial ports (the complete list can be seen in
69the `kvm(1)` man page) all of them emulated in software. All these devices
70are the exact software equivalent of existing hardware devices, and if the OS
71running in the guest has the proper drivers it will use the devices as if it
c4cba5d7
EK
72were running on real hardware. This allows Qemu to runs _unmodified_ operating
73systems.
74
75This however has a performance cost, as running in software what was meant to
76run in hardware involves a lot of extra work for the host CPU. To mitigate this,
77Qemu can present to the guest operating system _paravirtualized devices_, where
78the guest OS recognizes it is running inside Qemu and cooperates with the
79hypervisor.
80
81Qemu relies on the virtio virtualization standard, and is thus able to presente
189d3661
DC
82paravirtualized virtio devices, which includes a paravirtualized generic disk
83controller, a paravirtualized network card, a paravirtualized serial port,
c4cba5d7
EK
84a paravirtualized SCSI controller, etc ...
85
189d3661
DC
86It is highly recommended to use the virtio devices whenever you can, as they
87provide a big performance improvement. Using the virtio generic disk controller
88versus an emulated IDE controller will double the sequential write throughput,
89as measured with `bonnie++(8)`. Using the virtio network interface can deliver
c4cba5d7 90up to three times the throughput of an emulated Intel E1000 network card, as
189d3661 91measured with `iperf(1)`. footnote:[See this benchmark on the KVM wiki
c4cba5d7
EK
92http://www.linux-kvm.org/page/Using_VirtIO_NIC]
93
5eba0743 94
c4cba5d7
EK
95Virtual Machines settings
96-------------------------
97Generally speaking {pve} tries to choose sane defaults for virtual machines
98(VM). Make sure you understand the meaning of the settings you change, as it
99could incur a performance slowdown, or putting your data at risk.
100
5eba0743 101
c4cba5d7
EK
102General Settings
103~~~~~~~~~~~~~~~~
104General settings of a VM include
105
106* the *Node* : the physical server on which the VM will run
107* the *VM ID*: a unique number in this {pve} installation used to identify your VM
108* *Name*: a free form text string you can use to describe the VM
109* *Resource Pool*: a logical group of VMs
110
5eba0743 111
c4cba5d7
EK
112OS Settings
113~~~~~~~~~~~
114When creating a VM, setting the proper Operating System(OS) allows {pve} to
115optimize some low level parameters. For instance Windows OS expect the BIOS
116clock to use the local time, while Unix based OS expect the BIOS clock to have
117the UTC time.
118
5eba0743 119
c4cba5d7
EK
120Hard Disk
121~~~~~~~~~
2ec49380 122Qemu can emulate a number of storage controllers:
c4cba5d7
EK
123
124* the *IDE* controller, has a design which goes back to the 1984 PC/AT disk
125controller. Even if this controller has been superseded by more more designs,
126each and every OS you can think has support for it, making it a great choice
127if you want to run an OS released before 2003. You can connect up to 4 devices
128on this controller.
129
130* the *SATA* (Serial ATA) controller, dating from 2003, has a more modern
131design, allowing higher throughput and a greater number of devices to be
132connected. You can connect up to 6 devices on this controller.
133
b0b6802b
EK
134* the *SCSI* controller, designed in 1985, is commonly found on server grade
135hardware, and can connect up to 14 storage devices. {pve} emulates by default a
136LSI 53C895A controller. +
137A SCSI controller of type _Virtio_ is the recommended setting if you aim for
138performance and is automatically selected for newly created Linux VMs since
139{pve} 4.3. Linux distributions have support for this controller since 2012, and
c4cba5d7 140FreeBSD since 2014. For Windows OSes, you need to provide an extra iso
b0b6802b
EK
141containing the drivers during the installation.
142// https://pve.proxmox.com/wiki/Paravirtualized_Block_Drivers_for_Windows#During_windows_installation.
143
144* The *Virtio* controller, also called virtio-blk to distinguish from
145the Virtio SCSI controller, is an older type of paravirtualized controller
146which has been superseded in features by the Virtio SCSI Controller.
c4cba5d7
EK
147
148On each controller you attach a number of emulated hard disks, which are backed
149by a file or a block device residing in the configured storage. The choice of
150a storage type will determine the format of the hard disk image. Storages which
151present block devices (LVM, ZFS, Ceph) will require the *raw disk image format*,
152whereas files based storages (Ext4, NFS, GlusterFS) will let you to choose
153either the *raw disk image format* or the *QEMU image format*.
154
155 * the *QEMU image format* is a copy on write format which allows snapshots, and
156 thin provisioning of the disk image.
189d3661
DC
157 * the *raw disk image* is a bit-to-bit image of a hard disk, similar to what
158 you would get when executing the `dd` command on a block device in Linux. This
159 format do not support thin provisioning or snapshotting by itself, requiring
160 cooperation from the storage layer for these tasks. It is however 10% faster
161 than the *QEMU image format*. footnote:[See this benchmark for details
c4cba5d7 162 http://events.linuxfoundation.org/sites/events/files/slides/CloudOpen2013_Khoa_Huynh_v3.pdf]
189d3661 163 * the *VMware image format* only makes sense if you intend to import/export the
c4cba5d7
EK
164 disk image to other hypervisors.
165
166Setting the *Cache* mode of the hard drive will impact how the host system will
167notify the guest systems of block write completions. The *No cache* default
168means that the guest system will be notified that a write is complete when each
169block reaches the physical storage write queue, ignoring the host page cache.
170This provides a good balance between safety and speed.
171
172If you want the {pve} backup manager to skip a disk when doing a backup of a VM,
173you can set the *No backup* option on that disk.
174
175If your storage supports _thin provisioning_ (see the storage chapter in the
176{pve} guide), and your VM has a *SCSI* controller you can activate the *Discard*
177option on the hard disks connected to that controller. With *Discard* enabled,
178when the filesystem of a VM marks blocks as unused after removing files, the
179emulated SCSI controller will relay this information to the storage, which will
180then shrink the disk image accordingly.
181
af9c6de1 182.IO Thread
2b6e4b66
EK
183The option *IO Thread* can only be enabled when using a disk with the *VirtIO* controller,
184or with the *SCSI* controller, when the emulated controller type is *VirtIO SCSI*.
c564fc52
DC
185With this enabled, Qemu uses one thread per disk, instead of one thread for all,
186so it should increase performance when using multiple disks.
187Note that backups do not currently work with *IO Thread* enabled.
188
34e541c5
EK
189CPU
190~~~
191A *CPU socket* is a physical slot on a PC motherboard where you can plug a CPU.
192This CPU can then contain one or many *cores*, which are independent
193processing units. Whether you have a single CPU socket with 4 cores, or two CPU
194sockets with two cores is mostly irrelevant from a performance point of view.
195However some software is licensed depending on the number of sockets you have in
196your machine, in that case it makes sense to set the number of of sockets to
197what the license allows you, and increase the number of cores. +
198Increasing the number of virtual cpus (cores and sockets) will usually provide a
199performance improvement though that is heavily dependent on the use of the VM.
200Multithreaded applications will of course benefit from a large number of
201virtual cpus, as for each virtual cpu you add, Qemu will create a new thread of
202execution on the host system. If you're not sure about the workload of your VM,
203it is usually a safe bet to set the number of *Total cores* to 2.
204
205NOTE: It is perfectly safe to set the _overall_ number of total cores in all
206your VMs to be greater than the number of of cores you have on your server (ie.
2074 VMs with each 4 Total cores running in a 8 core machine is OK) In that case
208the host system will balance the Qemu execution threads between your server
209cores just like if you were running a standard multithreaded application.
210However {pve} will prevent you to allocate on a _single_ machine more vcpus than
211physically available, as this will only bring the performance down due to the
212cost of context switches.
213
214Qemu can emulate a number different of *CPU types* from 486 to the latest Xeon
215processors. Each new processor generation adds new features, like hardware
216assisted 3d rendering, random number generation, memory protection, etc ...
217Usually you should select for your VM a processor type which closely matches the
218CPU of the host system, as it means that the host CPU features (also called _CPU
219flags_ ) will be available in your VMs. If you want an exact match, you can set
220the CPU type to *host* in which case the VM will have exactly the same CPU flags
221as your host system. +
222This has a downside though. If you want to do a live migration of VMs between
223different hosts, your VM might end up on a new system with a different CPU type.
224If the CPU flags passed to the guest are missing, the qemu process will stop. To
225remedy this Qemu has also its own CPU type *kvm64*, that {pve} uses by defaults.
226kvm64 is a Pentium 4 look a like CPU type, which has a reduced CPU flags set,
227but is guaranteed to work everywhere. +
228 In short, if you care about live migration and moving VMs between nodes, leave
229the kvm64 default. If you don’t care about live migration, set the CPU type to
230host, as in theory this will give your guests maximum performance.
231
232You can also optionally emulate a *NUMA* architecture in your VMs. The basics of
233the NUMA architecture mean that instead of having a global memory pool available
234to all your cores, the memory is spread into local banks close to each socket.
235This can bring speed improvements as the memory bus is not a bottleneck
236anymore. If your system has a NUMA architecture footnote:[if the command
237`numactl --hardware | grep available` returns more than one node, then your host
238system has a NUMA architecture] we recommend to activate the option, as this
239will allow proper distribution of the VM resources on the host system. This
240option is also required in {pve} to allow hotplugging of cores and RAM to a VM.
241
242If the NUMA option is used, it is recommended to set the number of sockets to
243the number of sockets of the host system.
244
245Memory
246~~~~~~
247For each VM you have the option to set a fixed size memory or asking
248{pve} to dynamically allocate memory based on the current RAM usage of the
249host.
250
251When choosing a *fixed size memory* {pve} will simply allocate what you
252specify to your VM.
253
254// see autoballoon() in pvestatd.pm
255When choosing to *automatically allocate memory*, {pve} will make sure that the
256minimum amount you specified is always available to the VM, and if RAM usage on
257the host is below 80%, will dynamically add memory to the guest up to the
258maximum memory specified. +
259When the host is becoming short on RAM, the VM will then release some memory
260back to the host, swapping running processes if needed and starting the oom
261killer in last resort. The passing around of memory between host and guest is
262done via a special `balloon` kernel driver running inside the guest, which will
263grab or release memory pages from the host.
264footnote:[A good explanation of the inner workings of the balloon driver can be found here https://rwmj.wordpress.com/2010/07/17/virtio-balloon/]
265
c9f6e1a4
EK
266When multiple VMs use the autoallocate facility, it is possible to set a
267*Shares* coefficient which indicates the relative amount of the free host memory
268that each VM shoud take. Suppose for instance you have four VMs, three of them
269running a HTTP server and the last one is a database server. To cache more
270database blocks in the database server RAM, you would like to prioritize the
271database VM when spare RAM is available. For this you assign a Shares property
272of 3000 to the database VM, leaving the other VMs to the Shares default setting
273of 1000. The host server has 32GB of RAM, and is curring using 16GB, leaving 32
274* 80/100 - 16 = 9GB RAM to be allocated to the VMs. The database VM will get 9 *
2753000 / (3000 + 1000 + 1000 + 1000) = 4.5 GB extra RAM and each HTTP server will
276get 1/5 GB.
277
34e541c5
EK
278All Linux distributions released after 2010 have the balloon kernel driver
279included. For Windows OSes, the balloon driver needs to be added manually and can
280incur a slowdown of the guest, so we don't recommend using it on critical
281systems.
282// see https://forum.proxmox.com/threads/solved-hyper-threading-vs-no-hyper-threading-fixed-vs-variable-memory.20265/
283
284When allocating RAMs to your VMs, a good rule of thumb is always to leave 1GB
285of RAM available to the host.
286
1ff7835b
EK
287Network Device
288~~~~~~~~~~~~~~
289Each VM can have many _Network interface controllers_ (NIC), of four different
290types:
291
292 * *Intel E1000* is the default, and emulates an Intel Gigabit network card.
293 * the *VirtIO* paravirtualized NIC should be used if you aim for maximum
294performance. Like all VirtIO devices, the guest OS should have the proper driver
295installed.
296 * the *Realtek 8139* emulates an older 100 MB/s network card, and should
297only be used when emulating older operating systems ( released before 2002 )
298 * the *vmxnet3* is another paravirtualized device, which should only be used
299when importing a VM from another hypervisor.
300
301{pve} will generate for each NIC a random *MAC address*, so that your VM is
302addressable on Ethernet networks.
303
af9c6de1
EK
304The NIC you added to the VM can follow one of two differents models:
305
306 * in the default *Bridged mode* each virtual NIC is backed on the host by a
307_tap device_, ( a software loopback device simulating an Ethernet NIC ). This
308tap device is added to a bridge, by default vmbr0 in {pve}. In this mode, VMs
309have direct access to the Ethernet LAN on which the host is located.
310 * in the alternative *NAT mode*, each virtual NIC will only communicate with
311the Qemu user networking stack, where a builting router and DHCP server can
312provide network access. This built-in DHCP will serve adresses in the private
31310.0.2.0/24 range. The NAT mode is much slower than the bridged mode, and
314should only be used for testing.
315
316You can also skip adding a network device when creating a VM by selecting *No
317network device*.
318
319.Multiqueue
1ff7835b 320If you are using the VirtIO driver, you can optionally activate the
af9c6de1 321*Multiqueue* option. This option allows the guest OS to process networking
1ff7835b
EK
322packets using multiple virtual CPUs, providing an increase in the total number
323of packets transfered.
324
325//http://blog.vmsplice.net/2011/09/qemu-internals-vhost-architecture.html
326When using the VirtIO driver with {pve}, each NIC network queue is passed to the
327host kernel, where the queue will be processed by a kernel thread spawn by the
328vhost driver. With this option activated, it is possible to pass _multiple_
329network queues to the host kernel for each NIC.
330
331//https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Tuning_and_Optimization_Guide/sect-Virtualization_Tuning_Optimization_Guide-Networking-Techniques.html#sect-Virtualization_Tuning_Optimization_Guide-Networking-Multi-queue_virtio-net
af9c6de1 332When using Multiqueue, it is recommended to set it to a value equal
1ff7835b
EK
333to the number of Total Cores of your guest. You also need to set in
334the VM the number of multi-purpose channels on each VirtIO NIC with the ethtool
335command:
336
337`ethtool -L eth0 combined X`
338
339where X is the number of the number of vcpus of the VM.
340
af9c6de1 341You should note that setting the Multiqueue parameter to a value greater
1ff7835b
EK
342than one will increase the CPU load on the host and guest systems as the
343traffic increases. We recommend to set this option only when the VM has to
344process a great number of incoming connections, such as when the VM is running
345as a router, reverse proxy or a busy HTTP server doing long polling.
346
685cc8e0
DC
347USB Passthrough
348~~~~~~~~~~~~~~~
349There are two different types of USB passthrough devices:
350
351* Host USB passtrough
352* SPICE USB passthrough
353
354Host USB passthrough works by giving a VM a USB device of the host.
355This can either be done via the vendor- and product-id, or
356via the host bus and port.
357
358The vendor/product-id looks like this: *0123:abcd*,
359where *0123* is the id of the vendor, and *abcd* is the id
360of the product, meaning two pieces of the same usb device
361have the same id.
362
363The bus/port looks like this: *1-2.3.4*, where *1* is the bus
364and *2.3.4* is the port path. This represents the physical
365ports of your host (depending of the internal order of the
366usb controllers).
367
368If a device is present in a VM configuration when the VM starts up,
369but the device is not present in the host, the VM can boot without problems.
370As soon as the device/port ist available in the host, it gets passed through.
371
372WARNING: Using this kind of USB passthrough, means that you cannot move
373a VM online to another host, since the hardware is only available
374on the host the VM is currently residing.
375
376The second type of passthrough is SPICE USB passthrough. This is useful
377if you use a SPICE client which supports it. If you add a SPICE USB port
378to your VM, you can passthrough a USB device from where your SPICE client is,
379directly to the VM (for example an input device or hardware dongle).
380
076d60ae
DC
381BIOS and UEFI
382~~~~~~~~~~~~~
383
384In order to properly emulate a computer, QEMU needs to use a firmware.
385By default QEMU uses *SeaBIOS* for this, which is an open-source, x86 BIOS
386implementation. SeaBIOS is a good choice for most standard setups.
387
388There are, however, some scenarios in which a BIOS is not a good firmware
389to boot from, e.g. if you want to do VGA passthrough. footnote:[Alex Williamson has a very good blog entry about this.
390http://vfio.blogspot.co.at/2014/08/primary-graphics-assignment-without-vga.html]
391In such cases, you should rather use *OVMF*, which is an open-source UEFI implemenation. footnote:[See the OVMF Project http://www.tianocore.org/ovmf/]
392
393If you want to use OVMF, there are several things to consider:
394
395In order to save things like the *boot order*, there needs to be an EFI Disk.
396This disk will be included in backups and snapshots, and there can only be one.
397
398You can create such a disk with the following command:
399
400 qm set <vmid> -efidisk0 <storage>:1,format=<format>
401
402Where *<storage>* is the storage where you want to have the disk, and
403*<format>* is a format which the storage supports. Alternatively, you can
404create such a disk through the web interface with 'Add' -> 'EFI Disk' in the
405hardware section of a VM.
406
407When using OVMF with a virtual display (without VGA passthrough),
408you need to set the client resolution in the OVMF menu(which you can reach
409with a press of the ESC button during boot), or you have to choose
410SPICE as the display type.
411
412
8c1189b6 413Managing Virtual Machines with `qm`
dd042288 414------------------------------------
f69cfd23 415
dd042288 416qm is the tool to manage Qemu/Kvm virtual machines on {pve}. You can
f69cfd23
DM
417create and destroy virtual machines, and control execution
418(start/stop/suspend/resume). Besides that, you can use qm to set
419parameters in the associated config file. It is also possible to
420create and delete virtual disks.
421
dd042288
EK
422CLI Usage Examples
423~~~~~~~~~~~~~~~~~~
424
425Create a new VM with 4 GB IDE disk.
426
427 qm create 300 -ide0 4 -net0 e1000 -cdrom proxmox-mailgateway_2.1.iso
428
429Start the new VM
430
431 qm start 300
432
433Send a shutdown request, then wait until the VM is stopped.
434
435 qm shutdown 300 && qm wait 300
436
437Same as above, but only wait for 40 seconds.
438
439 qm shutdown 300 && qm wait 300 -timeout 40
440
f69cfd23
DM
441Configuration
442-------------
443
444All configuration files consists of lines in the form
445
446 PARAMETER: value
447
871e1fd6 448Configuration files are stored inside the Proxmox cluster file
8c1189b6 449system, and can be accessed at `/etc/pve/qemu-server/<VMID>.conf`.
f69cfd23 450
a7f36905
DM
451Options
452~~~~~~~
453
454include::qm.conf.5-opts.adoc[]
455
f69cfd23
DM
456
457Locks
458-----
459
8c1189b6 460Online migrations and backups (`vzdump`) set a lock to prevent incompatible
871e1fd6
FG
461concurrent actions on the affected VMs. Sometimes you need to remove such a
462lock manually (e.g., after a power failure).
f69cfd23
DM
463
464 qm unlock <vmid>
465
f69cfd23
DM
466
467ifdef::manvolnum[]
468include::pve-copyright.adoc[]
469endif::manvolnum[]