+[[qm_ivshmem]]
+Inter-VM shared memory
+~~~~~~~~~~~~~~~~~~~~~~
+
+You can add an Inter-VM shared memory device (`ivshmem`), which allows one to
+share memory between the host and a guest, or also between multiple guests.
+
+To add such a device, you can use `qm`:
+
+ qm set <vmid> -ivshmem size=32,name=foo
+
+Where the size is in MiB. The file will be located under
+`/dev/shm/pve-shm-$name` (the default name is the vmid).
+
+NOTE: Currently the device will get deleted as soon as any VM using it got
+shutdown or stopped. Open connections will still persist, but new connections
+to the exact same device cannot be made anymore.
+
+A use case for such a device is the Looking Glass
+footnote:[Looking Glass: https://looking-glass.hostfission.com/] project,
+which enables high performance, low-latency display mirroring between
+host and guest.
+
+[[qm_audio_device]]
+Audio Device
+~~~~~~~~~~~~
+
+To add an audio device run the following command:
+
+----
+qm set <vmid> -audio0 device=<device>
+----
+
+Supported audio devices are:
+
+* `ich9-intel-hda`: Intel HD Audio Controller, emulates ICH9
+* `intel-hda`: Intel HD Audio Controller, emulates ICH6
+* `AC97`: Audio Codec '97, useful for older operating systems like Windows XP
+
+NOTE: The audio device works only in combination with SPICE. Remote protocols
+like Microsoft's RDP have options to play sound. To use the physical audio
+device of the host use device passthrough (see
+xref:qm_pci_passthrough[PCI Passthrough] and
+xref:qm_usb_passthrough[USB Passthrough]).
+
+[[qm_virtio_rng]]
+VirtIO RNG
+~~~~~~~~~~
+
+A RNG (Random Number Generator) is a device providing entropy ('randomness') to
+a system. A virtual hardware-RNG can be used to provide such entropy from the
+host system to a guest VM. This helps to avoid entropy starvation problems in
+the guest (a situation where not enough entropy is available and the system may
+slow down or run into problems), especially during the guests boot process.
+
+To add a VirtIO-based emulated RNG, run the following command:
+
+----
+qm set <vmid> -rng0 source=<source>[,max_bytes=X,period=Y]
+----
+
+`source` specifies where entropy is read from on the host and has to be one of
+the following:
+
+* `/dev/urandom`: Non-blocking kernel entropy pool (preferred)
+* `/dev/random`: Blocking kernel pool (not recommended, can lead to entropy
+ starvation on the host system)
+* `/dev/hwrng`: To pass through a hardware RNG attached to the host (if multiple
+ are available, the one selected in
+ `/sys/devices/virtual/misc/hw_random/rng_current` will be used)
+
+A limit can be specified via the `max_bytes` and `period` parameters, they are
+read as `max_bytes` per `period` in milliseconds. However, it does not represent
+a linear relationship: 1024B/1000ms would mean that up to 1 KiB of data becomes
+available on a 1 second timer, not that 1 KiB is streamed to the guest over the
+course of one second. Reducing the `period` can thus be used to inject entropy
+into the guest at a faster rate.
+
+By default, the limit is set to 1024 bytes per 1000 ms (1 KiB/s). It is
+recommended to always use a limiter to avoid guests using too many host
+resources. If desired, a value of '0' for `max_bytes` can be used to disable
+all limits.
+
+[[qm_startup_and_shutdown]]
+Automatic Start and Shutdown of Virtual Machines
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+After creating your VMs, you probably want them to start automatically
+when the host system boots. For this you need to select the option 'Start at
+boot' from the 'Options' Tab of your VM in the web interface, or set it with
+the following command:
+
+ qm set <vmid> -onboot 1
+
+.Start and Shutdown Order
+
+[thumbnail="screenshot/gui-qemu-edit-start-order.png"]
+
+In some case you want to be able to fine tune the boot order of your
+VMs, for instance if one of your VM is providing firewalling or DHCP
+to other guest systems. For this you can use the following
+parameters:
+
+* *Start/Shutdown order*: Defines the start order priority. E.g. set it to 1 if
+you want the VM to be the first to be started. (We use the reverse startup
+order for shutdown, so a machine with a start order of 1 would be the last to
+be shut down). If multiple VMs have the same order defined on a host, they will
+additionally be ordered by 'VMID' in ascending order.
+* *Startup delay*: Defines the interval between this VM start and subsequent
+VMs starts . E.g. set it to 240 if you want to wait 240 seconds before starting
+other VMs.
+* *Shutdown timeout*: Defines the duration in seconds {pve} should wait
+for the VM to be offline after issuing a shutdown command.
+By default this value is set to 180, which means that {pve} will issue a
+shutdown request and wait 180 seconds for the machine to be offline. If
+the machine is still online after the timeout it will be stopped forcefully.
+
+NOTE: VMs managed by the HA stack do not follow the 'start on boot' and
+'boot order' options currently. Those VMs will be skipped by the startup and
+shutdown algorithm as the HA manager itself ensures that VMs get started and
+stopped.
+
+Please note that machines without a Start/Shutdown order parameter will always
+start after those where the parameter is set. Further, this parameter can only
+be enforced between virtual machines running on the same host, not
+cluster-wide.
+
+[[qm_spice_enhancements]]
+SPICE Enhancements
+~~~~~~~~~~~~~~~~~~
+
+SPICE Enhancements are optional features that can improve the remote viewer
+experience.
+
+To enable them via the GUI go to the *Options* panel of the virtual machine. Run
+the following command to enable them via the CLI:
+
+----
+qm set <vmid> -spice_enhancements foldersharing=1,videostreaming=all
+----
+
+NOTE: To use these features the <<qm_display,*Display*>> of the virtual machine
+must be set to SPICE (qxl).
+
+Folder Sharing
+^^^^^^^^^^^^^^
+
+Share a local folder with the guest. The `spice-webdavd` daemon needs to be
+installed in the guest. It makes the shared folder available through a local
+WebDAV server located at http://localhost:9843.
+
+For Windows guests the installer for the 'Spice WebDAV daemon' can be downloaded
+from the
+https://www.spice-space.org/download.html#windows-binaries[official SPICE website].
+
+Most Linux distributions have a package called `spice-webdavd` that can be
+installed.
+
+To share a folder in Virt-Viewer (Remote Viewer) go to 'File -> Preferences'.
+Select the folder to share and then enable the checkbox.
+
+NOTE: Folder sharing currently only works in the Linux version of Virt-Viewer.
+
+CAUTION: Experimental! Currently this feature does not work reliably.
+
+Video Streaming
+^^^^^^^^^^^^^^^
+
+Fast refreshing areas are encoded into a video stream. Two options exist:
+
+* *all*: Any fast refreshing area will be encoded into a video stream.
+* *filter*: Additional filters are used to decide if video streaming should be
+ used (currently only small window surfaces are skipped).
+
+A general recommendation if video streaming should be enabled and which option
+to choose from cannot be given. Your mileage may vary depending on the specific
+circumstances.
+
+Troubleshooting
+^^^^^^^^^^^^^^^
+
+.Shared folder does not show up
+
+Make sure the WebDAV service is enabled and running in the guest. On Windows it
+is called 'Spice webdav proxy'. In Linux the name is 'spice-webdavd' but can be
+different depending on the distribution.
+
+If the service is running, check the WebDAV server by opening
+http://localhost:9843 in a browser in the guest.
+
+It can help to restart the SPICE session.
+
+[[qm_migration]]
+Migration
+---------
+
+[thumbnail="screenshot/gui-qemu-migrate.png"]
+
+If you have a cluster, you can migrate your VM to another host with
+
+ qm migrate <vmid> <target>
+
+There are generally two mechanisms for this
+
+* Online Migration (aka Live Migration)
+* Offline Migration
+
+Online Migration
+~~~~~~~~~~~~~~~~
+
+When your VM is running and it has no local resources defined (such as disks
+on local storage, passed through devices, etc.) you can initiate a live
+migration with the -online flag.
+
+How it works
+^^^^^^^^^^^^
+
+This starts a Qemu Process on the target host with the 'incoming' flag, which
+means that the process starts and waits for the memory data and device states
+from the source Virtual Machine (since all other resources, e.g. disks,
+are shared, the memory content and device state are the only things left
+to transmit).
+
+Once this connection is established, the source begins to send the memory
+content asynchronously to the target. If the memory on the source changes,
+those sections are marked dirty and there will be another pass of sending data.
+This happens until the amount of data to send is so small that it can
+pause the VM on the source, send the remaining data to the target and start
+the VM on the target in under a second.
+
+Requirements
+^^^^^^^^^^^^
+
+For Live Migration to work, there are some things required:
+
+* The VM has no local resources (e.g. passed through devices, local disks, etc.)
+* The hosts are in the same {pve} cluster.
+* The hosts have a working (and reliable) network connection.
+* The target host must have the same or higher versions of the
+ {pve} packages. (It *might* work the other way, but this is never guaranteed)
+
+Offline Migration
+~~~~~~~~~~~~~~~~~
+
+If you have local resources, you can still offline migrate your VMs,
+as long as all disk are on storages, which are defined on both hosts.
+Then the migration will copy the disk over the network to the target host.
+
+[[qm_copy_and_clone]]
+Copies and Clones
+-----------------
+
+[thumbnail="screenshot/gui-qemu-full-clone.png"]
+
+VM installation is usually done using an installation media (CD-ROM)
+from the operation system vendor. Depending on the OS, this can be a
+time consuming task one might want to avoid.
+
+An easy way to deploy many VMs of the same type is to copy an existing
+VM. We use the term 'clone' for such copies, and distinguish between
+'linked' and 'full' clones.
+
+Full Clone::
+
+The result of such copy is an independent VM. The
+new VM does not share any storage resources with the original.
++
+
+It is possible to select a *Target Storage*, so one can use this to
+migrate a VM to a totally different storage. You can also change the
+disk image *Format* if the storage driver supports several formats.
++
+
+NOTE: A full clone needs to read and copy all VM image data. This is
+usually much slower than creating a linked clone.
++
+
+Some storage types allows to copy a specific *Snapshot*, which
+defaults to the 'current' VM data. This also means that the final copy
+never includes any additional snapshots from the original VM.
+
+
+Linked Clone::
+
+Modern storage drivers support a way to generate fast linked
+clones. Such a clone is a writable copy whose initial contents are the
+same as the original data. Creating a linked clone is nearly
+instantaneous, and initially consumes no additional space.
++
+
+They are called 'linked' because the new image still refers to the
+original. Unmodified data blocks are read from the original image, but
+modification are written (and afterwards read) from a new
+location. This technique is called 'Copy-on-write'.
++
+
+This requires that the original volume is read-only. With {pve} one
+can convert any VM into a read-only <<qm_templates, Template>>). Such
+templates can later be used to create linked clones efficiently.
++
+
+NOTE: You cannot delete an original template while linked clones
+exist.
++
+
+It is not possible to change the *Target storage* for linked clones,
+because this is a storage internal feature.
+
+
+The *Target node* option allows you to create the new VM on a
+different node. The only restriction is that the VM is on shared
+storage, and that storage is also available on the target node.
+
+To avoid resource conflicts, all network interface MAC addresses get
+randomized, and we generate a new 'UUID' for the VM BIOS (smbios1)
+setting.
+
+
+[[qm_templates]]
+Virtual Machine Templates
+-------------------------
+
+One can convert a VM into a Template. Such templates are read-only,
+and you can use them to create linked clones.
+
+NOTE: It is not possible to start templates, because this would modify
+the disk images. If you want to change the template, create a linked
+clone and modify that.
+
+VM Generation ID
+----------------
+
+{pve} supports Virtual Machine Generation ID ('vmgenid') footnote:[Official
+'vmgenid' Specification
+https://docs.microsoft.com/en-us/windows/desktop/hyperv_v2/virtual-machine-generation-identifier]
+for virtual machines.
+This can be used by the guest operating system to detect any event resulting
+in a time shift event, for example, restoring a backup or a snapshot rollback.
+
+When creating new VMs, a 'vmgenid' will be automatically generated and saved
+in its configuration file.
+
+To create and add a 'vmgenid' to an already existing VM one can pass the
+special value `1' to let {pve} autogenerate one or manually set the 'UUID'
+footnote:[Online GUID generator http://guid.one/] by using it as value,
+e.g.:
+
+----
+ qm set VMID -vmgenid 1
+ qm set VMID -vmgenid 00000000-0000-0000-0000-000000000000
+----
+
+NOTE: The initial addition of a 'vmgenid' device to an existing VM, may result
+in the same effects as a change on snapshot rollback, backup restore, etc., has
+as the VM can interpret this as generation change.
+
+In the rare case the 'vmgenid' mechanism is not wanted one can pass `0' for
+its value on VM creation, or retroactively delete the property in the
+configuration with:
+
+----
+ qm set VMID -delete vmgenid
+----
+
+The most prominent use case for 'vmgenid' are newer Microsoft Windows
+operating systems, which use it to avoid problems in time sensitive or
+replicate services (e.g., databases, domain controller
+footnote:[https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/get-started/virtual-dc/virtualized-domain-controller-architecture])
+on snapshot rollback, backup restore or a whole VM clone operation.
+
+Importing Virtual Machines and disk images
+------------------------------------------
+
+A VM export from a foreign hypervisor takes usually the form of one or more disk
+ images, with a configuration file describing the settings of the VM (RAM,
+ number of cores). +
+The disk images can be in the vmdk format, if the disks come from
+VMware or VirtualBox, or qcow2 if the disks come from a KVM hypervisor.
+The most popular configuration format for VM exports is the OVF standard, but in
+practice interoperation is limited because many settings are not implemented in
+the standard itself, and hypervisors export the supplementary information
+in non-standard extensions.
+
+Besides the problem of format, importing disk images from other hypervisors
+may fail if the emulated hardware changes too much from one hypervisor to
+another. Windows VMs are particularly concerned by this, as the OS is very
+picky about any changes of hardware. This problem may be solved by
+installing the MergeIDE.zip utility available from the Internet before exporting
+and choosing a hard disk type of *IDE* before booting the imported Windows VM.
+
+Finally there is the question of paravirtualized drivers, which improve the
+speed of the emulated system and are specific to the hypervisor.
+GNU/Linux and other free Unix OSes have all the necessary drivers installed by
+default and you can switch to the paravirtualized drivers right after importing
+the VM. For Windows VMs, you need to install the Windows paravirtualized
+drivers by yourself.
+
+GNU/Linux and other free Unix can usually be imported without hassle. Note
+that we cannot guarantee a successful import/export of Windows VMs in all
+cases due to the problems above.
+
+Step-by-step example of a Windows OVF import
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Microsoft provides
+https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/[Virtual Machines downloads]
+ to get started with Windows development.We are going to use one of these
+to demonstrate the OVF import feature.
+
+Download the Virtual Machine zip
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+After getting informed about the user agreement, choose the _Windows 10
+Enterprise (Evaluation - Build)_ for the VMware platform, and download the zip.
+
+Extract the disk image from the zip
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Using the `unzip` utility or any archiver of your choice, unpack the zip,
+and copy via ssh/scp the ovf and vmdk files to your {pve} host.
+
+Import the Virtual Machine
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This will create a new virtual machine, using cores, memory and
+VM name as read from the OVF manifest, and import the disks to the +local-lvm+
+ storage. You have to configure the network manually.
+
+ qm importovf 999 WinDev1709Eval.ovf local-lvm
+
+The VM is ready to be started.
+
+Adding an external disk image to a Virtual Machine
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can also add an existing disk image to a VM, either coming from a
+foreign hypervisor, or one that you created yourself.
+
+Suppose you created a Debian/Ubuntu disk image with the 'vmdebootstrap' tool:
+
+ vmdebootstrap --verbose \
+ --size 10GiB --serial-console \
+ --grub --no-extlinux \
+ --package openssh-server \
+ --package avahi-daemon \
+ --package qemu-guest-agent \
+ --hostname vm600 --enable-dhcp \
+ --customize=./copy_pub_ssh.sh \
+ --sparse --image vm600.raw
+
+You can now create a new target VM for this image.
+
+ qm create 600 --net0 virtio,bridge=vmbr0 --name vm600 --serial0 socket \
+ --bootdisk scsi0 --scsihw virtio-scsi-pci --ostype l26
+
+Add the disk image as +unused0+ to the VM, using the storage +pvedir+:
+
+ qm importdisk 600 vm600.raw pvedir
+
+Finally attach the unused disk to the SCSI controller of the VM:
+
+ qm set 600 --scsi0 pvedir:600/vm-600-disk-1.raw
+
+The VM is ready to be started.
+
+
+ifndef::wiki[]
+include::qm-cloud-init.adoc[]
+endif::wiki[]
+
+ifndef::wiki[]
+include::qm-pci-passthrough.adoc[]
+endif::wiki[]
+
+Hookscripts
+-----------
+
+You can add a hook script to VMs with the config property `hookscript`.
+
+ qm set 100 -hookscript local:snippets/hookscript.pl
+
+It will be called during various phases of the guests lifetime.
+For an example and documentation see the example script under
+`/usr/share/pve-docs/examples/guest-example-hookscript.pl`.
+
+[[qm_hibernate]]
+Hibernation
+-----------
+
+You can suspend a VM to disk with the GUI option `Hibernate` or with
+
+ qm suspend ID --todisk
+
+That means that the current content of the memory will be saved onto disk
+and the VM gets stopped. On the next start, the memory content will be
+loaded and the VM can continue where it was left off.
+
+[[qm_vmstatestorage]]
+.State storage selection
+If no target storage for the memory is given, it will be automatically
+chosen, the first of:
+
+1. The storage `vmstatestorage` from the VM config.
+2. The first shared storage from any VM disk.
+3. The first non-shared storage from any VM disk.
+4. The storage `local` as a fallback.