]>
Commit | Line | Data |
---|---|---|
9cd9313f DH |
1 | QEMU VM templating |
2 | ================== | |
3 | ||
4 | This document explains how to use VM templating in QEMU. | |
5 | ||
6 | For now, the focus is on VM memory aspects, and not about how to save and | |
7 | restore other VM state (i.e., migrate-to-file with ``x-ignore-shared``). | |
8 | ||
9 | Overview | |
10 | -------- | |
11 | ||
12 | With VM templating, a single template VM serves as the starting point for | |
13 | new VMs. This allows for fast and efficient replication of VMs, resulting | |
14 | in fast startup times and reduced memory consumption. | |
15 | ||
16 | Conceptually, the VM state is frozen, to then be used as a basis for new | |
17 | VMs. The Copy-On-Write mechanism in the operating systems makes sure that | |
18 | new VMs are able to read template VM memory; however, any modifications | |
19 | stay private and don't modify the original template VM or any other | |
20 | created VM. | |
21 | ||
22 | !!! Security Alert !!! | |
23 | ---------------------- | |
24 | ||
25 | When effectively cloning VMs by VM templating, hardware identifiers | |
26 | (such as UUIDs and NIC MAC addresses), and similar data in the guest OS | |
27 | (such as machine IDs, SSH keys, certificates) that are supposed to be | |
28 | *unique* are no longer unique, which can be a security concern. | |
29 | ||
30 | Please be aware of these implications and how to mitigate them for your | |
31 | use case, which might involve vmgenid, hot(un)plug of NIC, etc.. | |
32 | ||
33 | Memory configuration | |
34 | -------------------- | |
35 | ||
36 | In order to create the template VM, we have to make sure that VM memory | |
37 | ends up in a file, from where it can be reused for the new VMs: | |
38 | ||
39 | Supply VM RAM via memory-backend-file, with ``share=on`` (modifications go | |
40 | to the file) and ``readonly=off`` (open the file writable). Note that | |
41 | ``readonly=off`` is implicit. | |
42 | ||
43 | In the following command-line example, a 2GB VM is created, whereby VM RAM | |
44 | is to be stored in the ``template`` file. | |
45 | ||
46 | .. parsed-literal:: | |
47 | ||
48 | |qemu_system| [...] -m 2g \\ | |
49 | -object memory-backend-file,id=pc.ram,mem-path=template,size=2g,share=on,... \\ | |
50 | -machine q35,memory-backend=pc.ram | |
51 | ||
52 | If multiple memory backends are used (vNUMA, DIMMs), configure all | |
53 | memory backends accordingly. | |
54 | ||
55 | Once the VM is in the desired state, stop the VM and save other VM state, | |
56 | leaving the current state of VM RAM reside in the file. | |
57 | ||
58 | In order to have a new VM be based on a template VM, we have to | |
59 | configure VM RAM to be based on a template VM RAM file; however, the VM | |
60 | should not be able to modify file content. | |
61 | ||
62 | Supply VM RAM via memory-backend-file, with ``share=off`` (modifications | |
63 | stay private), ``readonly=on`` (open the file readonly) and ``rom=off`` | |
64 | (don't make the memory readonly for the VM). Note that ``share=off`` is | |
65 | implicit and that other VM state has to be restored separately. | |
66 | ||
67 | In the following command-line example, a 2GB VM is created based on the | |
68 | existing 2GB file ``template``. | |
69 | ||
70 | .. parsed-literal:: | |
71 | ||
72 | |qemu_system| [...] -m 2g \\ | |
73 | -object memory-backend-file,id=pc.ram,mem-path=template,size=2g,readonly=on,rom=off,... \\ | |
74 | -machine q35,memory-backend=pc.ram | |
75 | ||
76 | If multiple memory backends are used (vNUMA, DIMMs), configure all | |
77 | memory backends accordingly. | |
78 | ||
79 | Note that ``-mem-path`` cannot be used for VM templating when creating the | |
80 | template VM or when starting new VMs based on a template VM. | |
81 | ||
82 | Incompatible features | |
83 | --------------------- | |
84 | ||
85 | Some features are incompatible with VM templating, as the underlying file | |
86 | cannot be modified to discard VM RAM, or to actually share memory with | |
87 | another process. | |
88 | ||
89 | vhost-user and multi-process QEMU | |
90 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | |
91 | ||
92 | vhost-user and multi-process QEMU are incompatible with VM templating. | |
93 | These technologies rely on shared memory, however, the template VMs | |
94 | don't actually share memory (``share=off``), even though they are | |
95 | file-based. | |
96 | ||
97 | virtio-balloon | |
98 | ~~~~~~~~~~~~~~ | |
99 | ||
100 | virtio-balloon inflation and "free page reporting" cannot discard VM RAM | |
101 | and will repeatedly report errors. While virtio-balloon can be used | |
102 | for template VMs (e.g., report VM RAM stats), "free page reporting" | |
103 | should be disabled and the balloon should not be inflated. | |
104 | ||
105 | virtio-mem | |
106 | ~~~~~~~~~~ | |
107 | ||
108 | virtio-mem cannot discard VM RAM that is managed by the virtio-mem | |
109 | device. virtio-mem will fail early when realizing the device. To use | |
110 | VM templating with virtio-mem, either hotplug virtio-mem devices to the | |
111 | new VM, or don't supply any memory to the template VM using virtio-mem | |
112 | (requested-size=0), not using a template VM file as memory backend for the | |
113 | virtio-mem device. | |
114 | ||
115 | VM migration | |
116 | ~~~~~~~~~~~~ | |
117 | ||
118 | For VM migration, "x-release-ram" similarly relies on discarding of VM | |
119 | RAM on the migration source to free up migrated RAM, and will | |
120 | repeatedly report errors. | |
121 | ||
122 | Postcopy live migration fails discarding VM RAM on the migration | |
123 | destination early and refuses to activate postcopy live migration. Note | |
124 | that postcopy live migration usually only works on selected filesystems | |
125 | (shmem/tmpfs, hugetlbfs) either way. |