]>
Commit | Line | Data |
---|---|---|
2e3c8f8d DDAG |
1 | ========= |
2 | Migration | |
3 | ========= | |
f58ae59c JQ |
4 | |
5 | QEMU has code to load/save the state of the guest that it is running. | |
dda5336e | 6 | These are two complementary operations. Saving the state just does |
f58ae59c JQ |
7 | that, saves the state for each device that the guest is running. |
8 | Restoring a guest is just the opposite operation: we need to load the | |
9 | state of each device. | |
10 | ||
dda5336e | 11 | For this to work, QEMU has to be launched with the same arguments the |
f58ae59c JQ |
12 | two times. I.e. it can only restore the state in one guest that has |
13 | the same devices that the one it was saved (this last requirement can | |
dda5336e | 14 | be relaxed a bit, but for now we can consider that configuration has |
f58ae59c JQ |
15 | to be exactly the same). |
16 | ||
17 | Once that we are able to save/restore a guest, a new functionality is | |
18 | requested: migration. This means that QEMU is able to start in one | |
dda5336e SW |
19 | machine and being "migrated" to another machine. I.e. being moved to |
20 | another machine. | |
f58ae59c JQ |
21 | |
22 | Next was the "live migration" functionality. This is important | |
23 | because some guests run with a lot of state (specially RAM), and it | |
24 | can take a while to move all state from one machine to another. Live | |
25 | migration allows the guest to continue running while the state is | |
26 | transferred. Only while the last part of the state is transferred has | |
27 | the guest to be stopped. Typically the time that the guest is | |
28 | unresponsive during live migration is the low hundred of milliseconds | |
dda5336e | 29 | (notice that this depends on a lot of things). |
f58ae59c | 30 | |
edd70806 DDAG |
31 | Transports |
32 | ========== | |
f58ae59c | 33 | |
edd70806 DDAG |
34 | The migration stream is normally just a byte stream that can be passed |
35 | over any transport. | |
f58ae59c JQ |
36 | |
37 | - tcp migration: do the migration using tcp sockets | |
38 | - unix migration: do the migration using unix sockets | |
39 | - exec migration: do the migration using the stdin/stdout through a process. | |
9277d81f | 40 | - fd migration: do the migration using a file descriptor that is |
dda5336e | 41 | passed to QEMU. QEMU doesn't care how this file descriptor is opened. |
f58ae59c | 42 | |
edd70806 DDAG |
43 | In addition, support is included for migration using RDMA, which |
44 | transports the page data using ``RDMA``, where the hardware takes care of | |
45 | transporting the pages, and the load on the CPU is much lower. While the | |
46 | internals of RDMA migration are a bit different, this isn't really visible | |
47 | outside the RAM migration code. | |
48 | ||
49 | All these migration protocols use the same infrastructure to | |
f58ae59c JQ |
50 | save/restore state devices. This infrastructure is shared with the |
51 | savevm/loadvm functionality. | |
52 | ||
979da8b3 MAL |
53 | Debugging |
54 | ========= | |
55 | ||
4df3a7bf | 56 | The migration stream can be analyzed thanks to ``scripts/analyze-migration.py``. |
979da8b3 MAL |
57 | |
58 | Example usage: | |
59 | ||
60 | .. code-block:: shell | |
61 | ||
243e7480 MA |
62 | $ qemu-system-x86_64 -display none -monitor stdio |
63 | (qemu) migrate "exec:cat > mig" | |
64 | (qemu) q | |
65 | $ ./scripts/analyze-migration.py -f mig | |
979da8b3 MAL |
66 | { |
67 | "ram (3)": { | |
68 | "section sizes": { | |
69 | "pc.ram": "0x0000000008000000", | |
70 | ... | |
71 | ||
243e7480 | 72 | See also ``analyze-migration.py -h`` help for more options. |
979da8b3 | 73 | |
2e3c8f8d DDAG |
74 | Common infrastructure |
75 | ===================== | |
f58ae59c | 76 | |
2e3c8f8d | 77 | The files, sockets or fd's that carry the migration stream are abstracted by |
4df3a7bf PM |
78 | the ``QEMUFile`` type (see ``migration/qemu-file.h``). In most cases this |
79 | is connected to a subtype of ``QIOChannel`` (see ``io/``). | |
f58ae59c | 80 | |
edd70806 | 81 | |
2e3c8f8d DDAG |
82 | Saving the state of one device |
83 | ============================== | |
f58ae59c | 84 | |
edd70806 DDAG |
85 | For most devices, the state is saved in a single call to the migration |
86 | infrastructure; these are *non-iterative* devices. The data for these | |
87 | devices is sent at the end of precopy migration, when the CPUs are paused. | |
88 | There are also *iterative* devices, which contain a very large amount of | |
89 | data (e.g. RAM or large tables). See the iterative device section below. | |
90 | ||
91 | General advice for device developers | |
92 | ------------------------------------ | |
93 | ||
94 | - The migration state saved should reflect the device being modelled rather | |
95 | than the way your implementation works. That way if you change the implementation | |
96 | later the migration stream will stay compatible. That model may include | |
97 | internal state that's not directly visible in a register. | |
98 | ||
99 | - When saving a migration stream the device code may walk and check | |
100 | the state of the device. These checks might fail in various ways (e.g. | |
101 | discovering internal state is corrupt or that the guest has done something bad). | |
102 | Consider carefully before asserting/aborting at this point, since the | |
103 | normal response from users is that *migration broke their VM* since it had | |
104 | apparently been running fine until then. In these error cases, the device | |
105 | should log a message indicating the cause of error, and should consider | |
106 | putting the device into an error state, allowing the rest of the VM to | |
107 | continue execution. | |
108 | ||
109 | - The migration might happen at an inconvenient point, | |
110 | e.g. right in the middle of the guest reprogramming the device, during | |
111 | guest reboot or shutdown or while the device is waiting for external IO. | |
112 | It's strongly preferred that migrations do not fail in this situation, | |
113 | since in the cloud environment migrations might happen automatically to | |
114 | VMs that the administrator doesn't directly control. | |
115 | ||
116 | - If you do need to fail a migration, ensure that sufficient information | |
117 | is logged to identify what went wrong. | |
118 | ||
119 | - The destination should treat an incoming migration stream as hostile | |
120 | (which we do to varying degrees in the existing code). Check that offsets | |
121 | into buffers and the like can't cause overruns. Fail the incoming migration | |
122 | in the case of a corrupted stream like this. | |
123 | ||
124 | - Take care with internal device state or behaviour that might become | |
125 | migration version dependent. For example, the order of PCI capabilities | |
126 | is required to stay constant across migration. Another example would | |
127 | be that a special case handled by subsections (see below) might become | |
128 | much more common if a default behaviour is changed. | |
129 | ||
130 | - The state of the source should not be changed or destroyed by the | |
131 | outgoing migration. Migrations timing out or being failed by | |
132 | higher levels of management, or failures of the destination host are | |
133 | not unusual, and in that case the VM is restarted on the source. | |
134 | Note that the management layer can validly revert the migration | |
135 | even though the QEMU level of migration has succeeded as long as it | |
136 | does it before starting execution on the destination. | |
137 | ||
138 | - Buses and devices should be able to explicitly specify addresses when | |
139 | instantiated, and management tools should use those. For example, | |
140 | when hot adding USB devices it's important to specify the ports | |
141 | and addresses, since implicit ordering based on the command line order | |
142 | may be different on the destination. This can result in the | |
143 | device state being loaded into the wrong device. | |
f58ae59c | 144 | |
2e3c8f8d DDAG |
145 | VMState |
146 | ------- | |
f58ae59c | 147 | |
edd70806 DDAG |
148 | Most device data can be described using the ``VMSTATE`` macros (mostly defined |
149 | in ``include/migration/vmstate.h``). | |
f58ae59c | 150 | |
7465dfec | 151 | An example (from hw/input/pckbd.c) |
f58ae59c | 152 | |
2e3c8f8d DDAG |
153 | .. code:: c |
154 | ||
155 | static const VMStateDescription vmstate_kbd = { | |
156 | .name = "pckbd", | |
157 | .version_id = 3, | |
158 | .minimum_version_id = 3, | |
159 | .fields = (VMStateField[]) { | |
160 | VMSTATE_UINT8(write_cmd, KBDState), | |
161 | VMSTATE_UINT8(status, KBDState), | |
162 | VMSTATE_UINT8(mode, KBDState), | |
163 | VMSTATE_UINT8(pending, KBDState), | |
164 | VMSTATE_END_OF_LIST() | |
165 | } | |
166 | }; | |
f58ae59c JQ |
167 | |
168 | We are declaring the state with name "pckbd". | |
4df3a7bf | 169 | The ``version_id`` is 3, and the fields are 4 uint8_t in a KBDState structure. |
f58ae59c JQ |
170 | We registered this with: |
171 | ||
2e3c8f8d DDAG |
172 | .. code:: c |
173 | ||
f58ae59c JQ |
174 | vmstate_register(NULL, 0, &vmstate_kbd, s); |
175 | ||
4df3a7bf | 176 | For devices that are ``qdev`` based, we can register the device in the class |
edd70806 | 177 | init function: |
f58ae59c | 178 | |
edd70806 | 179 | .. code:: c |
f58ae59c | 180 | |
edd70806 | 181 | dc->vmsd = &vmstate_kbd_isa; |
f58ae59c | 182 | |
edd70806 DDAG |
183 | The VMState macros take care of ensuring that the device data section |
184 | is formatted portably (normally big endian) and make some compile time checks | |
185 | against the types of the fields in the structures. | |
f58ae59c | 186 | |
edd70806 DDAG |
187 | VMState macros can include other VMStateDescriptions to store substructures |
188 | (see ``VMSTATE_STRUCT_``), arrays (``VMSTATE_ARRAY_``) and variable length | |
189 | arrays (``VMSTATE_VARRAY_``). Various other macros exist for special | |
190 | cases. | |
5f9412bb | 191 | |
edd70806 DDAG |
192 | Note that the format on the wire is still very raw; i.e. a VMSTATE_UINT32 |
193 | ends up with a 4 byte bigendian representation on the wire; in the future | |
194 | it might be possible to use a more structured format. | |
f58ae59c | 195 | |
edd70806 DDAG |
196 | Legacy way |
197 | ---------- | |
f58ae59c | 198 | |
edd70806 DDAG |
199 | This way is going to disappear as soon as all current users are ported to VMSTATE; |
200 | although converting existing code can be tricky, and thus 'soon' is relative. | |
f58ae59c | 201 | |
edd70806 DDAG |
202 | Each device has to register two functions, one to save the state and |
203 | another to load the state back. | |
f58ae59c | 204 | |
edd70806 | 205 | .. code:: c |
f58ae59c | 206 | |
ce62df53 | 207 | int register_savevm_live(const char *idstr, |
edd70806 DDAG |
208 | int instance_id, |
209 | int version_id, | |
210 | SaveVMHandlers *ops, | |
211 | void *opaque); | |
f58ae59c | 212 | |
4df3a7bf PM |
213 | Two functions in the ``ops`` structure are the ``save_state`` |
214 | and ``load_state`` functions. Notice that ``load_state`` receives a version_id | |
215 | parameter to know what state format is receiving. ``save_state`` doesn't | |
edd70806 | 216 | have a version_id parameter because it always uses the latest version. |
f58ae59c | 217 | |
edd70806 DDAG |
218 | Note that because the VMState macros still save the data in a raw |
219 | format, in many cases it's possible to replace legacy code | |
220 | with a carefully constructed VMState description that matches the | |
221 | byte layout of the existing code. | |
f58ae59c | 222 | |
edd70806 DDAG |
223 | Changing migration data structures |
224 | ---------------------------------- | |
f58ae59c | 225 | |
edd70806 DDAG |
226 | When we migrate a device, we save/load the state as a series |
227 | of fields. Sometimes, due to bugs or new functionality, we need to | |
228 | change the state to store more/different information. Changing the migration | |
229 | state saved for a device can break migration compatibility unless | |
230 | care is taken to use the appropriate techniques. In general QEMU tries | |
231 | to maintain forward migration compatibility (i.e. migrating from | |
232 | QEMU n->n+1) and there are users who benefit from backward compatibility | |
233 | as well. | |
a6c5c079 | 234 | |
2e3c8f8d DDAG |
235 | Subsections |
236 | ----------- | |
f58ae59c | 237 | |
edd70806 DDAG |
238 | The most common structure change is adding new data, e.g. when adding |
239 | a newer form of device, or adding that state that you previously | |
240 | forgot to migrate. This is best solved using a subsection. | |
f58ae59c | 241 | |
edd70806 DDAG |
242 | A subsection is "like" a device vmstate, but with a particularity, it |
243 | has a Boolean function that tells if that values are needed to be sent | |
244 | or not. If this functions returns false, the subsection is not sent. | |
245 | Subsections have a unique name, that is looked for on the receiving | |
246 | side. | |
f58ae59c JQ |
247 | |
248 | On the receiving side, if we found a subsection for a device that we | |
249 | don't understand, we just fail the migration. If we understand all | |
edd70806 DDAG |
250 | the subsections, then we load the state with success. There's no check |
251 | that a subsection is loaded, so a newer QEMU that knows about a subsection | |
252 | can (with care) load a stream from an older QEMU that didn't send | |
253 | the subsection. | |
254 | ||
255 | If the new data is only needed in a rare case, then the subsection | |
256 | can be made conditional on that case and the migration will still | |
257 | succeed to older QEMUs in most cases. This is OK for data that's | |
258 | critical, but in some use cases it's preferred that the migration | |
259 | should succeed even with the data missing. To support this the | |
260 | subsection can be connected to a device property and from there | |
261 | to a versioned machine type. | |
f58ae59c | 262 | |
3eb21fe9 DDAG |
263 | The 'pre_load' and 'post_load' functions on subsections are only |
264 | called if the subsection is loaded. | |
265 | ||
266 | One important note is that the outer post_load() function is called "after" | |
267 | loading all subsections, because a newer subsection could change the same | |
268 | value that it uses. A flag, and the combination of outer pre_load and | |
269 | post_load can be used to detect whether a subsection was loaded, and to | |
edd70806 | 270 | fall back on default behaviour when the subsection isn't present. |
f58ae59c JQ |
271 | |
272 | Example: | |
273 | ||
2e3c8f8d DDAG |
274 | .. code:: c |
275 | ||
276 | static bool ide_drive_pio_state_needed(void *opaque) | |
277 | { | |
278 | IDEState *s = opaque; | |
279 | ||
280 | return ((s->status & DRQ_STAT) != 0) | |
281 | || (s->bus->error_status & BM_STATUS_PIO_RETRY); | |
282 | } | |
283 | ||
284 | const VMStateDescription vmstate_ide_drive_pio_state = { | |
285 | .name = "ide_drive/pio_state", | |
286 | .version_id = 1, | |
287 | .minimum_version_id = 1, | |
288 | .pre_save = ide_drive_pio_pre_save, | |
289 | .post_load = ide_drive_pio_post_load, | |
290 | .needed = ide_drive_pio_state_needed, | |
291 | .fields = (VMStateField[]) { | |
292 | VMSTATE_INT32(req_nb_sectors, IDEState), | |
293 | VMSTATE_VARRAY_INT32(io_buffer, IDEState, io_buffer_total_len, 1, | |
294 | vmstate_info_uint8, uint8_t), | |
295 | VMSTATE_INT32(cur_io_buffer_offset, IDEState), | |
296 | VMSTATE_INT32(cur_io_buffer_len, IDEState), | |
297 | VMSTATE_UINT8(end_transfer_fn_idx, IDEState), | |
298 | VMSTATE_INT32(elementary_transfer_size, IDEState), | |
299 | VMSTATE_INT32(packet_transfer_size, IDEState), | |
300 | VMSTATE_END_OF_LIST() | |
301 | } | |
302 | }; | |
303 | ||
304 | const VMStateDescription vmstate_ide_drive = { | |
305 | .name = "ide_drive", | |
306 | .version_id = 3, | |
307 | .minimum_version_id = 0, | |
308 | .post_load = ide_drive_post_load, | |
309 | .fields = (VMStateField[]) { | |
310 | .... several fields .... | |
311 | VMSTATE_END_OF_LIST() | |
312 | }, | |
313 | .subsections = (const VMStateDescription*[]) { | |
314 | &vmstate_ide_drive_pio_state, | |
315 | NULL | |
316 | } | |
317 | }; | |
f58ae59c JQ |
318 | |
319 | Here we have a subsection for the pio state. We only need to | |
320 | save/send this state when we are in the middle of a pio operation | |
2e3c8f8d | 321 | (that is what ``ide_drive_pio_state_needed()`` checks). If DRQ_STAT is |
f58ae59c JQ |
322 | not enabled, the values on that fields are garbage and don't need to |
323 | be sent. | |
2bfdd1c8 | 324 | |
edd70806 DDAG |
325 | Connecting subsections to properties |
326 | ------------------------------------ | |
327 | ||
5f9412bb | 328 | Using a condition function that checks a 'property' to determine whether |
edd70806 DDAG |
329 | to send a subsection allows backward migration compatibility when |
330 | new subsections are added, especially when combined with versioned | |
331 | machine types. | |
5f9412bb | 332 | |
2e3c8f8d DDAG |
333 | For example: |
334 | ||
335 | a) Add a new property using ``DEFINE_PROP_BOOL`` - e.g. support-foo and | |
5f9412bb | 336 | default it to true. |
ac78f737 | 337 | b) Add an entry to the ``hw_compat_`` for the previous version that sets |
2e3c8f8d | 338 | the property to false. |
5f9412bb DDAG |
339 | c) Add a static bool support_foo function that tests the property. |
340 | d) Add a subsection with a .needed set to the support_foo function | |
3eb21fe9 DDAG |
341 | e) (potentially) Add an outer pre_load that sets up a default value |
342 | for 'foo' to be used if the subsection isn't loaded. | |
5f9412bb DDAG |
343 | |
344 | Now that subsection will not be generated when using an older | |
345 | machine type and the migration stream will be accepted by older | |
edd70806 | 346 | QEMU versions. |
5f9412bb | 347 | |
2e3c8f8d DDAG |
348 | Not sending existing elements |
349 | ----------------------------- | |
350 | ||
351 | Sometimes members of the VMState are no longer needed: | |
5f9412bb | 352 | |
2e3c8f8d DDAG |
353 | - removing them will break migration compatibility |
354 | ||
edd70806 DDAG |
355 | - making them version dependent and bumping the version will break backward migration |
356 | compatibility. | |
357 | ||
358 | Adding a dummy field into the migration stream is normally the best way to preserve | |
359 | compatibility. | |
5f9412bb | 360 | |
edd70806 | 361 | If the field really does need to be removed then: |
2e3c8f8d DDAG |
362 | |
363 | a) Add a new property/compatibility/function in the same way for subsections above. | |
5f9412bb | 364 | b) replace the VMSTATE macro with the _TEST version of the macro, e.g.: |
2e3c8f8d DDAG |
365 | |
366 | ``VMSTATE_UINT32(foo, barstruct)`` | |
367 | ||
5f9412bb | 368 | becomes |
5f9412bb | 369 | |
2e3c8f8d DDAG |
370 | ``VMSTATE_UINT32_TEST(foo, barstruct, pre_version_baz)`` |
371 | ||
372 | Sometime in the future when we no longer care about the ancient versions these can be killed off. | |
edd70806 DDAG |
373 | Note that for backward compatibility it's important to fill in the structure with |
374 | data that the destination will understand. | |
375 | ||
376 | Any difference in the predicates on the source and destination will end up | |
377 | with different fields being enabled and data being loaded into the wrong | |
378 | fields; for this reason conditional fields like this are very fragile. | |
379 | ||
380 | Versions | |
381 | -------- | |
382 | ||
383 | Version numbers are intended for major incompatible changes to the | |
384 | migration of a device, and using them breaks backward-migration | |
385 | compatibility; in general most changes can be made by adding Subsections | |
386 | (see above) or _TEST macros (see above) which won't break compatibility. | |
387 | ||
4df3a7bf PM |
388 | Each version is associated with a series of fields saved. The ``save_state`` always saves |
389 | the state as the newer version. But ``load_state`` sometimes is able to | |
edd70806 DDAG |
390 | load state from an older version. |
391 | ||
392 | You can see that there are several version fields: | |
393 | ||
4df3a7bf PM |
394 | - ``version_id``: the maximum version_id supported by VMState for that device. |
395 | - ``minimum_version_id``: the minimum version_id that VMState is able to understand | |
edd70806 | 396 | for that device. |
4df3a7bf | 397 | - ``minimum_version_id_old``: For devices that were not able to port to vmstate, we can |
edd70806 | 398 | assign a function that knows how to read this old state. This field is |
4df3a7bf | 399 | ignored if there is no ``load_state_old`` handler. |
edd70806 DDAG |
400 | |
401 | VMState is able to read versions from minimum_version_id to | |
402 | version_id. And the function ``load_state_old()`` (if present) is able to | |
403 | load state from minimum_version_id_old to minimum_version_id. This | |
404 | function is deprecated and will be removed when no more users are left. | |
405 | ||
406 | There are *_V* forms of many ``VMSTATE_`` macros to load fields for version dependent fields, | |
407 | e.g. | |
408 | ||
409 | .. code:: c | |
410 | ||
411 | VMSTATE_UINT16_V(ip_id, Slirp, 2), | |
412 | ||
413 | only loads that field for versions 2 and newer. | |
414 | ||
415 | Saving state will always create a section with the 'version_id' value | |
416 | and thus can't be loaded by any older QEMU. | |
417 | ||
418 | Massaging functions | |
419 | ------------------- | |
420 | ||
421 | Sometimes, it is not enough to be able to save the state directly | |
422 | from one structure, we need to fill the correct values there. One | |
423 | example is when we are using kvm. Before saving the cpu state, we | |
424 | need to ask kvm to copy to QEMU the state that it is using. And the | |
425 | opposite when we are loading the state, we need a way to tell kvm to | |
426 | load the state for the cpu that we have just loaded from the QEMUFile. | |
427 | ||
428 | The functions to do that are inside a vmstate definition, and are called: | |
429 | ||
430 | - ``int (*pre_load)(void *opaque);`` | |
431 | ||
432 | This function is called before we load the state of one device. | |
433 | ||
434 | - ``int (*post_load)(void *opaque, int version_id);`` | |
435 | ||
436 | This function is called after we load the state of one device. | |
437 | ||
438 | - ``int (*pre_save)(void *opaque);`` | |
439 | ||
440 | This function is called before we save the state of one device. | |
441 | ||
8c07559f AL |
442 | - ``int (*post_save)(void *opaque);`` |
443 | ||
444 | This function is called after we save the state of one device | |
445 | (even upon failure, unless the call to pre_save returned an error). | |
446 | ||
447 | Example: You can look at hpet.c, that uses the first three functions | |
448 | to massage the state that is transferred. | |
edd70806 DDAG |
449 | |
450 | The ``VMSTATE_WITH_TMP`` macro may be useful when the migration | |
451 | data doesn't match the stored device data well; it allows an | |
452 | intermediate temporary structure to be populated with migration | |
453 | data and then transferred to the main structure. | |
454 | ||
455 | If you use memory API functions that update memory layout outside | |
456 | initialization (i.e., in response to a guest action), this is a strong | |
4df3a7bf | 457 | indication that you need to call these functions in a ``post_load`` callback. |
edd70806 DDAG |
458 | Examples of such memory API functions are: |
459 | ||
460 | - memory_region_add_subregion() | |
461 | - memory_region_del_subregion() | |
462 | - memory_region_set_readonly() | |
c26763f8 | 463 | - memory_region_set_nonvolatile() |
edd70806 DDAG |
464 | - memory_region_set_enabled() |
465 | - memory_region_set_address() | |
466 | - memory_region_set_alias_offset() | |
467 | ||
468 | Iterative device migration | |
469 | -------------------------- | |
470 | ||
471 | Some devices, such as RAM, Block storage or certain platform devices, | |
472 | have large amounts of data that would mean that the CPUs would be | |
473 | paused for too long if they were sent in one section. For these | |
474 | devices an *iterative* approach is taken. | |
475 | ||
476 | The iterative devices generally don't use VMState macros | |
477 | (although it may be possible in some cases) and instead use | |
478 | qemu_put_*/qemu_get_* macros to read/write data to the stream. Specialist | |
479 | versions exist for high bandwidth IO. | |
480 | ||
481 | ||
482 | An iterative device must provide: | |
483 | ||
484 | - A ``save_setup`` function that initialises the data structures and | |
485 | transmits a first section containing information on the device. In the | |
486 | case of RAM this transmits a list of RAMBlocks and sizes. | |
487 | ||
488 | - A ``load_setup`` function that initialises the data structures on the | |
489 | destination. | |
490 | ||
491 | - A ``save_live_pending`` function that is called repeatedly and must | |
492 | indicate how much more data the iterative data must save. The core | |
493 | migration code will use this to determine when to pause the CPUs | |
494 | and complete the migration. | |
495 | ||
496 | - A ``save_live_iterate`` function (called after ``save_live_pending`` | |
497 | when there is significant data still to be sent). It should send | |
498 | a chunk of data until the point that stream bandwidth limits tell it | |
499 | to stop. Each call generates one section. | |
500 | ||
501 | - A ``save_live_complete_precopy`` function that must transmit the | |
502 | last section for the device containing any remaining data. | |
503 | ||
504 | - A ``load_state`` function used to load sections generated by | |
505 | any of the save functions that generate sections. | |
506 | ||
507 | - ``cleanup`` functions for both save and load that are called | |
508 | at the end of migration. | |
509 | ||
510 | Note that the contents of the sections for iterative migration tend | |
511 | to be open-coded by the devices; care should be taken in parsing | |
512 | the results and structuring the stream to make them easy to validate. | |
513 | ||
514 | Device ordering | |
515 | --------------- | |
516 | ||
517 | There are cases in which the ordering of device loading matters; for | |
518 | example in some systems where a device may assert an interrupt during loading, | |
519 | if the interrupt controller is loaded later then it might lose the state. | |
520 | ||
521 | Some ordering is implicitly provided by the order in which the machine | |
522 | definition creates devices, however this is somewhat fragile. | |
523 | ||
524 | The ``MigrationPriority`` enum provides a means of explicitly enforcing | |
525 | ordering. Numerically higher priorities are loaded earlier. | |
526 | The priority is set by setting the ``priority`` field of the top level | |
527 | ``VMStateDescription`` for the device. | |
528 | ||
529 | Stream structure | |
530 | ================ | |
531 | ||
532 | The stream tries to be word and endian agnostic, allowing migration between hosts | |
533 | of different characteristics running the same VM. | |
534 | ||
535 | - Header | |
536 | ||
537 | - Magic | |
538 | - Version | |
539 | - VM configuration section | |
540 | ||
541 | - Machine type | |
542 | - Target page bits | |
543 | - List of sections | |
544 | Each section contains a device, or one iteration of a device save. | |
545 | ||
546 | - section type | |
547 | - section id | |
548 | - ID string (First section of each device) | |
549 | - instance id (First section of each device) | |
550 | - version id (First section of each device) | |
551 | - <device data> | |
552 | - Footer mark | |
553 | - EOF mark | |
554 | - VM Description structure | |
555 | Consisting of a JSON description of the contents for analysis only | |
556 | ||
557 | The ``device data`` in each section consists of the data produced | |
558 | by the code described above. For non-iterative devices they have a single | |
559 | section; iterative devices have an initial and last section and a set | |
560 | of parts in between. | |
561 | Note that there is very little checking by the common code of the integrity | |
562 | of the ``device data`` contents, that's up to the devices themselves. | |
563 | The ``footer mark`` provides a little bit of protection for the case where | |
564 | the receiving side reads more or less data than expected. | |
565 | ||
566 | The ``ID string`` is normally unique, having been formed from a bus name | |
567 | and device address, PCI devices and storage devices hung off PCI controllers | |
568 | fit this pattern well. Some devices are fixed single instances (e.g. "pc-ram"). | |
569 | Others (especially either older devices or system devices which for | |
570 | some reason don't have a bus concept) make use of the ``instance id`` | |
571 | for otherwise identically named devices. | |
5f9412bb | 572 | |
2e3c8f8d DDAG |
573 | Return path |
574 | ----------- | |
2bfdd1c8 | 575 | |
edd70806 DDAG |
576 | Only a unidirectional stream is required for normal migration, however a |
577 | ``return path`` can be created when bidirectional communication is desired. | |
578 | This is primarily used by postcopy, but is also used to return a success | |
579 | flag to the source at the end of migration. | |
2bfdd1c8 | 580 | |
2e3c8f8d | 581 | ``qemu_file_get_return_path(QEMUFile* fwdpath)`` gives the QEMUFile* for the return |
2bfdd1c8 DDAG |
582 | path. |
583 | ||
584 | Source side | |
2e3c8f8d | 585 | |
2bfdd1c8 DDAG |
586 | Forward path - written by migration thread |
587 | Return path - opened by main thread, read by return-path thread | |
588 | ||
589 | Destination side | |
2e3c8f8d | 590 | |
2bfdd1c8 DDAG |
591 | Forward path - read by main thread |
592 | Return path - opened by main thread, written by main thread AND postcopy | |
2e3c8f8d DDAG |
593 | thread (protected by rp_mutex) |
594 | ||
595 | Postcopy | |
596 | ======== | |
2bfdd1c8 | 597 | |
2bfdd1c8 DDAG |
598 | 'Postcopy' migration is a way to deal with migrations that refuse to converge |
599 | (or take too long to converge) its plus side is that there is an upper bound on | |
600 | the amount of migration traffic and time it takes, the down side is that during | |
601 | the postcopy phase, a failure of *either* side or the network connection causes | |
602 | the guest to be lost. | |
603 | ||
604 | In postcopy the destination CPUs are started before all the memory has been | |
605 | transferred, and accesses to pages that are yet to be transferred cause | |
606 | a fault that's translated by QEMU into a request to the source QEMU. | |
607 | ||
608 | Postcopy can be combined with precopy (i.e. normal migration) so that if precopy | |
609 | doesn't finish in a given time the switch is made to postcopy. | |
610 | ||
2e3c8f8d DDAG |
611 | Enabling postcopy |
612 | ----------------- | |
2bfdd1c8 | 613 | |
c2eb7f21 GK |
614 | To enable postcopy, issue this command on the monitor (both source and |
615 | destination) prior to the start of migration: | |
2bfdd1c8 | 616 | |
2e3c8f8d | 617 | ``migrate_set_capability postcopy-ram on`` |
2bfdd1c8 DDAG |
618 | |
619 | The normal commands are then used to start a migration, which is still | |
620 | started in precopy mode. Issuing: | |
621 | ||
2e3c8f8d | 622 | ``migrate_start_postcopy`` |
2bfdd1c8 DDAG |
623 | |
624 | will now cause the transition from precopy to postcopy. | |
625 | It can be issued immediately after migration is started or any | |
626 | time later on. Issuing it after the end of a migration is harmless. | |
627 | ||
9ed01779 | 628 | Blocktime is a postcopy live migration metric, intended to show how |
76ca4b58 | 629 | long the vCPU was in state of interruptible sleep due to pagefault. |
9ed01779 AP |
630 | That metric is calculated both for all vCPUs as overlapped value, and |
631 | separately for each vCPU. These values are calculated on destination | |
632 | side. To enable postcopy blocktime calculation, enter following | |
633 | command on destination monitor: | |
634 | ||
635 | ``migrate_set_capability postcopy-blocktime on`` | |
636 | ||
637 | Postcopy blocktime can be retrieved by query-migrate qmp command. | |
638 | postcopy-blocktime value of qmp command will show overlapped blocking | |
639 | time for all vCPU, postcopy-vcpu-blocktime will show list of blocking | |
640 | time per vCPU. | |
641 | ||
2e3c8f8d DDAG |
642 | .. note:: |
643 | During the postcopy phase, the bandwidth limits set using | |
cbde7be9 | 644 | ``migrate_set_parameter`` is ignored (to avoid delaying requested pages that |
2e3c8f8d | 645 | the destination is waiting for). |
2bfdd1c8 | 646 | |
2e3c8f8d DDAG |
647 | Postcopy device transfer |
648 | ------------------------ | |
2bfdd1c8 DDAG |
649 | |
650 | Loading of device data may cause the device emulation to access guest RAM | |
651 | that may trigger faults that have to be resolved by the source, as such | |
652 | the migration stream has to be able to respond with page data *during* the | |
653 | device load, and hence the device data has to be read from the stream completely | |
654 | before the device load begins to free the stream up. This is achieved by | |
655 | 'packaging' the device data into a blob that's read in one go. | |
656 | ||
657 | Source behaviour | |
2e3c8f8d | 658 | ---------------- |
2bfdd1c8 DDAG |
659 | |
660 | Until postcopy is entered the migration stream is identical to normal | |
661 | precopy, except for the addition of a 'postcopy advise' command at | |
662 | the beginning, to tell the destination that postcopy might happen. | |
663 | When postcopy starts the source sends the page discard data and then | |
664 | forms the 'package' containing: | |
665 | ||
2e3c8f8d DDAG |
666 | - Command: 'postcopy listen' |
667 | - The device state | |
2bfdd1c8 | 668 | |
2e3c8f8d DDAG |
669 | A series of sections, identical to the precopy streams device state stream |
670 | containing everything except postcopiable devices (i.e. RAM) | |
671 | - Command: 'postcopy run' | |
672 | ||
673 | The 'package' is sent as the data part of a Command: ``CMD_PACKAGED``, and the | |
2bfdd1c8 DDAG |
674 | contents are formatted in the same way as the main migration stream. |
675 | ||
676 | During postcopy the source scans the list of dirty pages and sends them | |
677 | to the destination without being requested (in much the same way as precopy), | |
678 | however when a page request is received from the destination, the dirty page | |
679 | scanning restarts from the requested location. This causes requested pages | |
680 | to be sent quickly, and also causes pages directly after the requested page | |
681 | to be sent quickly in the hope that those pages are likely to be used | |
682 | by the destination soon. | |
683 | ||
684 | Destination behaviour | |
2e3c8f8d | 685 | --------------------- |
2bfdd1c8 DDAG |
686 | |
687 | Initially the destination looks the same as precopy, with a single thread | |
688 | reading the migration stream; the 'postcopy advise' and 'discard' commands | |
689 | are processed to change the way RAM is managed, but don't affect the stream | |
690 | processing. | |
691 | ||
2e3c8f8d DDAG |
692 | :: |
693 | ||
694 | ------------------------------------------------------------------------------ | |
695 | 1 2 3 4 5 6 7 | |
696 | main -----DISCARD-CMD_PACKAGED ( LISTEN DEVICE DEVICE DEVICE RUN ) | |
697 | thread | | | |
698 | | (page request) | |
699 | | \___ | |
700 | v \ | |
701 | listen thread: --- page -- page -- page -- page -- page -- | |
702 | ||
703 | a b c | |
704 | ------------------------------------------------------------------------------ | |
705 | ||
706 | - On receipt of ``CMD_PACKAGED`` (1) | |
707 | ||
708 | All the data associated with the package - the ( ... ) section in the diagram - | |
709 | is read into memory, and the main thread recurses into qemu_loadvm_state_main | |
710 | to process the contents of the package (2) which contains commands (3,6) and | |
711 | devices (4...) | |
712 | ||
713 | - On receipt of 'postcopy listen' - 3 -(i.e. the 1st command in the package) | |
714 | ||
715 | a new thread (a) is started that takes over servicing the migration stream, | |
716 | while the main thread carries on loading the package. It loads normal | |
717 | background page data (b) but if during a device load a fault happens (5) | |
718 | the returned page (c) is loaded by the listen thread allowing the main | |
719 | threads device load to carry on. | |
720 | ||
721 | - The last thing in the ``CMD_PACKAGED`` is a 'RUN' command (6) | |
722 | ||
723 | letting the destination CPUs start running. At the end of the | |
724 | ``CMD_PACKAGED`` (7) the main thread returns to normal running behaviour and | |
725 | is no longer used by migration, while the listen thread carries on servicing | |
726 | page data until the end of migration. | |
727 | ||
728 | Postcopy states | |
729 | --------------- | |
2bfdd1c8 DDAG |
730 | |
731 | Postcopy moves through a series of states (see postcopy_state) from | |
732 | ADVISE->DISCARD->LISTEN->RUNNING->END | |
733 | ||
2e3c8f8d DDAG |
734 | - Advise |
735 | ||
736 | Set at the start of migration if postcopy is enabled, even | |
737 | if it hasn't had the start command; here the destination | |
738 | checks that its OS has the support needed for postcopy, and performs | |
739 | setup to ensure the RAM mappings are suitable for later postcopy. | |
740 | The destination will fail early in migration at this point if the | |
741 | required OS support is not present. | |
742 | (Triggered by reception of POSTCOPY_ADVISE command) | |
743 | ||
744 | - Discard | |
745 | ||
746 | Entered on receipt of the first 'discard' command; prior to | |
747 | the first Discard being performed, hugepages are switched off | |
748 | (using madvise) to ensure that no new huge pages are created | |
749 | during the postcopy phase, and to cause any huge pages that | |
750 | have discards on them to be broken. | |
751 | ||
752 | - Listen | |
753 | ||
754 | The first command in the package, POSTCOPY_LISTEN, switches | |
755 | the destination state to Listen, and starts a new thread | |
756 | (the 'listen thread') which takes over the job of receiving | |
757 | pages off the migration stream, while the main thread carries | |
758 | on processing the blob. With this thread able to process page | |
759 | reception, the destination now 'sensitises' the RAM to detect | |
760 | any access to missing pages (on Linux using the 'userfault' | |
761 | system). | |
762 | ||
763 | - Running | |
764 | ||
765 | POSTCOPY_RUN causes the destination to synchronise all | |
766 | state and start the CPUs and IO devices running. The main | |
767 | thread now finishes processing the migration package and | |
768 | now carries on as it would for normal precopy migration | |
769 | (although it can't do the cleanup it would do as it | |
770 | finishes a normal migration). | |
771 | ||
772 | - End | |
773 | ||
774 | The listen thread can now quit, and perform the cleanup of migration | |
775 | state, the migration is now complete. | |
776 | ||
777 | Source side page maps | |
778 | --------------------- | |
2bfdd1c8 DDAG |
779 | |
780 | The source side keeps two bitmaps during postcopy; 'the migration bitmap' | |
781 | and 'unsent map'. The 'migration bitmap' is basically the same as in | |
782 | the precopy case, and holds a bit to indicate that page is 'dirty' - | |
783 | i.e. needs sending. During the precopy phase this is updated as the CPU | |
784 | dirties pages, however during postcopy the CPUs are stopped and nothing | |
785 | should dirty anything any more. | |
786 | ||
787 | The 'unsent map' is used for the transition to postcopy. It is a bitmap that | |
788 | has a bit cleared whenever a page is sent to the destination, however during | |
789 | the transition to postcopy mode it is combined with the migration bitmap | |
790 | to form a set of pages that: | |
2e3c8f8d | 791 | |
2bfdd1c8 DDAG |
792 | a) Have been sent but then redirtied (which must be discarded) |
793 | b) Have not yet been sent - which also must be discarded to cause any | |
794 | transparent huge pages built during precopy to be broken. | |
795 | ||
796 | Note that the contents of the unsentmap are sacrificed during the calculation | |
797 | of the discard set and thus aren't valid once in postcopy. The dirtymap | |
798 | is still valid and is used to ensure that no page is sent more than once. Any | |
799 | request for a page that has already been sent is ignored. Duplicate requests | |
800 | such as this can happen as a page is sent at about the same time the | |
801 | destination accesses it. | |
802 | ||
2e3c8f8d DDAG |
803 | Postcopy with hugepages |
804 | ----------------------- | |
0c1f4036 DDAG |
805 | |
806 | Postcopy now works with hugetlbfs backed memory: | |
2e3c8f8d | 807 | |
0c1f4036 DDAG |
808 | a) The linux kernel on the destination must support userfault on hugepages. |
809 | b) The huge-page configuration on the source and destination VMs must be | |
810 | identical; i.e. RAMBlocks on both sides must use the same page size. | |
2e3c8f8d | 811 | c) Note that ``-mem-path /dev/hugepages`` will fall back to allocating normal |
0c1f4036 | 812 | RAM if it doesn't have enough hugepages, triggering (b) to fail. |
2e3c8f8d | 813 | Using ``-mem-prealloc`` enforces the allocation using hugepages. |
0c1f4036 DDAG |
814 | d) Care should be taken with the size of hugepage used; postcopy with 2MB |
815 | hugepages works well, however 1GB hugepages are likely to be problematic | |
816 | since it takes ~1 second to transfer a 1GB hugepage across a 10Gbps link, | |
817 | and until the full page is transferred the destination thread is blocked. | |
1dc61e7b DDAG |
818 | |
819 | Postcopy with shared memory | |
820 | --------------------------- | |
821 | ||
822 | Postcopy migration with shared memory needs explicit support from the other | |
823 | processes that share memory and from QEMU. There are restrictions on the type of | |
824 | memory that userfault can support shared. | |
825 | ||
4df3a7bf PM |
826 | The Linux kernel userfault support works on ``/dev/shm`` memory and on ``hugetlbfs`` |
827 | (although the kernel doesn't provide an equivalent to ``madvise(MADV_DONTNEED)`` | |
1dc61e7b DDAG |
828 | for hugetlbfs which may be a problem in some configurations). |
829 | ||
830 | The vhost-user code in QEMU supports clients that have Postcopy support, | |
4df3a7bf | 831 | and the ``vhost-user-bridge`` (in ``tests/``) and the DPDK package have changes |
1dc61e7b DDAG |
832 | to support postcopy. |
833 | ||
834 | The client needs to open a userfaultfd and register the areas | |
835 | of memory that it maps with userfault. The client must then pass the | |
836 | userfaultfd back to QEMU together with a mapping table that allows | |
837 | fault addresses in the clients address space to be converted back to | |
838 | RAMBlock/offsets. The client's userfaultfd is added to the postcopy | |
839 | fault-thread and page requests are made on behalf of the client by QEMU. | |
840 | QEMU performs 'wake' operations on the client's userfaultfd to allow it | |
841 | to continue after a page has arrived. | |
842 | ||
843 | .. note:: | |
844 | There are two future improvements that would be nice: | |
845 | a) Some way to make QEMU ignorant of the addresses in the clients | |
846 | address space | |
847 | b) Avoiding the need for QEMU to perform ufd-wake calls after the | |
848 | pages have arrived | |
849 | ||
850 | Retro-fitting postcopy to existing clients is possible: | |
851 | a) A mechanism is needed for the registration with userfault as above, | |
852 | and the registration needs to be coordinated with the phases of | |
853 | postcopy. In vhost-user extra messages are added to the existing | |
854 | control channel. | |
855 | b) Any thread that can block due to guest memory accesses must be | |
856 | identified and the implication understood; for example if the | |
857 | guest memory access is made while holding a lock then all other | |
858 | threads waiting for that lock will also be blocked. | |
edd70806 DDAG |
859 | |
860 | Firmware | |
861 | ======== | |
862 | ||
863 | Migration migrates the copies of RAM and ROM, and thus when running | |
864 | on the destination it includes the firmware from the source. Even after | |
865 | resetting a VM, the old firmware is used. Only once QEMU has been restarted | |
866 | is the new firmware in use. | |
867 | ||
868 | - Changes in firmware size can cause changes in the required RAMBlock size | |
869 | to hold the firmware and thus migration can fail. In practice it's best | |
870 | to pad firmware images to convenient powers of 2 with plenty of space | |
871 | for growth. | |
872 | ||
873 | - Care should be taken with device emulation code so that newer | |
874 | emulation code can work with older firmware to allow forward migration. | |
875 | ||
876 | - Care should be taken with newer firmware so that backward migration | |
877 | to older systems with older device emulation code will work. | |
878 | ||
879 | In some cases it may be best to tie specific firmware versions to specific | |
880 | versioned machine types to cut down on the combinations that will need | |
881 | support. This is also useful when newer versions of firmware outgrow | |
882 | the padding. | |
883 |