]>
Commit | Line | Data |
---|---|---|
2e3c8f8d DDAG |
1 | ========= |
2 | Migration | |
3 | ========= | |
f58ae59c JQ |
4 | |
5 | QEMU has code to load/save the state of the guest that it is running. | |
dda5336e | 6 | These are two complementary operations. Saving the state just does |
f58ae59c JQ |
7 | that, saves the state for each device that the guest is running. |
8 | Restoring a guest is just the opposite operation: we need to load the | |
9 | state of each device. | |
10 | ||
dda5336e | 11 | For this to work, QEMU has to be launched with the same arguments the |
f58ae59c JQ |
12 | two times. I.e. it can only restore the state in one guest that has |
13 | the same devices that the one it was saved (this last requirement can | |
dda5336e | 14 | be relaxed a bit, but for now we can consider that configuration has |
f58ae59c JQ |
15 | to be exactly the same). |
16 | ||
17 | Once that we are able to save/restore a guest, a new functionality is | |
18 | requested: migration. This means that QEMU is able to start in one | |
dda5336e SW |
19 | machine and being "migrated" to another machine. I.e. being moved to |
20 | another machine. | |
f58ae59c JQ |
21 | |
22 | Next was the "live migration" functionality. This is important | |
23 | because some guests run with a lot of state (specially RAM), and it | |
24 | can take a while to move all state from one machine to another. Live | |
25 | migration allows the guest to continue running while the state is | |
26 | transferred. Only while the last part of the state is transferred has | |
27 | the guest to be stopped. Typically the time that the guest is | |
28 | unresponsive during live migration is the low hundred of milliseconds | |
dda5336e | 29 | (notice that this depends on a lot of things). |
f58ae59c | 30 | |
d8a0f054 JQ |
31 | .. contents:: |
32 | ||
edd70806 DDAG |
33 | Transports |
34 | ========== | |
f58ae59c | 35 | |
edd70806 DDAG |
36 | The migration stream is normally just a byte stream that can be passed |
37 | over any transport. | |
f58ae59c JQ |
38 | |
39 | - tcp migration: do the migration using tcp sockets | |
40 | - unix migration: do the migration using unix sockets | |
41 | - exec migration: do the migration using the stdin/stdout through a process. | |
9277d81f | 42 | - fd migration: do the migration using a file descriptor that is |
dda5336e | 43 | passed to QEMU. QEMU doesn't care how this file descriptor is opened. |
f58ae59c | 44 | |
edd70806 DDAG |
45 | In addition, support is included for migration using RDMA, which |
46 | transports the page data using ``RDMA``, where the hardware takes care of | |
47 | transporting the pages, and the load on the CPU is much lower. While the | |
48 | internals of RDMA migration are a bit different, this isn't really visible | |
49 | outside the RAM migration code. | |
50 | ||
51 | All these migration protocols use the same infrastructure to | |
f58ae59c JQ |
52 | save/restore state devices. This infrastructure is shared with the |
53 | savevm/loadvm functionality. | |
54 | ||
979da8b3 MAL |
55 | Debugging |
56 | ========= | |
57 | ||
4df3a7bf | 58 | The migration stream can be analyzed thanks to ``scripts/analyze-migration.py``. |
979da8b3 MAL |
59 | |
60 | Example usage: | |
61 | ||
62 | .. code-block:: shell | |
63 | ||
243e7480 MA |
64 | $ qemu-system-x86_64 -display none -monitor stdio |
65 | (qemu) migrate "exec:cat > mig" | |
66 | (qemu) q | |
67 | $ ./scripts/analyze-migration.py -f mig | |
979da8b3 MAL |
68 | { |
69 | "ram (3)": { | |
70 | "section sizes": { | |
71 | "pc.ram": "0x0000000008000000", | |
72 | ... | |
73 | ||
243e7480 | 74 | See also ``analyze-migration.py -h`` help for more options. |
979da8b3 | 75 | |
2e3c8f8d DDAG |
76 | Common infrastructure |
77 | ===================== | |
f58ae59c | 78 | |
2e3c8f8d | 79 | The files, sockets or fd's that carry the migration stream are abstracted by |
4df3a7bf PM |
80 | the ``QEMUFile`` type (see ``migration/qemu-file.h``). In most cases this |
81 | is connected to a subtype of ``QIOChannel`` (see ``io/``). | |
f58ae59c | 82 | |
edd70806 | 83 | |
2e3c8f8d DDAG |
84 | Saving the state of one device |
85 | ============================== | |
f58ae59c | 86 | |
edd70806 DDAG |
87 | For most devices, the state is saved in a single call to the migration |
88 | infrastructure; these are *non-iterative* devices. The data for these | |
89 | devices is sent at the end of precopy migration, when the CPUs are paused. | |
90 | There are also *iterative* devices, which contain a very large amount of | |
91 | data (e.g. RAM or large tables). See the iterative device section below. | |
92 | ||
93 | General advice for device developers | |
94 | ------------------------------------ | |
95 | ||
96 | - The migration state saved should reflect the device being modelled rather | |
97 | than the way your implementation works. That way if you change the implementation | |
98 | later the migration stream will stay compatible. That model may include | |
99 | internal state that's not directly visible in a register. | |
100 | ||
101 | - When saving a migration stream the device code may walk and check | |
102 | the state of the device. These checks might fail in various ways (e.g. | |
103 | discovering internal state is corrupt or that the guest has done something bad). | |
104 | Consider carefully before asserting/aborting at this point, since the | |
105 | normal response from users is that *migration broke their VM* since it had | |
106 | apparently been running fine until then. In these error cases, the device | |
107 | should log a message indicating the cause of error, and should consider | |
108 | putting the device into an error state, allowing the rest of the VM to | |
109 | continue execution. | |
110 | ||
111 | - The migration might happen at an inconvenient point, | |
112 | e.g. right in the middle of the guest reprogramming the device, during | |
113 | guest reboot or shutdown or while the device is waiting for external IO. | |
114 | It's strongly preferred that migrations do not fail in this situation, | |
115 | since in the cloud environment migrations might happen automatically to | |
116 | VMs that the administrator doesn't directly control. | |
117 | ||
118 | - If you do need to fail a migration, ensure that sufficient information | |
119 | is logged to identify what went wrong. | |
120 | ||
121 | - The destination should treat an incoming migration stream as hostile | |
122 | (which we do to varying degrees in the existing code). Check that offsets | |
123 | into buffers and the like can't cause overruns. Fail the incoming migration | |
124 | in the case of a corrupted stream like this. | |
125 | ||
126 | - Take care with internal device state or behaviour that might become | |
127 | migration version dependent. For example, the order of PCI capabilities | |
128 | is required to stay constant across migration. Another example would | |
129 | be that a special case handled by subsections (see below) might become | |
130 | much more common if a default behaviour is changed. | |
131 | ||
132 | - The state of the source should not be changed or destroyed by the | |
133 | outgoing migration. Migrations timing out or being failed by | |
134 | higher levels of management, or failures of the destination host are | |
135 | not unusual, and in that case the VM is restarted on the source. | |
136 | Note that the management layer can validly revert the migration | |
137 | even though the QEMU level of migration has succeeded as long as it | |
138 | does it before starting execution on the destination. | |
139 | ||
140 | - Buses and devices should be able to explicitly specify addresses when | |
141 | instantiated, and management tools should use those. For example, | |
142 | when hot adding USB devices it's important to specify the ports | |
143 | and addresses, since implicit ordering based on the command line order | |
144 | may be different on the destination. This can result in the | |
145 | device state being loaded into the wrong device. | |
f58ae59c | 146 | |
2e3c8f8d DDAG |
147 | VMState |
148 | ------- | |
f58ae59c | 149 | |
edd70806 DDAG |
150 | Most device data can be described using the ``VMSTATE`` macros (mostly defined |
151 | in ``include/migration/vmstate.h``). | |
f58ae59c | 152 | |
7465dfec | 153 | An example (from hw/input/pckbd.c) |
f58ae59c | 154 | |
2e3c8f8d DDAG |
155 | .. code:: c |
156 | ||
157 | static const VMStateDescription vmstate_kbd = { | |
158 | .name = "pckbd", | |
159 | .version_id = 3, | |
160 | .minimum_version_id = 3, | |
161 | .fields = (VMStateField[]) { | |
162 | VMSTATE_UINT8(write_cmd, KBDState), | |
163 | VMSTATE_UINT8(status, KBDState), | |
164 | VMSTATE_UINT8(mode, KBDState), | |
165 | VMSTATE_UINT8(pending, KBDState), | |
166 | VMSTATE_END_OF_LIST() | |
167 | } | |
168 | }; | |
f58ae59c JQ |
169 | |
170 | We are declaring the state with name "pckbd". | |
4df3a7bf | 171 | The ``version_id`` is 3, and the fields are 4 uint8_t in a KBDState structure. |
f58ae59c JQ |
172 | We registered this with: |
173 | ||
2e3c8f8d DDAG |
174 | .. code:: c |
175 | ||
f58ae59c JQ |
176 | vmstate_register(NULL, 0, &vmstate_kbd, s); |
177 | ||
4df3a7bf | 178 | For devices that are ``qdev`` based, we can register the device in the class |
edd70806 | 179 | init function: |
f58ae59c | 180 | |
edd70806 | 181 | .. code:: c |
f58ae59c | 182 | |
edd70806 | 183 | dc->vmsd = &vmstate_kbd_isa; |
f58ae59c | 184 | |
edd70806 DDAG |
185 | The VMState macros take care of ensuring that the device data section |
186 | is formatted portably (normally big endian) and make some compile time checks | |
187 | against the types of the fields in the structures. | |
f58ae59c | 188 | |
edd70806 DDAG |
189 | VMState macros can include other VMStateDescriptions to store substructures |
190 | (see ``VMSTATE_STRUCT_``), arrays (``VMSTATE_ARRAY_``) and variable length | |
191 | arrays (``VMSTATE_VARRAY_``). Various other macros exist for special | |
192 | cases. | |
5f9412bb | 193 | |
edd70806 DDAG |
194 | Note that the format on the wire is still very raw; i.e. a VMSTATE_UINT32 |
195 | ends up with a 4 byte bigendian representation on the wire; in the future | |
196 | it might be possible to use a more structured format. | |
f58ae59c | 197 | |
edd70806 DDAG |
198 | Legacy way |
199 | ---------- | |
f58ae59c | 200 | |
edd70806 DDAG |
201 | This way is going to disappear as soon as all current users are ported to VMSTATE; |
202 | although converting existing code can be tricky, and thus 'soon' is relative. | |
f58ae59c | 203 | |
edd70806 DDAG |
204 | Each device has to register two functions, one to save the state and |
205 | another to load the state back. | |
f58ae59c | 206 | |
edd70806 | 207 | .. code:: c |
f58ae59c | 208 | |
ce62df53 | 209 | int register_savevm_live(const char *idstr, |
edd70806 DDAG |
210 | int instance_id, |
211 | int version_id, | |
212 | SaveVMHandlers *ops, | |
213 | void *opaque); | |
f58ae59c | 214 | |
4df3a7bf PM |
215 | Two functions in the ``ops`` structure are the ``save_state`` |
216 | and ``load_state`` functions. Notice that ``load_state`` receives a version_id | |
217 | parameter to know what state format is receiving. ``save_state`` doesn't | |
edd70806 | 218 | have a version_id parameter because it always uses the latest version. |
f58ae59c | 219 | |
edd70806 DDAG |
220 | Note that because the VMState macros still save the data in a raw |
221 | format, in many cases it's possible to replace legacy code | |
222 | with a carefully constructed VMState description that matches the | |
223 | byte layout of the existing code. | |
f58ae59c | 224 | |
edd70806 DDAG |
225 | Changing migration data structures |
226 | ---------------------------------- | |
f58ae59c | 227 | |
edd70806 DDAG |
228 | When we migrate a device, we save/load the state as a series |
229 | of fields. Sometimes, due to bugs or new functionality, we need to | |
230 | change the state to store more/different information. Changing the migration | |
231 | state saved for a device can break migration compatibility unless | |
232 | care is taken to use the appropriate techniques. In general QEMU tries | |
233 | to maintain forward migration compatibility (i.e. migrating from | |
234 | QEMU n->n+1) and there are users who benefit from backward compatibility | |
235 | as well. | |
a6c5c079 | 236 | |
2e3c8f8d DDAG |
237 | Subsections |
238 | ----------- | |
f58ae59c | 239 | |
edd70806 DDAG |
240 | The most common structure change is adding new data, e.g. when adding |
241 | a newer form of device, or adding that state that you previously | |
242 | forgot to migrate. This is best solved using a subsection. | |
f58ae59c | 243 | |
edd70806 DDAG |
244 | A subsection is "like" a device vmstate, but with a particularity, it |
245 | has a Boolean function that tells if that values are needed to be sent | |
246 | or not. If this functions returns false, the subsection is not sent. | |
247 | Subsections have a unique name, that is looked for on the receiving | |
248 | side. | |
f58ae59c JQ |
249 | |
250 | On the receiving side, if we found a subsection for a device that we | |
251 | don't understand, we just fail the migration. If we understand all | |
edd70806 DDAG |
252 | the subsections, then we load the state with success. There's no check |
253 | that a subsection is loaded, so a newer QEMU that knows about a subsection | |
254 | can (with care) load a stream from an older QEMU that didn't send | |
255 | the subsection. | |
256 | ||
257 | If the new data is only needed in a rare case, then the subsection | |
258 | can be made conditional on that case and the migration will still | |
259 | succeed to older QEMUs in most cases. This is OK for data that's | |
260 | critical, but in some use cases it's preferred that the migration | |
261 | should succeed even with the data missing. To support this the | |
262 | subsection can be connected to a device property and from there | |
263 | to a versioned machine type. | |
f58ae59c | 264 | |
3eb21fe9 DDAG |
265 | The 'pre_load' and 'post_load' functions on subsections are only |
266 | called if the subsection is loaded. | |
267 | ||
268 | One important note is that the outer post_load() function is called "after" | |
269 | loading all subsections, because a newer subsection could change the same | |
270 | value that it uses. A flag, and the combination of outer pre_load and | |
271 | post_load can be used to detect whether a subsection was loaded, and to | |
edd70806 | 272 | fall back on default behaviour when the subsection isn't present. |
f58ae59c JQ |
273 | |
274 | Example: | |
275 | ||
2e3c8f8d DDAG |
276 | .. code:: c |
277 | ||
278 | static bool ide_drive_pio_state_needed(void *opaque) | |
279 | { | |
280 | IDEState *s = opaque; | |
281 | ||
282 | return ((s->status & DRQ_STAT) != 0) | |
283 | || (s->bus->error_status & BM_STATUS_PIO_RETRY); | |
284 | } | |
285 | ||
286 | const VMStateDescription vmstate_ide_drive_pio_state = { | |
287 | .name = "ide_drive/pio_state", | |
288 | .version_id = 1, | |
289 | .minimum_version_id = 1, | |
290 | .pre_save = ide_drive_pio_pre_save, | |
291 | .post_load = ide_drive_pio_post_load, | |
292 | .needed = ide_drive_pio_state_needed, | |
293 | .fields = (VMStateField[]) { | |
294 | VMSTATE_INT32(req_nb_sectors, IDEState), | |
295 | VMSTATE_VARRAY_INT32(io_buffer, IDEState, io_buffer_total_len, 1, | |
296 | vmstate_info_uint8, uint8_t), | |
297 | VMSTATE_INT32(cur_io_buffer_offset, IDEState), | |
298 | VMSTATE_INT32(cur_io_buffer_len, IDEState), | |
299 | VMSTATE_UINT8(end_transfer_fn_idx, IDEState), | |
300 | VMSTATE_INT32(elementary_transfer_size, IDEState), | |
301 | VMSTATE_INT32(packet_transfer_size, IDEState), | |
302 | VMSTATE_END_OF_LIST() | |
303 | } | |
304 | }; | |
305 | ||
306 | const VMStateDescription vmstate_ide_drive = { | |
307 | .name = "ide_drive", | |
308 | .version_id = 3, | |
309 | .minimum_version_id = 0, | |
310 | .post_load = ide_drive_post_load, | |
311 | .fields = (VMStateField[]) { | |
312 | .... several fields .... | |
313 | VMSTATE_END_OF_LIST() | |
314 | }, | |
315 | .subsections = (const VMStateDescription*[]) { | |
316 | &vmstate_ide_drive_pio_state, | |
317 | NULL | |
318 | } | |
319 | }; | |
f58ae59c JQ |
320 | |
321 | Here we have a subsection for the pio state. We only need to | |
322 | save/send this state when we are in the middle of a pio operation | |
2e3c8f8d | 323 | (that is what ``ide_drive_pio_state_needed()`` checks). If DRQ_STAT is |
f58ae59c JQ |
324 | not enabled, the values on that fields are garbage and don't need to |
325 | be sent. | |
2bfdd1c8 | 326 | |
edd70806 DDAG |
327 | Connecting subsections to properties |
328 | ------------------------------------ | |
329 | ||
5f9412bb | 330 | Using a condition function that checks a 'property' to determine whether |
edd70806 DDAG |
331 | to send a subsection allows backward migration compatibility when |
332 | new subsections are added, especially when combined with versioned | |
333 | machine types. | |
5f9412bb | 334 | |
2e3c8f8d DDAG |
335 | For example: |
336 | ||
337 | a) Add a new property using ``DEFINE_PROP_BOOL`` - e.g. support-foo and | |
5f9412bb | 338 | default it to true. |
ac78f737 | 339 | b) Add an entry to the ``hw_compat_`` for the previous version that sets |
2e3c8f8d | 340 | the property to false. |
5f9412bb DDAG |
341 | c) Add a static bool support_foo function that tests the property. |
342 | d) Add a subsection with a .needed set to the support_foo function | |
3eb21fe9 DDAG |
343 | e) (potentially) Add an outer pre_load that sets up a default value |
344 | for 'foo' to be used if the subsection isn't loaded. | |
5f9412bb DDAG |
345 | |
346 | Now that subsection will not be generated when using an older | |
347 | machine type and the migration stream will be accepted by older | |
edd70806 | 348 | QEMU versions. |
5f9412bb | 349 | |
2e3c8f8d DDAG |
350 | Not sending existing elements |
351 | ----------------------------- | |
352 | ||
353 | Sometimes members of the VMState are no longer needed: | |
5f9412bb | 354 | |
2e3c8f8d DDAG |
355 | - removing them will break migration compatibility |
356 | ||
edd70806 DDAG |
357 | - making them version dependent and bumping the version will break backward migration |
358 | compatibility. | |
359 | ||
360 | Adding a dummy field into the migration stream is normally the best way to preserve | |
361 | compatibility. | |
5f9412bb | 362 | |
edd70806 | 363 | If the field really does need to be removed then: |
2e3c8f8d DDAG |
364 | |
365 | a) Add a new property/compatibility/function in the same way for subsections above. | |
5f9412bb | 366 | b) replace the VMSTATE macro with the _TEST version of the macro, e.g.: |
2e3c8f8d DDAG |
367 | |
368 | ``VMSTATE_UINT32(foo, barstruct)`` | |
369 | ||
5f9412bb | 370 | becomes |
5f9412bb | 371 | |
2e3c8f8d DDAG |
372 | ``VMSTATE_UINT32_TEST(foo, barstruct, pre_version_baz)`` |
373 | ||
374 | Sometime in the future when we no longer care about the ancient versions these can be killed off. | |
edd70806 DDAG |
375 | Note that for backward compatibility it's important to fill in the structure with |
376 | data that the destination will understand. | |
377 | ||
378 | Any difference in the predicates on the source and destination will end up | |
379 | with different fields being enabled and data being loaded into the wrong | |
380 | fields; for this reason conditional fields like this are very fragile. | |
381 | ||
382 | Versions | |
383 | -------- | |
384 | ||
385 | Version numbers are intended for major incompatible changes to the | |
386 | migration of a device, and using them breaks backward-migration | |
387 | compatibility; in general most changes can be made by adding Subsections | |
388 | (see above) or _TEST macros (see above) which won't break compatibility. | |
389 | ||
4df3a7bf PM |
390 | Each version is associated with a series of fields saved. The ``save_state`` always saves |
391 | the state as the newer version. But ``load_state`` sometimes is able to | |
edd70806 DDAG |
392 | load state from an older version. |
393 | ||
18621987 | 394 | You can see that there are two version fields: |
edd70806 | 395 | |
4df3a7bf PM |
396 | - ``version_id``: the maximum version_id supported by VMState for that device. |
397 | - ``minimum_version_id``: the minimum version_id that VMState is able to understand | |
edd70806 | 398 | for that device. |
18621987 PM |
399 | |
400 | VMState is able to read versions from minimum_version_id to version_id. | |
edd70806 DDAG |
401 | |
402 | There are *_V* forms of many ``VMSTATE_`` macros to load fields for version dependent fields, | |
403 | e.g. | |
404 | ||
405 | .. code:: c | |
406 | ||
407 | VMSTATE_UINT16_V(ip_id, Slirp, 2), | |
408 | ||
409 | only loads that field for versions 2 and newer. | |
410 | ||
411 | Saving state will always create a section with the 'version_id' value | |
412 | and thus can't be loaded by any older QEMU. | |
413 | ||
414 | Massaging functions | |
415 | ------------------- | |
416 | ||
417 | Sometimes, it is not enough to be able to save the state directly | |
418 | from one structure, we need to fill the correct values there. One | |
419 | example is when we are using kvm. Before saving the cpu state, we | |
420 | need to ask kvm to copy to QEMU the state that it is using. And the | |
421 | opposite when we are loading the state, we need a way to tell kvm to | |
422 | load the state for the cpu that we have just loaded from the QEMUFile. | |
423 | ||
424 | The functions to do that are inside a vmstate definition, and are called: | |
425 | ||
426 | - ``int (*pre_load)(void *opaque);`` | |
427 | ||
428 | This function is called before we load the state of one device. | |
429 | ||
430 | - ``int (*post_load)(void *opaque, int version_id);`` | |
431 | ||
432 | This function is called after we load the state of one device. | |
433 | ||
434 | - ``int (*pre_save)(void *opaque);`` | |
435 | ||
436 | This function is called before we save the state of one device. | |
437 | ||
8c07559f AL |
438 | - ``int (*post_save)(void *opaque);`` |
439 | ||
440 | This function is called after we save the state of one device | |
441 | (even upon failure, unless the call to pre_save returned an error). | |
442 | ||
443 | Example: You can look at hpet.c, that uses the first three functions | |
444 | to massage the state that is transferred. | |
edd70806 DDAG |
445 | |
446 | The ``VMSTATE_WITH_TMP`` macro may be useful when the migration | |
447 | data doesn't match the stored device data well; it allows an | |
448 | intermediate temporary structure to be populated with migration | |
449 | data and then transferred to the main structure. | |
450 | ||
451 | If you use memory API functions that update memory layout outside | |
452 | initialization (i.e., in response to a guest action), this is a strong | |
4df3a7bf | 453 | indication that you need to call these functions in a ``post_load`` callback. |
edd70806 DDAG |
454 | Examples of such memory API functions are: |
455 | ||
456 | - memory_region_add_subregion() | |
457 | - memory_region_del_subregion() | |
458 | - memory_region_set_readonly() | |
c26763f8 | 459 | - memory_region_set_nonvolatile() |
edd70806 DDAG |
460 | - memory_region_set_enabled() |
461 | - memory_region_set_address() | |
462 | - memory_region_set_alias_offset() | |
463 | ||
464 | Iterative device migration | |
465 | -------------------------- | |
466 | ||
467 | Some devices, such as RAM, Block storage or certain platform devices, | |
468 | have large amounts of data that would mean that the CPUs would be | |
469 | paused for too long if they were sent in one section. For these | |
470 | devices an *iterative* approach is taken. | |
471 | ||
472 | The iterative devices generally don't use VMState macros | |
473 | (although it may be possible in some cases) and instead use | |
474 | qemu_put_*/qemu_get_* macros to read/write data to the stream. Specialist | |
475 | versions exist for high bandwidth IO. | |
476 | ||
477 | ||
478 | An iterative device must provide: | |
479 | ||
480 | - A ``save_setup`` function that initialises the data structures and | |
481 | transmits a first section containing information on the device. In the | |
482 | case of RAM this transmits a list of RAMBlocks and sizes. | |
483 | ||
484 | - A ``load_setup`` function that initialises the data structures on the | |
485 | destination. | |
486 | ||
c8df4a7a JQ |
487 | - A ``state_pending_exact`` function that indicates how much more |
488 | data we must save. The core migration code will use this to | |
489 | determine when to pause the CPUs and complete the migration. | |
490 | ||
491 | - A ``state_pending_estimate`` function that indicates how much more | |
492 | data we must save. When the estimated amount is smaller than the | |
493 | threshold, we call ``state_pending_exact``. | |
494 | ||
495 | - A ``save_live_iterate`` function should send a chunk of data until | |
496 | the point that stream bandwidth limits tell it to stop. Each call | |
497 | generates one section. | |
edd70806 DDAG |
498 | |
499 | - A ``save_live_complete_precopy`` function that must transmit the | |
500 | last section for the device containing any remaining data. | |
501 | ||
502 | - A ``load_state`` function used to load sections generated by | |
503 | any of the save functions that generate sections. | |
504 | ||
505 | - ``cleanup`` functions for both save and load that are called | |
506 | at the end of migration. | |
507 | ||
508 | Note that the contents of the sections for iterative migration tend | |
509 | to be open-coded by the devices; care should be taken in parsing | |
510 | the results and structuring the stream to make them easy to validate. | |
511 | ||
512 | Device ordering | |
513 | --------------- | |
514 | ||
515 | There are cases in which the ordering of device loading matters; for | |
516 | example in some systems where a device may assert an interrupt during loading, | |
517 | if the interrupt controller is loaded later then it might lose the state. | |
518 | ||
519 | Some ordering is implicitly provided by the order in which the machine | |
520 | definition creates devices, however this is somewhat fragile. | |
521 | ||
522 | The ``MigrationPriority`` enum provides a means of explicitly enforcing | |
523 | ordering. Numerically higher priorities are loaded earlier. | |
524 | The priority is set by setting the ``priority`` field of the top level | |
525 | ``VMStateDescription`` for the device. | |
526 | ||
527 | Stream structure | |
528 | ================ | |
529 | ||
530 | The stream tries to be word and endian agnostic, allowing migration between hosts | |
531 | of different characteristics running the same VM. | |
532 | ||
533 | - Header | |
534 | ||
535 | - Magic | |
536 | - Version | |
537 | - VM configuration section | |
538 | ||
539 | - Machine type | |
540 | - Target page bits | |
541 | - List of sections | |
542 | Each section contains a device, or one iteration of a device save. | |
543 | ||
544 | - section type | |
545 | - section id | |
546 | - ID string (First section of each device) | |
547 | - instance id (First section of each device) | |
548 | - version id (First section of each device) | |
549 | - <device data> | |
550 | - Footer mark | |
551 | - EOF mark | |
552 | - VM Description structure | |
553 | Consisting of a JSON description of the contents for analysis only | |
554 | ||
555 | The ``device data`` in each section consists of the data produced | |
556 | by the code described above. For non-iterative devices they have a single | |
557 | section; iterative devices have an initial and last section and a set | |
558 | of parts in between. | |
559 | Note that there is very little checking by the common code of the integrity | |
560 | of the ``device data`` contents, that's up to the devices themselves. | |
561 | The ``footer mark`` provides a little bit of protection for the case where | |
562 | the receiving side reads more or less data than expected. | |
563 | ||
564 | The ``ID string`` is normally unique, having been formed from a bus name | |
565 | and device address, PCI devices and storage devices hung off PCI controllers | |
566 | fit this pattern well. Some devices are fixed single instances (e.g. "pc-ram"). | |
567 | Others (especially either older devices or system devices which for | |
568 | some reason don't have a bus concept) make use of the ``instance id`` | |
569 | for otherwise identically named devices. | |
5f9412bb | 570 | |
2e3c8f8d DDAG |
571 | Return path |
572 | ----------- | |
2bfdd1c8 | 573 | |
edd70806 DDAG |
574 | Only a unidirectional stream is required for normal migration, however a |
575 | ``return path`` can be created when bidirectional communication is desired. | |
576 | This is primarily used by postcopy, but is also used to return a success | |
577 | flag to the source at the end of migration. | |
2bfdd1c8 | 578 | |
2e3c8f8d | 579 | ``qemu_file_get_return_path(QEMUFile* fwdpath)`` gives the QEMUFile* for the return |
2bfdd1c8 DDAG |
580 | path. |
581 | ||
582 | Source side | |
2e3c8f8d | 583 | |
2bfdd1c8 DDAG |
584 | Forward path - written by migration thread |
585 | Return path - opened by main thread, read by return-path thread | |
586 | ||
587 | Destination side | |
2e3c8f8d | 588 | |
2bfdd1c8 DDAG |
589 | Forward path - read by main thread |
590 | Return path - opened by main thread, written by main thread AND postcopy | |
2e3c8f8d DDAG |
591 | thread (protected by rp_mutex) |
592 | ||
593 | Postcopy | |
594 | ======== | |
2bfdd1c8 | 595 | |
2bfdd1c8 DDAG |
596 | 'Postcopy' migration is a way to deal with migrations that refuse to converge |
597 | (or take too long to converge) its plus side is that there is an upper bound on | |
598 | the amount of migration traffic and time it takes, the down side is that during | |
f014880a | 599 | the postcopy phase, a failure of *either* side causes the guest to be lost. |
2bfdd1c8 DDAG |
600 | |
601 | In postcopy the destination CPUs are started before all the memory has been | |
602 | transferred, and accesses to pages that are yet to be transferred cause | |
603 | a fault that's translated by QEMU into a request to the source QEMU. | |
604 | ||
605 | Postcopy can be combined with precopy (i.e. normal migration) so that if precopy | |
606 | doesn't finish in a given time the switch is made to postcopy. | |
607 | ||
2e3c8f8d DDAG |
608 | Enabling postcopy |
609 | ----------------- | |
2bfdd1c8 | 610 | |
c2eb7f21 GK |
611 | To enable postcopy, issue this command on the monitor (both source and |
612 | destination) prior to the start of migration: | |
2bfdd1c8 | 613 | |
2e3c8f8d | 614 | ``migrate_set_capability postcopy-ram on`` |
2bfdd1c8 DDAG |
615 | |
616 | The normal commands are then used to start a migration, which is still | |
617 | started in precopy mode. Issuing: | |
618 | ||
2e3c8f8d | 619 | ``migrate_start_postcopy`` |
2bfdd1c8 DDAG |
620 | |
621 | will now cause the transition from precopy to postcopy. | |
622 | It can be issued immediately after migration is started or any | |
623 | time later on. Issuing it after the end of a migration is harmless. | |
624 | ||
9ed01779 | 625 | Blocktime is a postcopy live migration metric, intended to show how |
76ca4b58 | 626 | long the vCPU was in state of interruptible sleep due to pagefault. |
9ed01779 AP |
627 | That metric is calculated both for all vCPUs as overlapped value, and |
628 | separately for each vCPU. These values are calculated on destination | |
629 | side. To enable postcopy blocktime calculation, enter following | |
630 | command on destination monitor: | |
631 | ||
632 | ``migrate_set_capability postcopy-blocktime on`` | |
633 | ||
634 | Postcopy blocktime can be retrieved by query-migrate qmp command. | |
635 | postcopy-blocktime value of qmp command will show overlapped blocking | |
636 | time for all vCPU, postcopy-vcpu-blocktime will show list of blocking | |
637 | time per vCPU. | |
638 | ||
2e3c8f8d DDAG |
639 | .. note:: |
640 | During the postcopy phase, the bandwidth limits set using | |
cbde7be9 | 641 | ``migrate_set_parameter`` is ignored (to avoid delaying requested pages that |
2e3c8f8d | 642 | the destination is waiting for). |
2bfdd1c8 | 643 | |
2e3c8f8d DDAG |
644 | Postcopy device transfer |
645 | ------------------------ | |
2bfdd1c8 DDAG |
646 | |
647 | Loading of device data may cause the device emulation to access guest RAM | |
648 | that may trigger faults that have to be resolved by the source, as such | |
649 | the migration stream has to be able to respond with page data *during* the | |
650 | device load, and hence the device data has to be read from the stream completely | |
651 | before the device load begins to free the stream up. This is achieved by | |
652 | 'packaging' the device data into a blob that's read in one go. | |
653 | ||
654 | Source behaviour | |
2e3c8f8d | 655 | ---------------- |
2bfdd1c8 DDAG |
656 | |
657 | Until postcopy is entered the migration stream is identical to normal | |
658 | precopy, except for the addition of a 'postcopy advise' command at | |
659 | the beginning, to tell the destination that postcopy might happen. | |
660 | When postcopy starts the source sends the page discard data and then | |
661 | forms the 'package' containing: | |
662 | ||
2e3c8f8d DDAG |
663 | - Command: 'postcopy listen' |
664 | - The device state | |
2bfdd1c8 | 665 | |
2e3c8f8d DDAG |
666 | A series of sections, identical to the precopy streams device state stream |
667 | containing everything except postcopiable devices (i.e. RAM) | |
668 | - Command: 'postcopy run' | |
669 | ||
670 | The 'package' is sent as the data part of a Command: ``CMD_PACKAGED``, and the | |
2bfdd1c8 DDAG |
671 | contents are formatted in the same way as the main migration stream. |
672 | ||
673 | During postcopy the source scans the list of dirty pages and sends them | |
674 | to the destination without being requested (in much the same way as precopy), | |
675 | however when a page request is received from the destination, the dirty page | |
676 | scanning restarts from the requested location. This causes requested pages | |
677 | to be sent quickly, and also causes pages directly after the requested page | |
678 | to be sent quickly in the hope that those pages are likely to be used | |
679 | by the destination soon. | |
680 | ||
681 | Destination behaviour | |
2e3c8f8d | 682 | --------------------- |
2bfdd1c8 DDAG |
683 | |
684 | Initially the destination looks the same as precopy, with a single thread | |
685 | reading the migration stream; the 'postcopy advise' and 'discard' commands | |
686 | are processed to change the way RAM is managed, but don't affect the stream | |
687 | processing. | |
688 | ||
2e3c8f8d DDAG |
689 | :: |
690 | ||
691 | ------------------------------------------------------------------------------ | |
692 | 1 2 3 4 5 6 7 | |
693 | main -----DISCARD-CMD_PACKAGED ( LISTEN DEVICE DEVICE DEVICE RUN ) | |
694 | thread | | | |
695 | | (page request) | |
696 | | \___ | |
697 | v \ | |
698 | listen thread: --- page -- page -- page -- page -- page -- | |
699 | ||
700 | a b c | |
701 | ------------------------------------------------------------------------------ | |
702 | ||
703 | - On receipt of ``CMD_PACKAGED`` (1) | |
704 | ||
705 | All the data associated with the package - the ( ... ) section in the diagram - | |
706 | is read into memory, and the main thread recurses into qemu_loadvm_state_main | |
707 | to process the contents of the package (2) which contains commands (3,6) and | |
708 | devices (4...) | |
709 | ||
710 | - On receipt of 'postcopy listen' - 3 -(i.e. the 1st command in the package) | |
711 | ||
712 | a new thread (a) is started that takes over servicing the migration stream, | |
713 | while the main thread carries on loading the package. It loads normal | |
714 | background page data (b) but if during a device load a fault happens (5) | |
715 | the returned page (c) is loaded by the listen thread allowing the main | |
716 | threads device load to carry on. | |
717 | ||
718 | - The last thing in the ``CMD_PACKAGED`` is a 'RUN' command (6) | |
719 | ||
720 | letting the destination CPUs start running. At the end of the | |
721 | ``CMD_PACKAGED`` (7) the main thread returns to normal running behaviour and | |
722 | is no longer used by migration, while the listen thread carries on servicing | |
723 | page data until the end of migration. | |
724 | ||
f014880a PX |
725 | Postcopy Recovery |
726 | ----------------- | |
727 | ||
728 | Comparing to precopy, postcopy is special on error handlings. When any | |
729 | error happens (in this case, mostly network errors), QEMU cannot easily | |
730 | fail a migration because VM data resides in both source and destination | |
731 | QEMU instances. On the other hand, when issue happens QEMU on both sides | |
732 | will go into a paused state. It'll need a recovery phase to continue a | |
733 | paused postcopy migration. | |
734 | ||
735 | The recovery phase normally contains a few steps: | |
736 | ||
737 | - When network issue occurs, both QEMU will go into PAUSED state | |
738 | ||
739 | - When the network is recovered (or a new network is provided), the admin | |
740 | can setup the new channel for migration using QMP command | |
741 | 'migrate-recover' on destination node, preparing for a resume. | |
742 | ||
743 | - On source host, the admin can continue the interrupted postcopy | |
744 | migration using QMP command 'migrate' with resume=true flag set. | |
745 | ||
746 | - After the connection is re-established, QEMU will continue the postcopy | |
747 | migration on both sides. | |
748 | ||
749 | During a paused postcopy migration, the VM can logically still continue | |
750 | running, and it will not be impacted from any page access to pages that | |
751 | were already migrated to destination VM before the interruption happens. | |
752 | However, if any of the missing pages got accessed on destination VM, the VM | |
753 | thread will be halted waiting for the page to be migrated, it means it can | |
754 | be halted until the recovery is complete. | |
755 | ||
756 | The impact of accessing missing pages can be relevant to different | |
757 | configurations of the guest. For example, when with async page fault | |
758 | enabled, logically the guest can proactively schedule out the threads | |
759 | accessing missing pages. | |
760 | ||
2e3c8f8d DDAG |
761 | Postcopy states |
762 | --------------- | |
2bfdd1c8 DDAG |
763 | |
764 | Postcopy moves through a series of states (see postcopy_state) from | |
765 | ADVISE->DISCARD->LISTEN->RUNNING->END | |
766 | ||
2e3c8f8d DDAG |
767 | - Advise |
768 | ||
769 | Set at the start of migration if postcopy is enabled, even | |
770 | if it hasn't had the start command; here the destination | |
771 | checks that its OS has the support needed for postcopy, and performs | |
772 | setup to ensure the RAM mappings are suitable for later postcopy. | |
773 | The destination will fail early in migration at this point if the | |
774 | required OS support is not present. | |
775 | (Triggered by reception of POSTCOPY_ADVISE command) | |
776 | ||
777 | - Discard | |
778 | ||
779 | Entered on receipt of the first 'discard' command; prior to | |
780 | the first Discard being performed, hugepages are switched off | |
781 | (using madvise) to ensure that no new huge pages are created | |
782 | during the postcopy phase, and to cause any huge pages that | |
783 | have discards on them to be broken. | |
784 | ||
785 | - Listen | |
786 | ||
787 | The first command in the package, POSTCOPY_LISTEN, switches | |
788 | the destination state to Listen, and starts a new thread | |
789 | (the 'listen thread') which takes over the job of receiving | |
790 | pages off the migration stream, while the main thread carries | |
791 | on processing the blob. With this thread able to process page | |
792 | reception, the destination now 'sensitises' the RAM to detect | |
793 | any access to missing pages (on Linux using the 'userfault' | |
794 | system). | |
795 | ||
796 | - Running | |
797 | ||
798 | POSTCOPY_RUN causes the destination to synchronise all | |
799 | state and start the CPUs and IO devices running. The main | |
800 | thread now finishes processing the migration package and | |
801 | now carries on as it would for normal precopy migration | |
802 | (although it can't do the cleanup it would do as it | |
803 | finishes a normal migration). | |
804 | ||
f014880a PX |
805 | - Paused |
806 | ||
807 | Postcopy can run into a paused state (normally on both sides when | |
808 | happens), where all threads will be temporarily halted mostly due to | |
809 | network errors. When reaching paused state, migration will make sure | |
810 | the qemu binary on both sides maintain the data without corrupting | |
811 | the VM. To continue the migration, the admin needs to fix the | |
812 | migration channel using the QMP command 'migrate-recover' on the | |
813 | destination node, then resume the migration using QMP command 'migrate' | |
814 | again on source node, with resume=true flag set. | |
815 | ||
2e3c8f8d DDAG |
816 | - End |
817 | ||
818 | The listen thread can now quit, and perform the cleanup of migration | |
819 | state, the migration is now complete. | |
820 | ||
f014880a PX |
821 | Source side page map |
822 | -------------------- | |
2bfdd1c8 | 823 | |
f014880a PX |
824 | The 'migration bitmap' in postcopy is basically the same as in the precopy, |
825 | where each of the bit to indicate that page is 'dirty' - i.e. needs | |
826 | sending. During the precopy phase this is updated as the CPU dirties | |
827 | pages, however during postcopy the CPUs are stopped and nothing should | |
828 | dirty anything any more. Instead, dirty bits are cleared when the relevant | |
829 | pages are sent during postcopy. | |
2bfdd1c8 | 830 | |
2e3c8f8d DDAG |
831 | Postcopy with hugepages |
832 | ----------------------- | |
0c1f4036 DDAG |
833 | |
834 | Postcopy now works with hugetlbfs backed memory: | |
2e3c8f8d | 835 | |
0c1f4036 DDAG |
836 | a) The linux kernel on the destination must support userfault on hugepages. |
837 | b) The huge-page configuration on the source and destination VMs must be | |
838 | identical; i.e. RAMBlocks on both sides must use the same page size. | |
2e3c8f8d | 839 | c) Note that ``-mem-path /dev/hugepages`` will fall back to allocating normal |
0c1f4036 | 840 | RAM if it doesn't have enough hugepages, triggering (b) to fail. |
2e3c8f8d | 841 | Using ``-mem-prealloc`` enforces the allocation using hugepages. |
0c1f4036 DDAG |
842 | d) Care should be taken with the size of hugepage used; postcopy with 2MB |
843 | hugepages works well, however 1GB hugepages are likely to be problematic | |
844 | since it takes ~1 second to transfer a 1GB hugepage across a 10Gbps link, | |
845 | and until the full page is transferred the destination thread is blocked. | |
1dc61e7b DDAG |
846 | |
847 | Postcopy with shared memory | |
848 | --------------------------- | |
849 | ||
850 | Postcopy migration with shared memory needs explicit support from the other | |
851 | processes that share memory and from QEMU. There are restrictions on the type of | |
852 | memory that userfault can support shared. | |
853 | ||
4df3a7bf PM |
854 | The Linux kernel userfault support works on ``/dev/shm`` memory and on ``hugetlbfs`` |
855 | (although the kernel doesn't provide an equivalent to ``madvise(MADV_DONTNEED)`` | |
1dc61e7b DDAG |
856 | for hugetlbfs which may be a problem in some configurations). |
857 | ||
858 | The vhost-user code in QEMU supports clients that have Postcopy support, | |
4df3a7bf | 859 | and the ``vhost-user-bridge`` (in ``tests/``) and the DPDK package have changes |
1dc61e7b DDAG |
860 | to support postcopy. |
861 | ||
862 | The client needs to open a userfaultfd and register the areas | |
863 | of memory that it maps with userfault. The client must then pass the | |
864 | userfaultfd back to QEMU together with a mapping table that allows | |
865 | fault addresses in the clients address space to be converted back to | |
866 | RAMBlock/offsets. The client's userfaultfd is added to the postcopy | |
867 | fault-thread and page requests are made on behalf of the client by QEMU. | |
868 | QEMU performs 'wake' operations on the client's userfaultfd to allow it | |
869 | to continue after a page has arrived. | |
870 | ||
871 | .. note:: | |
872 | There are two future improvements that would be nice: | |
873 | a) Some way to make QEMU ignorant of the addresses in the clients | |
874 | address space | |
875 | b) Avoiding the need for QEMU to perform ufd-wake calls after the | |
876 | pages have arrived | |
877 | ||
878 | Retro-fitting postcopy to existing clients is possible: | |
879 | a) A mechanism is needed for the registration with userfault as above, | |
880 | and the registration needs to be coordinated with the phases of | |
881 | postcopy. In vhost-user extra messages are added to the existing | |
882 | control channel. | |
883 | b) Any thread that can block due to guest memory accesses must be | |
884 | identified and the implication understood; for example if the | |
885 | guest memory access is made while holding a lock then all other | |
886 | threads waiting for that lock will also be blocked. | |
edd70806 | 887 | |
f014880a PX |
888 | Postcopy Preemption Mode |
889 | ------------------------ | |
890 | ||
891 | Postcopy preempt is a new capability introduced in 8.0 QEMU release, it | |
892 | allows urgent pages (those got page fault requested from destination QEMU | |
893 | explicitly) to be sent in a separate preempt channel, rather than queued in | |
894 | the background migration channel. Anyone who cares about latencies of page | |
895 | faults during a postcopy migration should enable this feature. By default, | |
896 | it's not enabled. | |
897 | ||
edd70806 DDAG |
898 | Firmware |
899 | ======== | |
900 | ||
901 | Migration migrates the copies of RAM and ROM, and thus when running | |
902 | on the destination it includes the firmware from the source. Even after | |
903 | resetting a VM, the old firmware is used. Only once QEMU has been restarted | |
904 | is the new firmware in use. | |
905 | ||
906 | - Changes in firmware size can cause changes in the required RAMBlock size | |
907 | to hold the firmware and thus migration can fail. In practice it's best | |
908 | to pad firmware images to convenient powers of 2 with plenty of space | |
909 | for growth. | |
910 | ||
911 | - Care should be taken with device emulation code so that newer | |
912 | emulation code can work with older firmware to allow forward migration. | |
913 | ||
914 | - Care should be taken with newer firmware so that backward migration | |
915 | to older systems with older device emulation code will work. | |
916 | ||
917 | In some cases it may be best to tie specific firmware versions to specific | |
918 | versioned machine types to cut down on the combinations that will need | |
919 | support. This is also useful when newer versions of firmware outgrow | |
920 | the padding. | |
921 | ||
1aefe2ca JQ |
922 | |
923 | Backwards compatibility | |
924 | ======================= | |
925 | ||
926 | How backwards compatibility works | |
927 | --------------------------------- | |
928 | ||
929 | When we do migration, we have two QEMU processes: the source and the | |
930 | target. There are two cases, they are the same version or they are | |
931 | different versions. The easy case is when they are the same version. | |
932 | The difficult one is when they are different versions. | |
933 | ||
934 | There are two things that are different, but they have very similar | |
935 | names and sometimes get confused: | |
936 | ||
937 | - QEMU version | |
938 | - machine type version | |
939 | ||
940 | Let's start with a practical example, we start with: | |
941 | ||
942 | - qemu-system-x86_64 (v5.2), from now on qemu-5.2. | |
943 | - qemu-system-x86_64 (v5.1), from now on qemu-5.1. | |
944 | ||
945 | Related to this are the "latest" machine types defined on each of | |
946 | them: | |
947 | ||
948 | - pc-q35-5.2 (newer one in qemu-5.2) from now on pc-5.2 | |
949 | - pc-q35-5.1 (newer one in qemu-5.1) from now on pc-5.1 | |
950 | ||
951 | First of all, migration is only supposed to work if you use the same | |
952 | machine type in both source and destination. The QEMU hardware | |
953 | configuration needs to be the same also on source and destination. | |
954 | Most aspects of the backend configuration can be changed at will, | |
955 | except for a few cases where the backend features influence frontend | |
956 | device feature exposure. But that is not relevant for this section. | |
957 | ||
958 | I am going to list the number of combinations that we can have. Let's | |
959 | start with the trivial ones, QEMU is the same on source and | |
960 | destination: | |
961 | ||
962 | 1 - qemu-5.2 -M pc-5.2 -> migrates to -> qemu-5.2 -M pc-5.2 | |
963 | ||
964 | This is the latest QEMU with the latest machine type. | |
965 | This have to work, and if it doesn't work it is a bug. | |
966 | ||
967 | 2 - qemu-5.1 -M pc-5.1 -> migrates to -> qemu-5.1 -M pc-5.1 | |
968 | ||
969 | Exactly the same case than the previous one, but for 5.1. | |
970 | Nothing to see here either. | |
971 | ||
972 | This are the easiest ones, we will not talk more about them in this | |
973 | section. | |
974 | ||
975 | Now we start with the more interesting cases. Consider the case where | |
976 | we have the same QEMU version in both sides (qemu-5.2) but we are using | |
977 | the latest machine type for that version (pc-5.2) but one of an older | |
978 | QEMU version, in this case pc-5.1. | |
979 | ||
980 | 3 - qemu-5.2 -M pc-5.1 -> migrates to -> qemu-5.2 -M pc-5.1 | |
981 | ||
982 | It needs to use the definition of pc-5.1 and the devices as they | |
983 | were configured on 5.1, but this should be easy in the sense that | |
984 | both sides are the same QEMU and both sides have exactly the same | |
985 | idea of what the pc-5.1 machine is. | |
986 | ||
987 | 4 - qemu-5.1 -M pc-5.2 -> migrates to -> qemu-5.1 -M pc-5.2 | |
988 | ||
989 | This combination is not possible as the qemu-5.1 doen't understand | |
990 | pc-5.2 machine type. So nothing to worry here. | |
991 | ||
992 | Now it comes the interesting ones, when both QEMU processes are | |
993 | different. Notice also that the machine type needs to be pc-5.1, | |
994 | because we have the limitation than qemu-5.1 doesn't know pc-5.2. So | |
995 | the possible cases are: | |
996 | ||
997 | 5 - qemu-5.2 -M pc-5.1 -> migrates to -> qemu-5.1 -M pc-5.1 | |
998 | ||
999 | This migration is known as newer to older. We need to make sure | |
1000 | when we are developing 5.2 we need to take care about not to break | |
1001 | migration to qemu-5.1. Notice that we can't make updates to | |
1002 | qemu-5.1 to understand whatever qemu-5.2 decides to change, so it is | |
1003 | in qemu-5.2 side to make the relevant changes. | |
1004 | ||
1005 | 6 - qemu-5.1 -M pc-5.1 -> migrates to -> qemu-5.2 -M pc-5.1 | |
1006 | ||
1007 | This migration is known as older to newer. We need to make sure | |
1008 | than we are able to receive migrations from qemu-5.1. The problem is | |
1009 | similar to the previous one. | |
1010 | ||
1011 | If qemu-5.1 and qemu-5.2 were the same, there will not be any | |
1012 | compatibility problems. But the reason that we create qemu-5.2 is to | |
1013 | get new features, devices, defaults, etc. | |
1014 | ||
1015 | If we get a device that has a new feature, or change a default value, | |
1016 | we have a problem when we try to migrate between different QEMU | |
1017 | versions. | |
1018 | ||
1019 | So we need a way to tell qemu-5.2 that when we are using machine type | |
1020 | pc-5.1, it needs to **not** use the feature, to be able to migrate to | |
1021 | real qemu-5.1. | |
1022 | ||
1023 | And the equivalent part when migrating from qemu-5.1 to qemu-5.2. | |
1024 | qemu-5.2 has to expect that it is not going to get data for the new | |
1025 | feature, because qemu-5.1 doesn't know about it. | |
1026 | ||
1027 | How do we tell QEMU about these device feature changes? In | |
1028 | hw/core/machine.c:hw_compat_X_Y arrays. | |
1029 | ||
1030 | If we change a default value, we need to put back the old value on | |
1031 | that array. And the device, during initialization needs to look at | |
1032 | that array to see what value it needs to get for that feature. And | |
1033 | what are we going to put in that array, the value of a property. | |
1034 | ||
1035 | To create a property for a device, we need to use one of the | |
1036 | DEFINE_PROP_*() macros. See include/hw/qdev-properties.h to find the | |
1037 | macros that exist. With it, we set the default value for that | |
1038 | property, and that is what it is going to get in the latest released | |
1039 | version. But if we want a different value for a previous version, we | |
1040 | can change that in the hw_compat_X_Y arrays. | |
1041 | ||
1042 | hw_compat_X_Y is an array of registers that have the format: | |
1043 | ||
1044 | - name_device | |
1045 | - name_property | |
1046 | - value | |
1047 | ||
1048 | Let's see a practical example. | |
1049 | ||
1050 | In qemu-5.2 virtio-blk-device got multi queue support. This is a | |
1051 | change that is not backward compatible. In qemu-5.1 it has one | |
1052 | queue. In qemu-5.2 it has the same number of queues as the number of | |
1053 | cpus in the system. | |
1054 | ||
1055 | When we are doing migration, if we migrate from a device that has 4 | |
1056 | queues to a device that have only one queue, we don't know where to | |
1057 | put the extra information for the other 3 queues, and we fail | |
1058 | migration. | |
1059 | ||
1060 | Similar problem when we migrate from qemu-5.1 that has only one queue | |
1061 | to qemu-5.2, we only sent information for one queue, but destination | |
1062 | has 4, and we have 3 queues that are not properly initialized and | |
1063 | anything can happen. | |
1064 | ||
1065 | So, how can we address this problem. Easy, just convince qemu-5.2 | |
1066 | that when it is running pc-5.1, it needs to set the number of queues | |
1067 | for virtio-blk-devices to 1. | |
1068 | ||
1069 | That way we fix the cases 5 and 6. | |
1070 | ||
1071 | 5 - qemu-5.2 -M pc-5.1 -> migrates to -> qemu-5.1 -M pc-5.1 | |
1072 | ||
1073 | qemu-5.2 -M pc-5.1 sets number of queues to be 1. | |
1074 | qemu-5.1 -M pc-5.1 expects number of queues to be 1. | |
1075 | ||
1076 | correct. migration works. | |
1077 | ||
1078 | 6 - qemu-5.1 -M pc-5.1 -> migrates to -> qemu-5.2 -M pc-5.1 | |
1079 | ||
1080 | qemu-5.1 -M pc-5.1 sets number of queues to be 1. | |
1081 | qemu-5.2 -M pc-5.1 expects number of queues to be 1. | |
1082 | ||
1083 | correct. migration works. | |
1084 | ||
1085 | And now the other interesting case, case 3. In this case we have: | |
1086 | ||
1087 | 3 - qemu-5.2 -M pc-5.1 -> migrates to -> qemu-5.2 -M pc-5.1 | |
1088 | ||
1089 | Here we have the same QEMU in both sides. So it doesn't matter a | |
1090 | lot if we have set the number of queues to 1 or not, because | |
1091 | they are the same. | |
1092 | ||
1093 | WRONG! | |
1094 | ||
1095 | Think what happens if we do one of this double migrations: | |
1096 | ||
1097 | A -> migrates -> B -> migrates -> C | |
1098 | ||
1099 | where: | |
1100 | ||
1101 | A: qemu-5.1 -M pc-5.1 | |
1102 | B: qemu-5.2 -M pc-5.1 | |
1103 | C: qemu-5.2 -M pc-5.1 | |
1104 | ||
1105 | migration A -> B is case 6, so number of queues needs to be 1. | |
1106 | ||
1107 | migration B -> C is case 3, so we don't care. But actually we | |
1108 | care because we haven't started the guest in qemu-5.2, it came | |
1109 | migrated from qemu-5.1. So to be in the safe place, we need to | |
1110 | always use number of queues 1 when we are using pc-5.1. | |
1111 | ||
1112 | Now, how was this done in reality? The following commit shows how it | |
1113 | was done:: | |
1114 | ||
1115 | commit 9445e1e15e66c19e42bea942ba810db28052cd05 | |
1116 | Author: Stefan Hajnoczi <stefanha@redhat.com> | |
1117 | Date: Tue Aug 18 15:33:47 2020 +0100 | |
1118 | ||
1119 | virtio-blk-pci: default num_queues to -smp N | |
1120 | ||
1121 | The relevant parts for migration are:: | |
1122 | ||
1123 | @@ -1281,7 +1284,8 @@ static Property virtio_blk_properties[] = { | |
1124 | #endif | |
1125 | DEFINE_PROP_BIT("request-merging", VirtIOBlock, conf.request_merging, 0, | |
1126 | true), | |
1127 | - DEFINE_PROP_UINT16("num-queues", VirtIOBlock, conf.num_queues, 1), | |
1128 | + DEFINE_PROP_UINT16("num-queues", VirtIOBlock, conf.num_queues, | |
1129 | + VIRTIO_BLK_AUTO_NUM_QUEUES), | |
1130 | DEFINE_PROP_UINT16("queue-size", VirtIOBlock, conf.queue_size, 256), | |
1131 | ||
1132 | It changes the default value of num_queues. But it fishes it for old | |
1133 | machine types to have the right value:: | |
1134 | ||
1135 | @@ -31,6 +31,7 @@ | |
1136 | GlobalProperty hw_compat_5_1[] = { | |
1137 | ... | |
1138 | + { "virtio-blk-device", "num-queues", "1"}, | |
1139 | ... | |
1140 | }; | |
593c28c0 JQ |
1141 | |
1142 | A device with diferent features on both sides | |
1143 | --------------------------------------------- | |
1144 | ||
1145 | Let's assume that we are using the same QEMU binary on both sides, | |
1146 | just to make the things easier. But we have a device that has | |
1147 | different features on both sides of the migration. That can be | |
1148 | because the devices are different, because the kernel driver of both | |
1149 | devices have different features, whatever. | |
1150 | ||
1151 | How can we get this to work with migration. The way to do that is | |
1152 | "theoretically" easy. You have to get the features that the device | |
1153 | has in the source of the migration. The features that the device has | |
1154 | on the target of the migration, you get the intersection of the | |
1155 | features of both sides, and that is the way that you should launch | |
1156 | QEMU. | |
1157 | ||
1158 | Notice that this is not completely related to QEMU. The most | |
1159 | important thing here is that this should be handled by the managing | |
1160 | application that launches QEMU. If QEMU is configured correctly, the | |
1161 | migration will succeed. | |
1162 | ||
1163 | That said, actually doing it is complicated. Almost all devices are | |
1164 | bad at being able to be launched with only some features enabled. | |
1165 | With one big exception: cpus. | |
1166 | ||
1167 | You can read the documentation for QEMU x86 cpu models here: | |
1168 | ||
1169 | https://qemu-project.gitlab.io/qemu/system/qemu-cpu-models.html | |
1170 | ||
1171 | See when they talk about migration they recommend that one chooses the | |
1172 | newest cpu model that is supported for all cpus. | |
1173 | ||
1174 | Let's say that we have: | |
1175 | ||
1176 | Host A: | |
1177 | ||
1178 | Device X has the feature Y | |
1179 | ||
1180 | Host B: | |
1181 | ||
1182 | Device X has not the feature Y | |
1183 | ||
1184 | If we try to migrate without any care from host A to host B, it will | |
1185 | fail because when migration tries to load the feature Y on | |
1186 | destination, it will find that the hardware is not there. | |
1187 | ||
1188 | Doing this would be the equivalent of doing with cpus: | |
1189 | ||
1190 | Host A: | |
1191 | ||
1192 | $ qemu-system-x86_64 -cpu host | |
1193 | ||
1194 | Host B: | |
1195 | ||
1196 | $ qemu-system-x86_64 -cpu host | |
1197 | ||
1198 | When both hosts have different cpu features this is guaranteed to | |
1199 | fail. Especially if Host B has less features than host A. If host A | |
1200 | has less features than host B, sometimes it works. Important word of | |
1201 | last sentence is "sometimes". | |
1202 | ||
1203 | So, forgetting about cpu models and continuing with the -cpu host | |
1204 | example, let's see that the differences of the cpus is that Host A and | |
1205 | B have the following features: | |
1206 | ||
1207 | Features: 'pcid' 'stibp' 'taa-no' | |
1208 | Host A: X X | |
1209 | Host B: X | |
1210 | ||
1211 | And we want to migrate between them, the way configure both QEMU cpu | |
1212 | will be: | |
1213 | ||
1214 | Host A: | |
1215 | ||
1216 | $ qemu-system-x86_64 -cpu host,pcid=off,stibp=off | |
1217 | ||
1218 | Host B: | |
1219 | ||
1220 | $ qemu-system-x86_64 -cpu host,taa-no=off | |
1221 | ||
1222 | And you would be able to migrate between them. It is responsability | |
1223 | of the management application or of the user to make sure that the | |
1224 | configuration is correct. QEMU doesn't know how to look at this kind | |
1225 | of features in general. | |
1226 | ||
1227 | Notice that we don't recomend to use -cpu host for migration. It is | |
1228 | used in this example because it makes the example simpler. | |
1229 | ||
1230 | Other devices have worse control about individual features. If they | |
1231 | want to be able to migrate between hosts that show different features, | |
1232 | the device needs a way to configure which ones it is going to use. | |
1233 | ||
1234 | In this section we have considered that we are using the same QEMU | |
1235 | binary in both sides of the migration. If we use different QEMU | |
1236 | versions process, then we need to have into account all other | |
1237 | differences and the examples become even more complicated. |