[mirror_qemu.git] / docs / devel / vfio-migration.rst

=====================
VFIO device Migration
=====================

Migration of virtual machine involves saving the state for each device that
the guest is running on source host and restoring this saved state on the
destination host. This document details how saving and restoring of VFIO
devices is done in QEMU.

Migration of VFIO devices consists of two phases: the optional pre-copy phase,
and the stop-and-copy phase. The pre-copy phase is iterative and allows to
accommodate VFIO devices that have a large amount of data that needs to be
transferred. The iterative pre-copy phase of migration allows for the guest to
continue whilst the VFIO device state is transferred to the destination, this
helps to reduce the total downtime of the VM. VFIO devices can choose to skip
the pre-copy phase of migration by returning pending_bytes as zero during the
pre-copy phase.

A detailed description of the UAPI for VFIO device migration can be found in
the comment for the ``vfio_device_migration_info`` structure in the header
file linux-headers/linux/vfio.h.

VFIO implements the device hooks for the iterative approach as follows:

* A ``save_setup`` function that sets up the migration region and sets _SAVING
  flag in the VFIO device state.

* A ``load_setup`` function that sets up the migration region on the
  destination and sets _RESUMING flag in the VFIO device state.

* A ``save_live_pending`` function that reads pending_bytes from the vendor
  driver, which indicates the amount of data that the vendor driver has yet to
  save for the VFIO device.

* A ``save_live_iterate`` function that reads the VFIO device's data from the
  vendor driver through the migration region during iterative phase.

* A ``save_state`` function to save the device config space if it is present.

* A ``save_live_complete_precopy`` function that resets _RUNNING flag from the
  VFIO device state and iteratively copies the remaining data for the VFIO
  device until the vendor driver indicates that no data remains (pending bytes
  is zero).

* A ``load_state`` function that loads the config section and the data
  sections that are generated by the save functions above

* ``cleanup`` functions for both save and load that perform any migration
  related cleanup, including unmapping the migration region


The VFIO migration code uses a VM state change handler to change the VFIO
device state when the VM state changes from running to not-running, and
vice versa.

Similarly, a migration state change handler is used to trigger a transition of
the VFIO device state when certain changes of the migration state occur. For
example, the VFIO device state is transitioned back to _RUNNING in case a
migration failed or was canceled.

System memory dirty pages tracking
----------------------------------

A ``log_global_start`` and ``log_global_stop`` memory listener callback informs
the VFIO IOMMU module to start and stop dirty page tracking. A ``log_sync``
memory listener callback marks those system memory pages as dirty which are
used for DMA by the VFIO device. The dirty pages bitmap is queried per
container. All pages pinned by the vendor driver through external APIs have to
be marked as dirty during migration. When there are CPU writes, CPU dirty page
tracking can identify dirtied pages, but any page pinned by the vendor driver
can also be written by the device. There is currently no device or IOMMU
support for dirty page tracking in hardware.

By default, dirty pages are tracked when the device is in pre-copy as well as
stop-and-copy phase. So, a page pinned by the vendor driver will be copied to
the destination in both phases. Copying dirty pages in pre-copy phase helps
QEMU to predict if it can achieve its downtime tolerances. If QEMU during
pre-copy phase keeps finding dirty pages continuously, then it understands
that even in stop-and-copy phase, it is likely to find dirty pages and can
predict the downtime accordingly.

QEMU also provides a per device opt-out option ``pre-copy-dirty-page-tracking``
which disables querying the dirty bitmap during pre-copy phase. If it is set to
off, all dirty pages will be copied to the destination in stop-and-copy phase
only.

System memory dirty pages tracking when vIOMMU is enabled
---------------------------------------------------------

With vIOMMU, an IO virtual address range can get unmapped while in pre-copy
phase of migration. In that case, the unmap ioctl returns any dirty pages in
that range and QEMU reports corresponding guest physical pages dirty. During
stop-and-copy phase, an IOMMU notifier is used to get a callback for mapped
pages and then dirty pages bitmap is fetched from VFIO IOMMU modules for those
mapped ranges.

Flow of state changes during Live migration
===========================================

Below is the flow of state change during live migration.
The values in the brackets represent the VM state, the migration state, and
the VFIO device state, respectively.

Live migration save path
------------------------

::

                        QEMU normal running state
                        (RUNNING, _NONE, _RUNNING)
                                  |
                     migrate_init spawns migration_thread
                Migration thread then calls each device's .save_setup()
                    (RUNNING, _SETUP, _RUNNING|_SAVING)
                                  |
                    (RUNNING, _ACTIVE, _RUNNING|_SAVING)
             If device is active, get pending_bytes by .save_live_pending()
          If total pending_bytes >= threshold_size, call .save_live_iterate()
                  Data of VFIO device for pre-copy phase is copied
        Iterate till total pending bytes converge and are less than threshold
                                  |
  On migration completion, vCPU stops and calls .save_live_complete_precopy for
   each active device. The VFIO device is then transitioned into _SAVING state
                   (FINISH_MIGRATE, _DEVICE, _SAVING)
                                  |
     For the VFIO device, iterate in .save_live_complete_precopy until
                         pending data is 0
                   (FINISH_MIGRATE, _DEVICE, _STOPPED)
                                  |
                 (FINISH_MIGRATE, _COMPLETED, _STOPPED)
             Migraton thread schedules cleanup bottom half and exits

Live migration resume path
--------------------------

::

              Incoming migration calls .load_setup for each device
                       (RESTORE_VM, _ACTIVE, _STOPPED)
                                 |
       For each device, .load_state is called for that device section data
                       (RESTORE_VM, _ACTIVE, _RESUMING)
                                 |
    At the end, .load_cleanup is called for each device and vCPUs are started
                       (RUNNING, _NONE, _RUNNING)

Postcopy
========

Postcopy migration is currently not supported for VFIO devices.
Commit	Line	Data
2a578133 TG	1	=====================
	2	VFIO device Migration
	3	=====================
	4
	5	Migration of virtual machine involves saving the state for each device that
	6	the guest is running on source host and restoring this saved state on the
	7	destination host. This document details how saving and restoring of VFIO
	8	devices is done in QEMU.
	9
	10	Migration of VFIO devices consists of two phases: the optional pre-copy phase,
	11	and the stop-and-copy phase. The pre-copy phase is iterative and allows to
	12	accommodate VFIO devices that have a large amount of data that needs to be
	13	transferred. The iterative pre-copy phase of migration allows for the guest to
	14	continue whilst the VFIO device state is transferred to the destination, this
	15	helps to reduce the total downtime of the VM. VFIO devices can choose to skip
	16	the pre-copy phase of migration by returning pending_bytes as zero during the
	17	pre-copy phase.
	18
	19	A detailed description of the UAPI for VFIO device migration can be found in
	20	the comment for the ``vfio_device_migration_info`` structure in the header
	21	file linux-headers/linux/vfio.h.
	22
	23	VFIO implements the device hooks for the iterative approach as follows:
	24
	25	* A ``save_setup`` function that sets up the migration region and sets _SAVING
	26	flag in the VFIO device state.
	27
	28	* A ``load_setup`` function that sets up the migration region on the
	29	destination and sets _RESUMING flag in the VFIO device state.
	30
	31	* A ``save_live_pending`` function that reads pending_bytes from the vendor
	32	driver, which indicates the amount of data that the vendor driver has yet to
	33	save for the VFIO device.
	34
	35	* A ``save_live_iterate`` function that reads the VFIO device's data from the
	36	vendor driver through the migration region during iterative phase.
	37
	38	* A ``save_state`` function to save the device config space if it is present.
	39
	40	* A ``save_live_complete_precopy`` function that resets _RUNNING flag from the
	41	VFIO device state and iteratively copies the remaining data for the VFIO
	42	device until the vendor driver indicates that no data remains (pending bytes
	43	is zero).
	44
	45	* A ``load_state`` function that loads the config section and the data
	46	sections that are generated by the save functions above
	47
	48	* ``cleanup`` functions for both save and load that perform any migration
	49	related cleanup, including unmapping the migration region
	50
	51
	52	The VFIO migration code uses a VM state change handler to change the VFIO
	53	device state when the VM state changes from running to not-running, and
	54	vice versa.
	55
	56	Similarly, a migration state change handler is used to trigger a transition of
	57	the VFIO device state when certain changes of the migration state occur. For
	58	example, the VFIO device state is transitioned back to _RUNNING in case a
	59	migration failed or was canceled.
	60
	61	System memory dirty pages tracking
	62	----------------------------------
	63
	64	A ``log_global_start`` and ``log_global_stop`` memory listener callback informs
65	the VFIO IOMMU module to start and stop dirty page tracking. A ``log_sync``
66	memory listener callback marks those system memory pages as dirty which are
67	used for DMA by the VFIO device. The dirty pages bitmap is queried per
68	container. All pages pinned by the vendor driver through external APIs have to
69	be marked as dirty during migration. When there are CPU writes, CPU dirty page
70	tracking can identify dirtied pages, but any page pinned by the vendor driver
71	can also be written by the device. There is currently no device or IOMMU
72	support for dirty page tracking in hardware.
73
74	By default, dirty pages are tracked when the device is in pre-copy as well as
75	stop-and-copy phase. So, a page pinned by the vendor driver will be copied to
76	the destination in both phases. Copying dirty pages in pre-copy phase helps
77	QEMU to predict if it can achieve its downtime tolerances. If QEMU during
78	pre-copy phase keeps finding dirty pages continuously, then it understands
79	that even in stop-and-copy phase, it is likely to find dirty pages and can
80	predict the downtime accordingly.
81
82	QEMU also provides a per device opt-out option ``pre-copy-dirty-page-tracking``
83	which disables querying the dirty bitmap during pre-copy phase. If it is set to
84	off, all dirty pages will be copied to the destination in stop-and-copy phase
85	only.
86
87	System memory dirty pages tracking when vIOMMU is enabled
88	---------------------------------------------------------
89
90	With vIOMMU, an IO virtual address range can get unmapped while in pre-copy
91	phase of migration. In that case, the unmap ioctl returns any dirty pages in
92	that range and QEMU reports corresponding guest physical pages dirty. During
93	stop-and-copy phase, an IOMMU notifier is used to get a callback for mapped
94	pages and then dirty pages bitmap is fetched from VFIO IOMMU modules for those
95	mapped ranges.
96
97	Flow of state changes during Live migration
98	===========================================
99
100	Below is the flow of state change during live migration.
101	The values in the brackets represent the VM state, the migration state, and
102	the VFIO device state, respectively.
103
104	Live migration save path
105	------------------------
106
107	::
108
109	QEMU normal running state
110	(RUNNING, _NONE, _RUNNING)
111	\|
112	migrate_init spawns migration_thread
113	Migration thread then calls each device's .save_setup()
114	(RUNNING, _SETUP, _RUNNING\|_SAVING)
115	\|
116	(RUNNING, _ACTIVE, _RUNNING\|_SAVING)
117	If device is active, get pending_bytes by .save_live_pending()
118	If total pending_bytes >= threshold_size, call .save_live_iterate()
119	Data of VFIO device for pre-copy phase is copied
120	Iterate till total pending bytes converge and are less than threshold
121	\|
122	On migration completion, vCPU stops and calls .save_live_complete_precopy for
123	each active device. The VFIO device is then transitioned into _SAVING state
124	(FINISH_MIGRATE, _DEVICE, _SAVING)
125	\|
126	For the VFIO device, iterate in .save_live_complete_precopy until
127	pending data is 0
128	(FINISH_MIGRATE, _DEVICE, _STOPPED)
129	\|
130	(FINISH_MIGRATE, _COMPLETED, _STOPPED)
131	Migraton thread schedules cleanup bottom half and exits
132
133	Live migration resume path
134	--------------------------
135
136	::
137
138	Incoming migration calls .load_setup for each device
139	(RESTORE_VM, _ACTIVE, _STOPPED)
140	\|
141	For each device, .load_state is called for that device section data
142	(RESTORE_VM, _ACTIVE, _RESUMING)
143	\|
144	At the end, .load_cleanup is called for each device and vCPUs are started
145	(RUNNING, _NONE, _RUNNING)
146
147	Postcopy
148	========
149
150	Postcopy migration is currently not supported for VFIO devices.