docs/devel/vfio-migration.rst

   1 =====================
   2 VFIO device Migration
   3 =====================
   4
   5 Migration of virtual machine involves saving the state for each device that
   6 the guest is running on source host and restoring this saved state on the
   7 destination host. This document details how saving and restoring of VFIO
   8 devices is done in QEMU.
   9
  10 Migration of VFIO devices consists of two phases: the optional pre-copy phase,
  11 and the stop-and-copy phase. The pre-copy phase is iterative and allows to
  12 accommodate VFIO devices that have a large amount of data that needs to be
  13 transferred. The iterative pre-copy phase of migration allows for the guest to
  14 continue whilst the VFIO device state is transferred to the destination, this
  15 helps to reduce the total downtime of the VM. VFIO devices can choose to skip
  16 the pre-copy phase of migration by returning pending_bytes as zero during the
  17 pre-copy phase.
  18
  19 A detailed description of the UAPI for VFIO device migration can be found in
  20 the comment for the ``vfio_device_migration_info`` structure in the header
  21 file linux-headers/linux/vfio.h.
  22
  23 VFIO implements the device hooks for the iterative approach as follows:
  24
  25 * A ``save_setup`` function that sets up the migration region and sets _SAVING
  26   flag in the VFIO device state.
  27
  28 * A ``load_setup`` function that sets up the migration region on the
  29   destination and sets _RESUMING flag in the VFIO device state.
  30
  31 * A ``save_live_pending`` function that reads pending_bytes from the vendor
  32   driver, which indicates the amount of data that the vendor driver has yet to
  33   save for the VFIO device.
  34
  35 * A ``save_live_iterate`` function that reads the VFIO device's data from the
  36   vendor driver through the migration region during iterative phase.
  37
  38 * A ``save_state`` function to save the device config space if it is present.
  39
  40 * A ``save_live_complete_precopy`` function that resets _RUNNING flag from the
  41   VFIO device state and iteratively copies the remaining data for the VFIO
  42   device until the vendor driver indicates that no data remains (pending bytes
  43   is zero).
  44
  45 * A ``load_state`` function that loads the config section and the data
  46   sections that are generated by the save functions above
  47
  48 * ``cleanup`` functions for both save and load that perform any migration
  49   related cleanup, including unmapping the migration region
  50
  51
  52 The VFIO migration code uses a VM state change handler to change the VFIO
  53 device state when the VM state changes from running to not-running, and
  54 vice versa.
  55
  56 Similarly, a migration state change handler is used to trigger a transition of
  57 the VFIO device state when certain changes of the migration state occur. For
  58 example, the VFIO device state is transitioned back to _RUNNING in case a
  59 migration failed or was canceled.
  60
  61 System memory dirty pages tracking
  62 ----------------------------------
  63
  64 A ``log_global_start`` and ``log_global_stop`` memory listener callback informs
  65 the VFIO IOMMU module to start and stop dirty page tracking. A ``log_sync``
  66 memory listener callback marks those system memory pages as dirty which are
  67 used for DMA by the VFIO device. The dirty pages bitmap is queried per
  68 container. All pages pinned by the vendor driver through external APIs have to
  69 be marked as dirty during migration. When there are CPU writes, CPU dirty page
  70 tracking can identify dirtied pages, but any page pinned by the vendor driver
  71 can also be written by the device. There is currently no device or IOMMU
  72 support for dirty page tracking in hardware.
  73
  74 By default, dirty pages are tracked when the device is in pre-copy as well as
  75 stop-and-copy phase. So, a page pinned by the vendor driver will be copied to
  76 the destination in both phases. Copying dirty pages in pre-copy phase helps
  77 QEMU to predict if it can achieve its downtime tolerances. If QEMU during
  78 pre-copy phase keeps finding dirty pages continuously, then it understands
  79 that even in stop-and-copy phase, it is likely to find dirty pages and can
  80 predict the downtime accordingly.
  81
  82 QEMU also provides a per device opt-out option ``pre-copy-dirty-page-tracking``
  83 which disables querying the dirty bitmap during pre-copy phase. If it is set to
  84 off, all dirty pages will be copied to the destination in stop-and-copy phase
  85 only.
  86
  87 System memory dirty pages tracking when vIOMMU is enabled
  88 ---------------------------------------------------------
  89
  90 With vIOMMU, an IO virtual address range can get unmapped while in pre-copy
  91 phase of migration. In that case, the unmap ioctl returns any dirty pages in
  92 that range and QEMU reports corresponding guest physical pages dirty. During
  93 stop-and-copy phase, an IOMMU notifier is used to get a callback for mapped
  94 pages and then dirty pages bitmap is fetched from VFIO IOMMU modules for those
  95 mapped ranges.
  96
  97 Flow of state changes during Live migration
  98 ===========================================
  99
 100 Below is the flow of state change during live migration.
 101 The values in the brackets represent the VM state, the migration state, and
 102 the VFIO device state, respectively.
 103
 104 Live migration save path
 105 ------------------------
 106
 107 ::
 108
 109                         QEMU normal running state
 110                         (RUNNING, _NONE, _RUNNING)
 111                                   |
 112                      migrate_init spawns migration_thread
 113                 Migration thread then calls each device's .save_setup()
 114                     (RUNNING, _SETUP, _RUNNING|_SAVING)
 115                                   |
 116                     (RUNNING, _ACTIVE, _RUNNING|_SAVING)
 117              If device is active, get pending_bytes by .save_live_pending()
 118           If total pending_bytes >= threshold_size, call .save_live_iterate()
 119                   Data of VFIO device for pre-copy phase is copied
 120         Iterate till total pending bytes converge and are less than threshold
 121                                   |
 122   On migration completion, vCPU stops and calls .save_live_complete_precopy for
 123    each active device. The VFIO device is then transitioned into _SAVING state
 124                    (FINISH_MIGRATE, _DEVICE, _SAVING)
 125                                   |
 126      For the VFIO device, iterate in .save_live_complete_precopy until
 127                          pending data is 0
 128                    (FINISH_MIGRATE, _DEVICE, _STOPPED)
 129                                   |
 130                  (FINISH_MIGRATE, _COMPLETED, _STOPPED)
 131              Migraton thread schedules cleanup bottom half and exits
 132
 133 Live migration resume path
 134 --------------------------
 135
 136 ::
 137
 138               Incoming migration calls .load_setup for each device
 139                        (RESTORE_VM, _ACTIVE, _STOPPED)
 140                                  |
 141        For each device, .load_state is called for that device section data
 142                        (RESTORE_VM, _ACTIVE, _RESUMING)
 143                                  |
 144     At the end, .load_cleanup is called for each device and vCPUs are started
 145                        (RUNNING, _NONE, _RUNNING)
 146
 147 Postcopy
 148 ========
 149
 150 Postcopy migration is currently not supported for VFIO devices.