ceph/doc/rbd/rbd-live-migration.rst

   1 ======================
   2  Image Live-Migration
   3 ======================
   4
   5 .. index:: Ceph Block Device; live-migration
   6
   7 RBD images can be live-migrated between different pools within the same cluster;
   8 between different image formats and layouts; or from external data sources.
   9 When started, the source will be deep-copied to the destination image, pulling
  10 all snapshot history while preserving the sparse allocation of data where
  11 possible.
  12
  13 By default, when live-migrating RBD images within the same Ceph cluster, the
  14 source image will be marked read-only and all clients will instead redirect
  15 IOs to the new target image. In addition, this mode can optionally preserve the
  16 link to the source image's parent to preserve sparseness, or it can flatten the
  17 image during the migration to remove the dependency on the source image's
  18 parent.
  19
  20 The live-migration process can also be used in an import-only mode where the
  21 source image remains unmodified and the target image can be linked to an
  22 external data source such as a backing file, HTTP(s) file, or S3 object.
  23
  24 The live-migration copy process can safely run in the background while the new
  25 target image is in use. There is currently a requirement to temporarily stop
  26 using the source image before preparing a migration when not using the
  27 import-only mode of operation. This helps to ensure that the client using the
  28 image is updated to point to the new target image.
  29
  30 .. note::
  31    Image live-migration requires the Ceph Nautilus release or later. Support for
  32    external data sources requires the Ceph Pacific release of later. The
  33    ``krbd`` kernel module does not support live-migration at this time.
  34
  35
  36 .. ditaa::
  37
  38            +-------------+               +-------------+
  39            | {s} c999    |               | {s}         |
  40            |  Live       | Target refers |  Live       |
  41            |  migration  |<-------------*|  migration  |
  42            |  source     |  to Source    |  target     |
  43            |             |               |             |
  44            | (read only) |               | (writable)  |
  45            +-------------+               +-------------+
  46
  47                Source                        Target
  48
  49 The live-migration process is comprised of three steps:
  50
  51 #. **Prepare Migration:** The initial step creates the new target image and
  52    links the target image to the source. When not configured in the import-only
  53    mode, the source image will also be linked to the target image and marked
  54    read-only.
  55
  56    Similar to `layered images`_, attempts to read uninitialized data extents
  57    within the target image will internally redirect the read to the source
  58    image, and writes to uninitialized extents within the target will internally
  59    deep-copy the overlapping source image block to the target image.
  60
  61
  62 #. **Execute Migration:** This is a background operation that deep-copies all
  63    initialized blocks from the source image to the target. This step can be
  64    run while clients are actively using the new target image.
  65
  66
  67 #. **Finish Migration:** Once the background migration process has completed,
  68    the migration can be committed or aborted. Committing the migration will
  69    remove the cross-links between the source and target images, and will
  70    remove the source image if not configured in the import-only mode. Aborting
  71    the migration will remove the cross-links, and will remove the target image.
  72
  73 Prepare Migration
  74 =================
  75
  76 The default live-migration process for images within the same Ceph cluster is
  77 initiated by running the `rbd migration prepare` command, providing the source
  78 and target images::
  79
  80     $ rbd migration prepare migration_source [migration_target]
  81
  82 The `rbd migration prepare` command accepts all the same layout optionals as the
  83 `rbd create` command, which allows changes to the immutable image on-disk
  84 layout. The `migration_target` can be skipped if the goal is only to change the
  85 on-disk layout, keeping the original image name.
  86
  87 All clients using the source image must be stopped prior to preparing a
  88 live-migration. The prepare step will fail if it finds any running clients with
  89 the image open in read/write mode. Once the prepare step is complete, the
  90 clients can be restarted using the new target image name. Attempting to restart
  91 the clients using the source image name will result in failure.
  92
  93 The `rbd status` command will show the current state of the live-migration::
  94
  95     $ rbd status migration_target
  96     Watchers: none
  97     Migration:
  98                 source: rbd/migration_source (5e2cba2f62e)
  99                 destination: rbd/migration_target (5e2ed95ed806)
 100                 state: prepared
 101
 102 Note that the source image will be moved to the RBD trash to avoid mistaken
 103 usage during the migration process::
 104
 105     $ rbd info migration_source
 106     rbd: error opening image migration_source: (2) No such file or directory
 107     $ rbd trash ls --all
 108     5e2cba2f62e migration_source
 109
 110
 111 Prepare Import-Only Migration
 112 =============================
 113
 114 The import-only live-migration process is initiated by running the same
 115 `rbd migration prepare` command, but adding the `--import-only` optional
 116 and providing a JSON-encoded ``source-spec`` to describe how to access
 117 the source image data. This ``source-spec`` can either be passed
 118 directly via the `--source-spec` optional, or via a file or STDIN via the
 119 `--source-spec-file` optional::
 120
 121         $ rbd migration prepare --import-only --source-spec "<JSON>" migration_target
 122
 123 The `rbd migration prepare` command accepts all the same layout optionals as the
 124 `rbd create` command.
 125
 126 The `rbd status` command will show the current state of the live-migration::
 127
 128         $ rbd status migration_target
 129         Watchers: none
 130         Migration:
 131                 source: {"stream":{"file_path":"/mnt/image.raw","type":"file"},"type":"raw"}
 132                 destination: rbd/migration_target (ac69113dc1d7)
 133                 state: prepared
 134
 135 The general format for the ``source-spec`` JSON is as follows::
 136
 137         {
 138             "type": "<format-type>",
 139             <format unique parameters>
 140             "stream": {
 141                 "type": "<stream-type>",
 142                 <stream unique parameters>
 143             }
 144         }
 145
 146 The following formats are currently supported: ``native``, ``qcow``, and
 147 ``raw``. The following streams are currently supported: ``file``, ``http``, and
 148 ``s3``.
 149
 150 Formats
 151 ~~~~~~~
 152
 153 The ``native`` format can be used to describe a native RBD image within a
 154 Ceph cluster as the source image. Its ``source-spec`` JSON is encoded
 155 as follows::
 156
 157         {
 158             "type": "native",
 159             "pool_name": "<pool-name>",
 160             ["pool_id": <pool-id>,] (optional alternative to "pool_name")
 161             ["pool_namespace": "<pool-namespace",] (optional)
 162             "image_name": "<image-name>",
 163             ["image_id": "<image-id>",] (optional if image in trash)
 164             "snap_name": "<snap-name>",
 165             ["snap_id": "<snap-id>",] (optional alternative to "snap_name")
 166         }
 167
 168 Note that the ``native`` format does not include the ``stream`` object since
 169 it utilizes native Ceph operations. For example, to import from the image
 170 ``rbd/ns1/image1@snap1``, the ``source-spec`` could be encoded as::
 171
 172         {
 173             "type": "native",
 174             "pool_name": "rbd",
 175             "pool_namespace": "ns1",
 176             "image_name": "image1",
 177             "snap_name": "snap1"
 178         }
 179
 180 The ``qcow`` format can be used to describe a QCOW (QEMU copy-on-write) block
 181 device. Both the QCOW (v1) and QCOW2 formats are currently supported with the
 182 exception of advanced features such as compression, encryption, backing
 183 files, and external data files. Support for these missing features may be added
 184 in a future release. The ``qcow`` format data can be linked to any supported
 185 stream source described below. For example, its base ``source-spec`` JSON is
 186 encoded as follows::
 187
 188         {
 189             "type": "qcow",
 190             "stream": {
 191                 <stream unique parameters>
 192             }
 193         }
 194
 195 The ``raw`` format can be used to describe a thick-provisioned, raw block device
 196 export (i.e. `rbd export --export-format 1 <snap-spec>`). The ``raw`` format
 197 data can be linked to any supported stream source described below. For example,
 198 its base ``source-spec`` JSON is encoded as follows::
 199
 200         {
 201             "type": "raw",
 202             "stream": {
 203                 <stream unique parameters for HEAD, non-snapshot revision>
 204             },
 205             "snapshots": [
 206                 {
 207                     "type": "raw",
 208                     "name": "<snapshot-name>",
 209                     "stream": {
 210                         <stream unique parameters for snapshot>
 211                     }
 212                 },
 213             ] (optional oldest to newest ordering of snapshots)
 214         }
 215
 216 The inclusion of the ``snapshots`` array is optional and currently only supports
 217 thick-provisioned ``raw`` snapshot exports.
 218
 219 Additional formats such as RBD export-format v2 and RBD export-diff
 220 snapshots will be added in a future release.
 221
 222 Streams
 223 ~~~~~~~
 224
 225 The ``file`` stream can be used to import from a locally accessible POSIX file
 226 source. Its ``source-spec`` JSON is encoded as follows::
 227
 228         {
 229             <format unique parameters>
 230             "stream": {
 231                 "type": "file",
 232                 "file_path": "<file-path>"
 233             }
 234         }
 235
 236 For example, to import a raw-format image from a file located at
 237 "/mnt/image.raw", its ``source-spec`` JSON is encoded as follows::
 238
 239         {
 240             "type": "raw",
 241             "stream": {
 242                 "type": "file",
 243                 "file_path": "/mnt/image.raw"
 244             }
 245         }
 246
 247 The ``http`` stream can be used to import from a remote HTTP or HTTPS web
 248 server. Its ``source-spec`` JSON is encoded as follows::
 249
 250         {
 251             <format unique parameters>
 252             "stream": {
 253                 "type": "http",
 254                 "url": "<url-path>"
 255             }
 256         }
 257
 258 For example, to import a raw-format image from a file located at
 259 ``http://download.ceph.com/image.raw``, its ``source-spec`` JSON is encoded
 260 as follows::
 261
 262         {
 263             "type": "raw",
 264             "stream": {
 265                 "type": "http",
 266                 "url": "http://download.ceph.com/image.raw"
 267             }
 268         }
 269
 270 The ``s3`` stream can be used to import from a remote S3 bucket. Its
 271 ``source-spec`` JSON is encoded as follows::
 272
 273         {
 274             <format unique parameters>
 275             "stream": {
 276                 "type": "s3",
 277                 "url": "<url-path>",
 278                 "access_key": "<access-key>",
 279                 "secret_key": "<secret-key>"
 280             }
 281         }
 282
 283 For example, to import a raw-format image from a file located at
 284 `http://s3.ceph.com/bucket/image.raw`, its ``source-spec`` JSON is encoded
 285 as follows::
 286
 287         {
 288             "type": "raw",
 289             "stream": {
 290                 "type": "s3",
 291                 "url": "http://s3.ceph.com/bucket/image.raw",
 292                 "access_key": "NX5QOQKC6BH2IDN8HC7A",
 293                 "secret_key": "LnEsqNNqZIpkzauboDcLXLcYaWwLQ3Kop0zAnKIn"
 294             }
 295         }
 296
 297 .. note::
 298   The ``access_key`` and ``secret_key`` parameters support storing the keys in
 299   the MON config-key store by prefixing the key values with ``config://``
 300   followed by the path in the MON config-key store to the value. Values can be
 301   stored in the config-key store via ``ceph config-key set <key-path> <value>``
 302   (e.g. ``ceph config-key set rbd/s3/access_key NX5QOQKC6BH2IDN8HC7A``).
 303
 304 Execute Migration
 305 =================
 306
 307 After preparing the live-migration, the image blocks from the source image
 308 must be copied to the target image. This is accomplished by running the
 309 `rbd migration execute` command::
 310
 311     $ rbd migration execute migration_target
 312     Image migration: 100% complete...done.
 313
 314 The `rbd status` command will also provide feedback on the progress of the
 315 migration block deep-copy process::
 316
 317     $ rbd status migration_target
 318     Watchers:
 319         watcher=1.2.3.4:0/3695551461 client.123 cookie=123
 320     Migration:
 321                 source: rbd/migration_source (5e2cba2f62e)
 322                 destination: rbd/migration_target (5e2ed95ed806)
 323                 state: executing (32% complete)
 324
 325
 326 Commit Migration
 327 ================
 328
 329 Once the live-migration has completed deep-copying all data blocks from the
 330 source image to the target, the migration can be committed::
 331
 332     $ rbd status migration_target
 333     Watchers: none
 334     Migration:
 335                 source: rbd/migration_source (5e2cba2f62e)
 336                 destination: rbd/migration_target (5e2ed95ed806)
 337                 state: executed
 338     $ rbd migration commit migration_target
 339     Commit image migration: 100% complete...done.
 340
 341 If the `migration_source` image is a parent of one or more clones, the `--force`
 342 option will need to be specified after ensuring all descendent clone images are
 343 not in use.
 344
 345 Committing the live-migration will remove the cross-links between the source
 346 and target images, and will remove the source image::
 347
 348     $ rbd trash list --all
 349
 350
 351 Abort Migration
 352 ===============
 353
 354 If you wish to revert the prepare or execute step, run the `rbd migration abort`
 355 command to revert the migration process::
 356
 357         $ rbd migration abort migration_target
 358         Abort image migration: 100% complete...done.
 359
 360 Aborting the migration will result in the target image being deleted and access
 361 to the original source image being restored::
 362
 363         $ rbd ls
 364         migration_source
 365
 366
 367 .. _layered images: ../rbd-snapshot/#layering