docs/bitmaps.md

   1 <!--
   2 Copyright 2015 John Snow <jsnow@redhat.com> and Red Hat, Inc.
   3 All rights reserved.
   4
   5 This file is licensed via The FreeBSD Documentation License, the full text of
   6 which is included at the end of this document.
   7 -->
   8
   9 # Dirty Bitmaps and Incremental Backup
  10
  11 * Dirty Bitmaps are objects that track which data needs to be backed up for the
  12   next incremental backup.
  13
  14 * Dirty bitmaps can be created at any time and attached to any node
  15   (not just complete drives.)
  16
  17 ## Dirty Bitmap Names
  18
  19 * A dirty bitmap's name is unique to the node, but bitmaps attached to different
  20   nodes can share the same name.
  21
  22 ## Bitmap Modes
  23
  24 * A Bitmap can be "frozen," which means that it is currently in-use by a backup
  25   operation and cannot be deleted, renamed, written to, reset,
  26   etc.
  27
  28 ## Basic QMP Usage
  29
  30 ### Supported Commands ###
  31
  32 * block-dirty-bitmap-add
  33 * block-dirty-bitmap-remove
  34 * block-dirty-bitmap-clear
  35
  36 ### Creation
  37
  38 * To create a new bitmap, enabled, on the drive with id=drive0:
  39
  40 ```json
  41 { "execute": "block-dirty-bitmap-add",
  42   "arguments": {
  43     "node": "drive0",
  44     "name": "bitmap0"
  45   }
  46 }
  47 ```
  48
  49 * This bitmap will have a default granularity that matches the cluster size of
  50   its associated drive, if available, clamped to between [4KiB, 64KiB].
  51   The current default for qcow2 is 64KiB.
  52
  53 * To create a new bitmap that tracks changes in 32KiB segments:
  54
  55 ```json
  56 { "execute": "block-dirty-bitmap-add",
  57   "arguments": {
  58     "node": "drive0",
  59     "name": "bitmap0",
  60     "granularity": 32768
  61   }
  62 }
  63 ```
  64
  65 ### Deletion
  66
  67 * Bitmaps that are frozen cannot be deleted.
  68
  69 * Deleting the bitmap does not impact any other bitmaps attached to the same
  70   node, nor does it affect any backups already created from this node.
  71
  72 * Because bitmaps are only unique to the node to which they are attached,
  73   you must specify the node/drive name here, too.
  74
  75 ```json
  76 { "execute": "block-dirty-bitmap-remove",
  77   "arguments": {
  78     "node": "drive0",
  79     "name": "bitmap0"
  80   }
  81 }
  82 ```
  83
  84 ### Resetting
  85
  86 * Resetting a bitmap will clear all information it holds.
  87
  88 * An incremental backup created from an empty bitmap will copy no data,
  89   as if nothing has changed.
  90
  91 ```json
  92 { "execute": "block-dirty-bitmap-clear",
  93   "arguments": {
  94     "node": "drive0",
  95     "name": "bitmap0"
  96   }
  97 }
  98 ```
  99
 100 ## Transactions
 101
 102 ### Justification
 103
 104 Bitmaps can be safely modified when the VM is paused or halted by using
 105 the basic QMP commands. For instance, you might perform the following actions:
 106
 107 1. Boot the VM in a paused state.
 108 2. Create a full drive backup of drive0.
 109 3. Create a new bitmap attached to drive0.
 110 4. Resume execution of the VM.
 111 5. Incremental backups are ready to be created.
 112
 113 At this point, the bitmap and drive backup would be correctly in sync,
 114 and incremental backups made from this point forward would be correctly aligned
 115 to the full drive backup.
 116
 117 This is not particularly useful if we decide we want to start incremental
 118 backups after the VM has been running for a while, for which we will need to
 119 perform actions such as the following:
 120
 121 1. Boot the VM and begin execution.
 122 2. Using a single transaction, perform the following operations:
 123     * Create bitmap0.
 124     * Create a full drive backup of drive0.
 125 3. Incremental backups are now ready to be created.
 126
 127 ### Supported Bitmap Transactions
 128
 129 * block-dirty-bitmap-add
 130 * block-dirty-bitmap-clear
 131
 132 The usages are identical to their respective QMP commands, but see below
 133 for examples.
 134
 135 ### Example: New Incremental Backup
 136
 137 As outlined in the justification, perhaps we want to create a new incremental
 138 backup chain attached to a drive.
 139
 140 ```json
 141 { "execute": "transaction",
 142   "arguments": {
 143     "actions": [
 144       {"type": "block-dirty-bitmap-add",
 145        "data": {"node": "drive0", "name": "bitmap0"} },
 146       {"type": "drive-backup",
 147        "data": {"device": "drive0", "target": "/path/to/full_backup.img",
 148                 "sync": "full", "format": "qcow2"} }
 149     ]
 150   }
 151 }
 152 ```
 153
 154 ### Example: New Incremental Backup Anchor Point
 155
 156 Maybe we just want to create a new full backup with an existing bitmap and
 157 want to reset the bitmap to track the new chain.
 158
 159 ```json
 160 { "execute": "transaction",
 161   "arguments": {
 162     "actions": [
 163       {"type": "block-dirty-bitmap-clear",
 164        "data": {"node": "drive0", "name": "bitmap0"} },
 165       {"type": "drive-backup",
 166        "data": {"device": "drive0", "target": "/path/to/new_full_backup.img",
 167                 "sync": "full", "format": "qcow2"} }
 168     ]
 169   }
 170 }
 171 ```
 172
 173 ## Incremental Backups
 174
 175 The star of the show.
 176
 177 **Nota Bene!** Only incremental backups of entire drives are supported for now.
 178 So despite the fact that you can attach a bitmap to any arbitrary node, they are
 179 only currently useful when attached to the root node. This is because
 180 drive-backup only supports drives/devices instead of arbitrary nodes.
 181
 182 ### Example: First Incremental Backup
 183
 184 1. Create a full backup and sync it to the dirty bitmap, as in the transactional
 185 examples above; or with the VM offline, manually create a full copy and then
 186 create a new bitmap before the VM begins execution.
 187
 188     * Let's assume the full backup is named 'full_backup.img'.
 189     * Let's assume the bitmap you created is 'bitmap0' attached to 'drive0'.
 190
 191 2. Create a destination image for the incremental backup that utilizes the
 192 full backup as a backing image.
 193
 194     * Let's assume it is named 'incremental.0.img'.
 195
 196     ```sh
 197     # qemu-img create -f qcow2 incremental.0.img -b full_backup.img -F qcow2
 198     ```
 199
 200 3. Issue the incremental backup command:
 201
 202     ```json
 203     { "execute": "drive-backup",
 204       "arguments": {
 205         "device": "drive0",
 206         "bitmap": "bitmap0",
 207         "target": "incremental.0.img",
 208         "format": "qcow2",
 209         "sync": "incremental",
 210         "mode": "existing"
 211       }
 212     }
 213     ```
 214
 215 ### Example: Second Incremental Backup
 216
 217 1. Create a new destination image for the incremental backup that points to the
 218    previous one, e.g.: 'incremental.1.img'
 219
 220     ```sh
 221     # qemu-img create -f qcow2 incremental.1.img -b incremental.0.img -F qcow2
 222     ```
 223
 224 2. Issue a new incremental backup command. The only difference here is that we
 225    have changed the target image below.
 226
 227     ```json
 228     { "execute": "drive-backup",
 229       "arguments": {
 230         "device": "drive0",
 231         "bitmap": "bitmap0",
 232         "target": "incremental.1.img",
 233         "format": "qcow2",
 234         "sync": "incremental",
 235         "mode": "existing"
 236       }
 237     }
 238     ```
 239
 240 ## Errors
 241
 242 * In the event of an error that occurs after a backup job is successfully
 243   launched, either by a direct QMP command or a QMP transaction, the user
 244   will receive a BLOCK_JOB_COMPLETE event with a failure message, accompanied
 245   by a BLOCK_JOB_ERROR event.
 246
 247 * In the case of an event being cancelled, the user will receive a
 248   BLOCK_JOB_CANCELLED event instead of a pair of COMPLETE and ERROR events.
 249
 250 * In either case, the incremental backup data contained within the bitmap is
 251   safely rolled back, and the data within the bitmap is not lost. The image
 252   file created for the failed attempt can be safely deleted.
 253
 254 * Once the underlying problem is fixed (e.g. more storage space is freed up),
 255   you can simply retry the incremental backup command with the same bitmap.
 256
 257 ### Example
 258
 259 1. Create a target image:
 260
 261     ```sh
 262     # qemu-img create -f qcow2 incremental.0.img -b full_backup.img -F qcow2
 263     ```
 264
 265 2. Attempt to create an incremental backup via QMP:
 266
 267     ```json
 268     { "execute": "drive-backup",
 269       "arguments": {
 270         "device": "drive0",
 271         "bitmap": "bitmap0",
 272         "target": "incremental.0.img",
 273         "format": "qcow2",
 274         "sync": "incremental",
 275         "mode": "existing"
 276       }
 277     }
 278     ```
 279
 280 3. Receive an event notifying us of failure:
 281
 282     ```json
 283     { "timestamp": { "seconds": 1424709442, "microseconds": 844524 },
 284       "data": { "speed": 0, "offset": 0, "len": 67108864,
 285                 "error": "No space left on device",
 286                 "device": "drive1", "type": "backup" },
 287       "event": "BLOCK_JOB_COMPLETED" }
 288     ```
 289
 290 4. Delete the failed incremental, and re-create the image.
 291
 292     ```sh
 293     # rm incremental.0.img
 294     # qemu-img create -f qcow2 incremental.0.img -b full_backup.img -F qcow2
 295     ```
 296
 297 5. Retry the command after fixing the underlying problem,
 298    such as freeing up space on the backup volume:
 299
 300     ```json
 301     { "execute": "drive-backup",
 302       "arguments": {
 303         "device": "drive0",
 304         "bitmap": "bitmap0",
 305         "target": "incremental.0.img",
 306         "format": "qcow2",
 307         "sync": "incremental",
 308         "mode": "existing"
 309       }
 310     }
 311     ```
 312
 313 6. Receive confirmation that the job completed successfully:
 314
 315     ```json
 316     { "timestamp": { "seconds": 1424709668, "microseconds": 526525 },
 317       "data": { "device": "drive1", "type": "backup",
 318                 "speed": 0, "len": 67108864, "offset": 67108864},
 319       "event": "BLOCK_JOB_COMPLETED" }
 320     ```
 321
 322 <!--
 323 The FreeBSD Documentation License
 324
 325 Redistribution and use in source (Markdown) and 'compiled' forms (SGML, HTML,
 326 PDF, PostScript, RTF and so forth) with or without modification, are permitted
 327 provided that the following conditions are met:
 328
 329 Redistributions of source code (Markdown) must retain the above copyright
 330 notice, this list of conditions and the following disclaimer of this file
 331 unmodified.
 332
 333 Redistributions in compiled form (transformed to other DTDs, converted to PDF,
 334 PostScript, RTF and other formats) must reproduce the above copyright notice,
 335 this list of conditions and the following disclaimer in the documentation and/or
 336 other materials provided with the distribution.
 337
 338 THIS DOCUMENTATION IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
 339 AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
 340 IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR  PURPOSE ARE
 341 DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS  BE LIABLE
 342 FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
 343 DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
 344 SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
 345 CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
 346 OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
 347 THIS DOCUMENTATION, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 348 -->