5 .. index:: Ceph Block Device; mirroring
7 RBD images can be asynchronously mirrored between two Ceph clusters. This
8 capability is available in two modes:
10 * **Journal-based**: This mode uses the RBD journaling image feature to ensure
11 point-in-time, crash-consistent replication between clusters. Every write to
12 the RBD image is first recorded to the associated journal before modifying the
13 actual image. The remote cluster will read from this associated journal and
14 replay the updates to its local copy of the image. Since each write to the
15 RBD image will result in two writes to the Ceph cluster, expect write
16 latencies to nearly double while using the RBD journaling image feature.
18 * **Snapshot-based**: This mode uses periodically scheduled or manually
19 created RBD image mirror-snapshots to replicate crash-consistent RBD images
20 between clusters. The remote cluster will determine any data or metadata
21 updates between two mirror-snapshots and copy the deltas to its local copy of
22 the image. With the help of the RBD ``fast-diff`` image feature, updated data
23 blocks can be quickly determined without the need to scan the full RBD image.
24 Since this mode is not as fine-grained as journaling, the complete delta
25 between two snapshots will need to be synced prior to use during a failover
26 scenario. Any partially applied set of deltas will be rolled back at moment
29 .. note:: journal-based mirroring requires the Ceph Jewel release or later;
30 snapshot-based mirroring requires the Ceph Octopus release or later.
32 Mirroring is configured on a per-pool basis within peer clusters and can be
33 configured on a specific subset of images within the pool. You can also mirror
34 all images within a given pool when using journal-based
35 mirroring. Mirroring is configured using the ``rbd`` command. The
36 ``rbd-mirror`` daemon is responsible for pulling image updates from the remote
37 peer cluster and applying them to the image within the local cluster.
39 Depending on the desired needs for replication, RBD mirroring can be configured
40 for either one- or two-way replication:
42 * **One-way Replication**: When data is only mirrored from a primary cluster to
43 a secondary cluster, the ``rbd-mirror`` daemon runs only on the secondary
46 * **Two-way Replication**: When data is mirrored from primary images on one
47 cluster to non-primary images on another cluster (and vice-versa), the
48 ``rbd-mirror`` daemon runs on both clusters.
50 .. important:: Each instance of the ``rbd-mirror`` daemon must be able to
51 connect to both the local and remote Ceph clusters simultaneously (i.e.
52 all monitor and OSD hosts). Additionally, the network must have sufficient
53 bandwidth between the two data centers to handle mirroring workload.
58 The following procedures demonstrate how to perform the basic administrative
59 tasks to configure mirroring using the ``rbd`` command. Mirroring is
60 configured on a per-pool basis.
62 These pool configuration steps should be performed on both peer clusters. These
63 procedures assume that both clusters, named "site-a" and "site-b", are accessible
64 from a single host for clarity.
66 See the `rbd`_ manpage for additional details of how to connect to different
69 .. note:: The cluster name in the following examples corresponds to a Ceph
70 configuration file of the same name (e.g. /etc/ceph/site-b.conf). See the
71 `ceph-conf`_ documentation for how to configure multiple clusters. Note
72 that ``rbd-mirror`` does **not** require the source and destination clusters
73 to have unique internal names; both can and should call themselves ``ceph``.
74 The config `files` that ``rbd-mirror`` needs for local and remote clusters
75 can be named arbitrarily, and containerizing the daemon is one strategy
76 for maintaining them outside of ``/etc/ceph`` to avoid confusion.
81 To enable mirroring on a pool with ``rbd``, issue the ``mirror pool enable``
82 subcommand with the pool name, the mirroring mode, and an optional friendly
83 site name to describe the local cluster::
85 rbd mirror pool enable [--site-name {local-site-name}] {pool-name} {mode}
87 The mirroring mode can either be ``image`` or ``pool``:
89 * **image**: When configured in ``image`` mode, mirroring must
90 `explicitly enabled`_ on each image.
91 * **pool** (default): When configured in ``pool`` mode, all images in the pool
92 with the journaling feature enabled are mirrored.
96 $ rbd --cluster site-a mirror pool enable --site-name site-a image-pool image
97 $ rbd --cluster site-b mirror pool enable --site-name site-b image-pool image
99 The site name can also be specified when creating or importing a new
102 The site name can be changed later using the same ``mirror pool enable``
103 subcommand but note that the local site name and the corresponding site name
104 used by the remote cluster generally must match.
109 To disable mirroring on a pool with ``rbd``, specify the ``mirror pool disable``
110 command and the pool name::
112 rbd mirror pool disable {pool-name}
114 When mirroring is disabled on a pool in this way, mirroring will also be
115 disabled on any images (within the pool) for which mirroring was enabled
120 $ rbd --cluster site-a mirror pool disable image-pool
121 $ rbd --cluster site-b mirror pool disable image-pool
126 In order for the ``rbd-mirror`` daemon to discover its peer cluster, the peer
127 must be registered and a user account must be created.
128 This process can be automated with ``rbd`` and the
129 ``mirror pool peer bootstrap create`` and ``mirror pool peer bootstrap import``
132 To manually create a new bootstrap token with ``rbd``, issue the
133 ``mirror pool peer bootstrap create`` subcommand, a pool name, and an
134 optional friendly site name to describe the local cluster::
136 rbd mirror pool peer bootstrap create [--site-name {local-site-name}] {pool-name}
138 The output of ``mirror pool peer bootstrap create`` will be a token that should
139 be provided to the ``mirror pool peer bootstrap import`` command. For example,
142 $ rbd --cluster site-a mirror pool peer bootstrap create --site-name site-a image-pool
143 eyJmc2lkIjoiOWY1MjgyZGItYjg5OS00NTk2LTgwOTgtMzIwYzFmYzM5NmYzIiwiY2xpZW50X2lkIjoicmJkLW1pcnJvci1wZWVyIiwia2V5IjoiQVFBUnczOWQwdkhvQmhBQVlMM1I4RmR5dHNJQU50bkFTZ0lOTVE9PSIsIm1vbl9ob3N0IjoiW3YyOjE5Mi4xNjguMS4zOjY4MjAsdjE6MTkyLjE2OC4xLjM6NjgyMV0ifQ==
145 To manually import the bootstrap token created by another cluster with ``rbd``,
146 specify the ``mirror pool peer bootstrap import`` command, the pool name, a file
147 path to the created token (or '-' to read from standard input), along with an
148 optional friendly site name to describe the local cluster and a mirroring
149 direction (defaults to rx-tx for bidirectional mirroring, but can also be set
150 to rx-only for unidirectional mirroring)::
152 rbd mirror pool peer bootstrap import [--site-name {local-site-name}] [--direction {rx-only or rx-tx}] {pool-name} {token-path}
154 For example, on site-b::
157 eyJmc2lkIjoiOWY1MjgyZGItYjg5OS00NTk2LTgwOTgtMzIwYzFmYzM5NmYzIiwiY2xpZW50X2lkIjoicmJkLW1pcnJvci1wZWVyIiwia2V5IjoiQVFBUnczOWQwdkhvQmhBQVlMM1I4RmR5dHNJQU50bkFTZ0lOTVE9PSIsIm1vbl9ob3N0IjoiW3YyOjE5Mi4xNjguMS4zOjY4MjAsdjE6MTkyLjE2OC4xLjM6NjgyMV0ifQ==
159 $ rbd --cluster site-b mirror pool peer bootstrap import --site-name site-b image-pool token
161 Add Cluster Peer Manually
162 -------------------------
164 Cluster peers can be specified manually if desired or if the above bootstrap
165 commands are not available with the currently installed Ceph release.
167 The remote ``rbd-mirror`` daemon will need access to the local cluster to
168 perform mirroring. A new local Ceph user should be created for the remote
169 daemon to use. To `create a Ceph user`_, with ``ceph`` specify the
170 ``auth get-or-create`` command, user name, monitor caps, and OSD caps::
172 $ ceph auth get-or-create client.rbd-mirror-peer mon 'profile rbd-mirror-peer' osd 'profile rbd'
174 The resulting keyring should be copied to the other cluster's ``rbd-mirror``
175 daemon hosts if not using the Ceph monitor ``config-key`` store described below.
177 To manually add a mirroring peer Ceph cluster with ``rbd``, specify the
178 ``mirror pool peer add`` command, the pool name, and a cluster specification::
180 rbd mirror pool peer add {pool-name} {client-name}@{cluster-name}
184 $ rbd --cluster site-a mirror pool peer add image-pool client.rbd-mirror-peer@site-b
185 $ rbd --cluster site-b mirror pool peer add image-pool client.rbd-mirror-peer@site-a
187 By default, the ``rbd-mirror`` daemon needs to have access to a Ceph
188 configuration file located at ``/etc/ceph/{cluster-name}.conf`` that provides
189 the addresses of the peer cluster's monitors, in addition to a keyring for
190 ``{client-name}`` located in the default or configured keyring search paths
191 (e.g. ``/etc/ceph/{cluster-name}.{client-name}.keyring``).
193 Alternatively, the peer cluster's monitor and/or client key can be securely
194 stored within the local Ceph monitor ``config-key`` store. To specify the
195 peer cluster connection attributes when adding a mirroring peer, use the
196 ``--remote-mon-host`` and ``--remote-key-file`` optionals. For example::
198 $ cat <<EOF > remote-key-file
199 AQAeuZdbMMoBChAAcj++/XUxNOLFaWdtTREEsw==
201 $ rbd --cluster site-a mirror pool peer add image-pool client.rbd-mirror-peer@site-b --remote-mon-host 192.168.1.1,192.168.1.2 --remote-key-file remote-key-file
202 $ rbd --cluster site-a mirror pool info image-pool --all
205 UUID NAME CLIENT MON_HOST KEY
206 587b08db-3d33-4f32-8af8-421e77abb081 site-b client.rbd-mirror-peer 192.168.1.1,192.168.1.2 AQAeuZdbMMoBChAAcj++/XUxNOLFaWdtTREEsw==
211 To remove a mirroring peer Ceph cluster with ``rbd``, specify the
212 ``mirror pool peer remove`` command, the pool name, and the peer UUID
213 (available from the ``rbd mirror pool info`` command)::
215 rbd mirror pool peer remove {pool-name} {peer-uuid}
219 $ rbd --cluster site-a mirror pool peer remove image-pool 55672766-c02b-4729-8567-f13a66893445
220 $ rbd --cluster site-b mirror pool peer remove image-pool 60c0e299-b38f-4234-91f6-eed0a367be08
225 When creating images in the destination cluster, ``rbd-mirror`` selects a data
228 #. If the destination cluster has a default data pool configured (with the
229 ``rbd_default_data_pool`` configuration option), it will be used.
230 #. Otherwise, if the source image uses a separate data pool, and a pool with the
231 same name exists on the destination cluster, that pool will be used.
232 #. If neither of the above is true, no data pool will be set.
237 Unlike pool configuration, image configuration only needs to be performed
238 against a single mirroring peer Ceph cluster.
240 Mirrored RBD images are designated as either primary or non-primary. This is a
241 property of the image and not the pool. Images that are designated as
242 non-primary cannot be modified.
244 Images are automatically promoted to primary when mirroring is first enabled on
245 an image (either implicitly if the pool mirror mode was ``pool`` and the image
246 has the journaling image feature enabled, or `explicitly enabled`_ by the
247 ``rbd`` command if the pool mirror mode was ``image``).
249 Enable Image Mirroring
250 ----------------------
252 If mirroring is configured in ``image`` mode for the image's pool, then it
253 is necessary to explicitly enable mirroring for each image within the pool.
254 To enable mirroring for a specific image with ``rbd``, specify the
255 ``mirror image enable`` command along with the pool, image name, and mode::
257 rbd mirror image enable {pool-name}/{image-name} {mode}
259 The mirror image mode can either be ``journal`` or ``snapshot``:
261 * **journal** (default): When configured in ``journal`` mode, mirroring will
262 utilize the RBD journaling image feature to replicate the image contents. If
263 the RBD journaling image feature is not yet enabled on the image, it will be
264 automatically enabled.
266 * **snapshot**: When configured in ``snapshot`` mode, mirroring will utilize
267 RBD image mirror-snapshots to replicate the image contents. Once enabled, an
268 initial mirror-snapshot will automatically be created. Additional RBD image
269 `mirror-snapshots`_ can be created by the ``rbd`` command.
273 $ rbd --cluster site-a mirror image enable image-pool/image-1 snapshot
274 $ rbd --cluster site-a mirror image enable image-pool/image-2 journal
276 Enable Image Journaling Feature
277 -------------------------------
279 RBD journal-based mirroring uses the RBD image journaling feature to ensure that
280 the replicated image always remains crash-consistent. When using the ``image``
281 mirroring mode, the journaling feature will be automatically enabled when
282 mirroring is enabled on the image. When using the ``pool`` mirroring mode,
283 before an image can be mirrored to a peer cluster, the RBD image journaling
284 feature must be enabled. The feature can be enabled at image creation time by
285 providing the ``--image-feature exclusive-lock,journaling`` option to the
288 Alternatively, the journaling feature can be dynamically enabled on
289 pre-existing RBD images. To enable journaling with ``rbd``, specify
290 the ``feature enable`` command, the pool and image name, and the feature name::
292 rbd feature enable {pool-name}/{image-name} {feature-name}
296 $ rbd --cluster site-a feature enable image-pool/image-1 journaling
298 .. note:: The journaling feature is dependent on the exclusive-lock feature. If
299 the exclusive-lock feature is not already enabled, it should be enabled prior
300 to enabling the journaling feature.
302 .. tip:: You can enable journaling on all new images by default by adding
303 ``rbd default features = 125`` to your Ceph configuration file.
305 .. tip:: ``rbd-mirror`` tunables are set by default to values suitable for
306 mirroring an entire pool. When using ``rbd-mirror`` to migrate single
307 volumes been clusters you may achieve substantial performance gains
308 by setting ``rbd_mirror_journal_max_fetch_bytes=33554432`` and
309 ``rbd_journal_max_payload_bytes=8388608`` within the ``[client]`` config
310 section of the local or centralized configuration. Note that these
311 settings may allow ``rbd-mirror`` to present a substantial write workload
312 to the destination cluster: monitor cluster performance closely during
313 migrations and test carefully before running multiple migrations in parallel.
315 Create Image Mirror-Snapshots
316 -----------------------------
318 When using snapshot-based mirroring, mirror-snapshots will need to be created
319 whenever it is desired to mirror the changed contents of the RBD image. To
320 create a mirror-snapshot manually with ``rbd``, specify the
321 ``mirror image snapshot`` command along with the pool and image name::
323 rbd mirror image snapshot {pool-name}/{image-name}
327 $ rbd --cluster site-a mirror image snapshot image-pool/image-1
329 By default up to ``5`` mirror-snapshots will be created per-image. The most
330 recent mirror-snapshot is automatically pruned if the limit is reached.
331 The limit can be overridden via the ``rbd_mirroring_max_mirroring_snapshots``
332 configuration option if required. Additionally, mirror-snapshots are
333 automatically deleted when the image is removed or when mirroring is disabled.
335 Mirror-snapshots can also be automatically created on a periodic basis if
336 mirror-snapshot schedules are defined. The mirror-snapshot can be scheduled
337 globally, per-pool, or per-image levels. Multiple mirror-snapshot schedules can
338 be defined at any level, but only the most-specific snapshot schedules that
339 match an individual mirrored image will run.
341 To create a mirror-snapshot schedule with ``rbd``, specify the
342 ``mirror snapshot schedule add`` command along with an optional pool or
343 image name; interval; and optional start time::
345 rbd mirror snapshot schedule add [--pool {pool-name}] [--image {image-name}] {interval} [{start-time}]
347 The ``interval`` can be specified in days, hours, or minutes using ``d``, ``h``,
348 ``m`` suffix respectively. The optional ``start-time`` can be specified using
349 the ISO 8601 time format. For example::
351 $ rbd --cluster site-a mirror snapshot schedule add --pool image-pool 24h 14:00:00-05:00
352 $ rbd --cluster site-a mirror snapshot schedule add --pool image-pool --image image1 6h
354 To remove a mirror-snapshot schedules with ``rbd``, specify the
355 ``mirror snapshot schedule remove`` command with options that match the
356 corresponding ``add`` schedule command.
358 To list all snapshot schedules for a specific level (global, pool, or image)
359 with ``rbd``, specify the ``mirror snapshot schedule ls`` command along with
360 an optional pool or image name. Additionally, the ``--recursive`` option can
361 be specified to list all schedules at the specified level and below. For
364 $ rbd --cluster site-a mirror snapshot schedule ls --pool image-pool --recursive
365 POOL NAMESPACE IMAGE SCHEDULE
366 image-pool - - every 1d starting at 14:00:00-05:00
367 image-pool image1 every 6h
369 To view the status for when the next snapshots will be created for
370 snapshot-based mirroring RBD images with ``rbd``, specify the
371 ``mirror snapshot schedule status`` command along with an optional pool or
374 rbd mirror snapshot schedule status [--pool {pool-name}] [--image {image-name}]
378 $ rbd --cluster site-a mirror snapshot schedule status
380 2020-02-26 18:00:00 image-pool/image1
382 Disable Image Mirroring
383 -----------------------
385 To disable mirroring for a specific image with ``rbd``, specify the
386 ``mirror image disable`` command along with the pool and image name::
388 rbd mirror image disable {pool-name}/{image-name}
392 $ rbd --cluster site-a mirror image disable image-pool/image-1
394 Image Promotion and Demotion
395 ----------------------------
397 In a failover scenario where the primary designation needs to be moved to the
398 image in the peer Ceph cluster, access to the primary image should be stopped
399 (e.g. power down the VM or remove the associated drive from a VM), demote the
400 current primary image, promote the new primary image, and resume access to the
401 image on the alternate cluster.
403 .. note:: RBD only provides the necessary tools to facilitate an orderly
404 failover of an image. An external mechanism is required to coordinate the
405 full failover process (e.g. closing the image before demotion).
407 To demote a specific image to non-primary with ``rbd``, specify the
408 ``mirror image demote`` command along with the pool and image name::
410 rbd mirror image demote {pool-name}/{image-name}
414 $ rbd --cluster site-a mirror image demote image-pool/image-1
416 To demote all primary images within a pool to non-primary with ``rbd``, specify
417 the ``mirror pool demote`` command along with the pool name::
419 rbd mirror pool demote {pool-name}
423 $ rbd --cluster site-a mirror pool demote image-pool
425 To promote a specific image to primary with ``rbd``, specify the
426 ``mirror image promote`` command along with the pool and image name::
428 rbd mirror image promote [--force] {pool-name}/{image-name}
432 $ rbd --cluster site-b mirror image promote image-pool/image-1
434 To promote all non-primary images within a pool to primary with ``rbd``, specify
435 the ``mirror pool promote`` command along with the pool name::
437 rbd mirror pool promote [--force] {pool-name}
441 $ rbd --cluster site-a mirror pool promote image-pool
443 .. tip:: Since the primary / non-primary status is per-image, it is possible to
444 have two clusters split the IO load and stage failover / failback.
446 .. note:: Promotion can be forced using the ``--force`` option. Forced
447 promotion is needed when the demotion cannot be propagated to the peer
448 Ceph cluster (e.g. Ceph cluster failure, communication outage). This will
449 result in a split-brain scenario between the two peers and the image will no
450 longer be in-sync until a `force resync command`_ is issued.
455 If a split-brain event is detected by the ``rbd-mirror`` daemon, it will not
456 attempt to mirror the affected image until corrected. To resume mirroring for an
457 image, first `demote the image`_ determined to be out-of-date and then request a
458 resync to the primary image. To request an image resync with ``rbd``, specify
459 the ``mirror image resync`` command along with the pool and image name::
461 rbd mirror image resync {pool-name}/{image-name}
465 $ rbd mirror image resync image-pool/image-1
467 .. note:: The ``rbd`` command only flags the image as requiring a resync. The
468 local cluster's ``rbd-mirror`` daemon process is responsible for performing
469 the resync asynchronously.
474 The peer cluster replication status is stored for every primary mirrored image.
475 This status can be retrieved using the ``mirror image status`` and
476 ``mirror pool status`` commands.
478 To request the mirror image status with ``rbd``, specify the
479 ``mirror image status`` command along with the pool and image name::
481 rbd mirror image status {pool-name}/{image-name}
485 $ rbd mirror image status image-pool/image-1
487 To request the mirror pool summary status with ``rbd``, specify the
488 ``mirror pool status`` command along with the pool name::
490 rbd mirror pool status {pool-name}
494 $ rbd mirror pool status image-pool
496 .. note:: Adding ``--verbose`` option to the ``mirror pool status`` command will
497 additionally output status details for every mirroring image in the pool.
502 The two ``rbd-mirror`` daemons are responsible for watching image journals on
503 the remote, peer cluster and replaying the journal events against the local
504 cluster. The RBD image journaling feature records all modifications to the
505 image in the order they occur. This ensures that a crash-consistent mirror of
506 the remote image is available locally.
508 The ``rbd-mirror`` daemon is available within the optional ``rbd-mirror``
509 distribution package.
511 .. important:: Each ``rbd-mirror`` daemon requires the ability to connect
512 to both clusters simultaneously.
513 .. warning:: Pre-Luminous releases: only run a single ``rbd-mirror`` daemon per
516 Each ``rbd-mirror`` daemon should use a unique Ceph user ID. To
517 `create a Ceph user`_, with ``ceph`` specify the ``auth get-or-create``
518 command, user name, monitor caps, and OSD caps::
520 ceph auth get-or-create client.rbd-mirror.{unique id} mon 'profile rbd-mirror' osd 'profile rbd'
522 The ``rbd-mirror`` daemon can be managed by ``systemd`` by specifying the user
523 ID as the daemon instance::
525 systemctl enable ceph-rbd-mirror@rbd-mirror.{unique id}
527 The ``rbd-mirror`` can also be run in foreground by ``rbd-mirror`` command::
529 rbd-mirror -f --log-file={log_path}
531 .. _rbd: ../../man/8/rbd
532 .. _ceph-conf: ../../rados/configuration/ceph-conf/#running-multiple-clusters
533 .. _explicitly enabled: #enable-image-mirroring
534 .. _bootstrap token: #bootstrap-peers
535 .. _force resync command: #force-image-resync
536 .. _demote the image: #image-promotion-and-demotion
537 .. _create a Ceph user: ../../rados/operations/user-management#add-a-user
538 .. _mirror-snapshots: #create-image-mirror-snapshots