5 .. index:: Ceph Block Device; mirroring
7 RBD images can be asynchronously mirrored between two Ceph clusters. This
8 capability uses the RBD journaling image feature to ensure crash-consistent
9 replication between clusters. Mirroring is configured on a per-pool basis
10 within peer clusters and can be configured to automatically mirror all
11 images within a pool or only a specific subset of images. Mirroring is
12 configured using the ``rbd`` command. The ``rbd-mirror`` daemon is responsible
13 for pulling image updates from the remote, peer cluster and applying them to
14 the image within the local cluster.
16 .. note:: RBD mirroring requires the Ceph Jewel release or later.
18 .. important:: To use RBD mirroring, you must have two Ceph clusters, each
19 running the ``rbd-mirror`` daemon.
24 The following procedures demonstrate how to perform the basic administrative
25 tasks to configure mirroring using the ``rbd`` command. Mirroring is
26 configured on a per-pool basis within the Ceph clusters.
28 The pool configuration steps should be performed on both peer clusters. These
29 procedures assume two clusters, named "local" and "remote", are accessible from
30 a single host for clarity.
32 See the `rbd`_ manpage for additional details of how to connect to different
35 .. note:: The cluster name in the following examples corresponds to a Ceph
36 configuration file of the same name (e.g. /etc/ceph/remote.conf). See the
37 `ceph-conf`_ documentation for how to configure multiple clusters.
42 To enable mirroring on a pool with ``rbd``, specify the ``mirror pool enable``
43 command, the pool name, and the mirroring mode::
45 rbd mirror pool enable {pool-name} {mode}
47 The mirroring mode can either be ``pool`` or ``image``:
49 * **pool**: When configured in ``pool`` mode, all images in the pool with the
50 journaling feature enabled are mirrored.
51 * **image**: When configured in ``image`` mode, mirroring needs to be
52 `explicitly enabled`_ on each image.
56 rbd --cluster local mirror pool enable image-pool pool
57 rbd --cluster remote mirror pool enable image-pool pool
62 To disable mirroring on a pool with ``rbd``, specify the ``mirror pool disable``
63 command and the pool name::
65 rbd mirror pool disable {pool-name}
67 When mirroring is disabled on a pool in this way, mirroring will also be
68 disabled on any images (within the pool) for which mirroring was enabled
73 rbd --cluster local mirror pool disable image-pool
74 rbd --cluster remote mirror pool disable image-pool
79 In order for the ``rbd-mirror`` daemon to discover its peer cluster, the peer
80 needs to be registered to the pool. To add a mirroring peer Ceph cluster with
81 ``rbd``, specify the ``mirror pool peer add`` command, the pool name, and a
82 cluster specification::
84 rbd mirror pool peer add {pool-name} {client-name}@{cluster-name}
88 rbd --cluster local mirror pool peer add image-pool client.remote@remote
89 rbd --cluster remote mirror pool peer add image-pool client.local@local
94 To remove a mirroring peer Ceph cluster with ``rbd``, specify the
95 ``mirror pool peer remove`` command, the pool name, and the peer UUID
96 (available from the ``rbd mirror pool info`` command)::
98 rbd mirror pool peer remove {pool-name} {peer-uuid}
102 rbd --cluster local mirror pool peer remove image-pool 55672766-c02b-4729-8567-f13a66893445
103 rbd --cluster remote mirror pool peer remove image-pool 60c0e299-b38f-4234-91f6-eed0a367be08
108 Unlike pool configuration, image configuration only needs to be performed against
109 a single mirroring peer Ceph cluster.
111 Mirrored RBD images are designated as either primary or non-primary. This is a
112 property of the image and not the pool. Images that are designated as
113 non-primary cannot be modified.
115 Images are automatically promoted to primary when mirroring is first enabled on
116 an image (either implicitly if the pool mirror mode was **pool** and the image
117 has the journaling image feature enabled, or `explicitly enabled`_ by the
120 Enable Image Journaling Support
121 -------------------------------
123 RBD mirroring uses the RBD journaling feature to ensure that the replicated
124 image always remains crash-consistent. Before an image can be mirrored to
125 a peer cluster, the journaling feature must be enabled. The feature can be
126 enabled at image creation time by providing the
127 ``--image-feature exclusive-lock,journaling`` option to the ``rbd`` command.
129 Alternatively, the journaling feature can be dynamically enabled on
130 pre-existing RBD images. To enable journaling with ``rbd``, specify
131 the ``feature enable`` command, the pool and image name, and the feature name::
133 rbd feature enable {pool-name}/{image-name} {feature-name}
137 rbd --cluster local feature enable image-pool/image-1 journaling
139 .. note:: The journaling feature is dependent on the exclusive-lock feature. If
140 the exclusive-lock feature is not already enabled, it should be enabled prior
141 to enabling the journaling feature.
143 .. tip:: You can enable journaling on all new images by default by adding
144 ``rbd default features = 125`` to your Ceph configuration file.
146 Enable Image Mirroring
147 ----------------------
149 If the mirroring is configured in ``image`` mode for the image's pool, then it
150 is necessary to explicitly enable mirroring for each image within the pool.
151 To enable mirroring for a specific image with ``rbd``, specify the
152 ``mirror image enable`` command along with the pool and image name::
154 rbd mirror image enable {pool-name}/{image-name}
158 rbd --cluster local mirror image enable image-pool/image-1
160 Disable Image Mirroring
161 -----------------------
163 To disable mirroring for a specific image with ``rbd``, specify the
164 ``mirror image disable`` command along with the pool and image name::
166 rbd mirror image disable {pool-name}/{image-name}
170 rbd --cluster local mirror image disable image-pool/image-1
172 Image Promotion and Demotion
173 ----------------------------
175 In a failover scenario where the primary designation needs to be moved to the
176 image in the peer Ceph cluster, access to the primary image should be stopped
177 (e.g. power down the VM or remove the associated drive from a VM), demote the
178 current primary image, promote the new primary image, and resume access to the
179 image on the alternate cluster.
181 .. note:: RBD only provides the necessary tools to facilitate an orderly
182 failover of an image. An external mechanism is required to coordinate the
183 full failover process (e.g. closing the image before demotion).
185 To demote a specific image to non-primary with ``rbd``, specify the
186 ``mirror image demote`` command along with the pool and image name::
188 rbd mirror image demote {pool-name}/{image-name}
192 rbd --cluster local mirror image demote image-pool/image-1
194 To demote all primary images within a pool to non-primary with ``rbd``, specify
195 the ``mirror pool demote`` command along with the pool name::
197 rbd mirror pool demote {pool-name}
201 rbd --cluster local mirror pool demote image-pool
203 To promote a specific image to primary with ``rbd``, specify the
204 ``mirror image promote`` command along with the pool and image name::
206 rbd mirror image promote [--force] {pool-name}/{image-name}
210 rbd --cluster remote mirror image promote image-pool/image-1
212 To promote all non-primary images within a pool to primary with ``rbd``, specify
213 the ``mirror pool promote`` command along with the pool name::
215 rbd mirror pool promote [--force] {pool-name}
219 rbd --cluster local mirror pool promote image-pool
221 .. tip:: Since the primary / non-primary status is per-image, it is possible to
222 have two clusters split the IO load and stage failover / failback.
224 .. note:: Promotion can be forced using the ``--force`` option. Forced
225 promotion is needed when the demotion cannot be propagated to the peer
226 Ceph cluster (e.g. Ceph cluster failure, communication outage). This will
227 result in a split-brain scenario between the two peers and the image will no
228 longer be in-sync until a `force resync command`_ is issued.
233 If a split-brain event is detected by the ``rbd-mirror`` daemon, it will not
234 attempt to mirror the affected image until corrected. To resume mirroring for an
235 image, first `demote the image`_ determined to be out-of-date and then request a
236 resync to the primary image. To request an image resync with ``rbd``, specify the
237 ``mirror image resync`` command along with the pool and image name::
239 rbd mirror image resync {pool-name}/{image-name}
243 rbd mirror image resync image-pool/image-1
245 .. note:: The ``rbd`` command only flags the image as requiring a resync. The
246 local cluster's ``rbd-mirror`` daemon process is responsible for performing
247 the resync asynchronously.
252 The peer cluster replication status is stored for every primary mirrored image.
253 This status can be retrieved using the ``mirror image status`` and
254 ``mirror pool status`` commands.
256 To request the mirror image status with ``rbd``, specify the
257 ``mirror image status`` command along with the pool and image name::
259 rbd mirror image status {pool-name}/{image-name}
263 rbd mirror image status image-pool/image-1
265 To request the mirror pool summary status with ``rbd``, specify the
266 ``mirror pool status`` command along with the pool name::
268 rbd mirror pool status {pool-name}
272 rbd mirror pool status image-pool
274 .. note:: Adding ``--verbose`` option to the ``mirror pool status`` command will
275 additionally output status details for every mirroring image in the pool.
280 The two ``rbd-mirror`` daemons are responsible for watching image journals on the
281 remote, peer cluster and replaying the journal events against the local
282 cluster. The RBD image journaling feature records all modifications to the
283 image in the order they occur. This ensures that a crash-consistent mirror of
284 the remote image is available locally.
286 The ``rbd-mirror`` daemon is available within the optional ``rbd-mirror``
287 distribution package.
289 .. important:: Each ``rbd-mirror`` daemon requires the ability to connect
290 to both clusters simultaneously.
291 .. warning:: Only run a single ``rbd-mirror`` daemon per Ceph cluster. A
292 future Ceph release will add support for horizontal scale-out of the
293 ``rbd-mirror`` daemon.
295 .. _rbd: ../../man/8/rbd
296 .. _ceph-conf: ../../rados/configuration/ceph-conf/#running-multiple-clusters
297 .. _explicitly enabled: #enable-image-mirroring
298 .. _force resync command: #force-image-resync
299 .. _demote the image: #image-promotion-and-demotion