5 .. index:: Ceph Block Device; mirroring
7 RBD images can be asynchronously mirrored between two Ceph clusters. This
8 capability uses the RBD journaling image feature to ensure crash-consistent
9 replication between clusters. Mirroring is configured on a per-pool basis
10 within peer clusters and can be configured to automatically mirror all
11 images within a pool or only a specific subset of images. Mirroring is
12 configured using the ``rbd`` command. The ``rbd-mirror`` daemon is responsible
13 for pulling image updates from the remote, peer cluster and applying them to
14 the image within the local cluster.
16 .. note:: RBD mirroring requires the Ceph Jewel release or later.
18 .. important:: To use RBD mirroring, you must have two Ceph clusters, each
19 running the ``rbd-mirror`` daemon.
24 The following procedures demonstrate how to perform the basic administrative
25 tasks to configure mirroring using the ``rbd`` command. Mirroring is
26 configured on a per-pool basis within the Ceph clusters.
28 The pool configuration steps should be performed on both peer clusters. These
29 procedures assume two clusters, named "local" and "remote", are accessible from
30 a single host for clarity.
32 See the `rbd`_ manpage for additional details of how to connect to different
35 .. note:: The cluster name in the following examples corresponds to a Ceph
36 configuration file of the same name (e.g. /etc/ceph/remote.conf). See the
37 `ceph-conf`_ documentation for how to configure multiple clusters.
42 To enable mirroring on a pool with ``rbd``, specify the ``mirror pool enable``
43 command, the pool name, and the mirroring mode::
45 rbd mirror pool enable {pool-name} {mode}
47 The mirroring mode can either be ``pool`` or ``image``:
49 * **pool**: When configured in ``pool`` mode, all images in the pool with the
50 journaling feature enabled are mirrored.
51 * **image**: When configured in ``image`` mode, mirroring needs to be
52 `explicitly enabled`_ on each image.
56 rbd --cluster local mirror pool enable image-pool pool
57 rbd --cluster remote mirror pool enable image-pool pool
62 To disable mirroring on a pool with ``rbd``, specify the ``mirror pool disable``
63 command and the pool name::
65 rbd mirror pool disable {pool-name}
67 When mirroring is disabled on a pool in this way, mirroring will also be
68 disabled on any images (within the pool) for which mirroring was enabled
73 rbd --cluster local mirror pool disable image-pool
74 rbd --cluster remote mirror pool disable image-pool
79 In order for the ``rbd-mirror`` daemon to discover its peer cluster, the peer
80 needs to be registered to the pool. To add a mirroring peer Ceph cluster with
81 ``rbd``, specify the ``mirror pool peer add`` command, the pool name, and a
82 cluster specification::
84 rbd mirror pool peer add {pool-name} {client-name}@{cluster-name}
88 rbd --cluster local mirror pool peer add image-pool client.remote@remote
89 rbd --cluster remote mirror pool peer add image-pool client.local@local
94 To remove a mirroring peer Ceph cluster with ``rbd``, specify the
95 ``mirror pool peer remove`` command, the pool name, and the peer UUID
96 (available from the ``rbd mirror pool info`` command)::
98 rbd mirror pool peer remove {pool-name} {peer-uuid}
102 rbd --cluster local mirror pool peer remove image-pool 55672766-c02b-4729-8567-f13a66893445
103 rbd --cluster remote mirror pool peer remove image-pool 60c0e299-b38f-4234-91f6-eed0a367be08
108 When creating images in the destination cluster, ``rbd-mirror`` selects a data
111 #. If the destination cluster has a default data pool configured (with the
112 ``rbd_default_data_pool`` configuration option), it will be used.
113 #. Otherwise, if the source image uses a separate data pool, and a pool with the
114 same name exists on the destination cluster, that pool will be used.
115 #. If neither of the above is true, no data pool will be set.
120 Unlike pool configuration, image configuration only needs to be performed against
121 a single mirroring peer Ceph cluster.
123 Mirrored RBD images are designated as either primary or non-primary. This is a
124 property of the image and not the pool. Images that are designated as
125 non-primary cannot be modified.
127 Images are automatically promoted to primary when mirroring is first enabled on
128 an image (either implicitly if the pool mirror mode was **pool** and the image
129 has the journaling image feature enabled, or `explicitly enabled`_ by the
132 Enable Image Journaling Support
133 -------------------------------
135 RBD mirroring uses the RBD journaling feature to ensure that the replicated
136 image always remains crash-consistent. Before an image can be mirrored to
137 a peer cluster, the journaling feature must be enabled. The feature can be
138 enabled at image creation time by providing the
139 ``--image-feature exclusive-lock,journaling`` option to the ``rbd`` command.
141 Alternatively, the journaling feature can be dynamically enabled on
142 pre-existing RBD images. To enable journaling with ``rbd``, specify
143 the ``feature enable`` command, the pool and image name, and the feature name::
145 rbd feature enable {pool-name}/{image-name} {feature-name}
149 rbd --cluster local feature enable image-pool/image-1 journaling
151 .. note:: The journaling feature is dependent on the exclusive-lock feature. If
152 the exclusive-lock feature is not already enabled, it should be enabled prior
153 to enabling the journaling feature.
155 .. tip:: You can enable journaling on all new images by default by adding
156 ``rbd default features = 125`` to your Ceph configuration file.
158 Enable Image Mirroring
159 ----------------------
161 If the mirroring is configured in ``image`` mode for the image's pool, then it
162 is necessary to explicitly enable mirroring for each image within the pool.
163 To enable mirroring for a specific image with ``rbd``, specify the
164 ``mirror image enable`` command along with the pool and image name::
166 rbd mirror image enable {pool-name}/{image-name}
170 rbd --cluster local mirror image enable image-pool/image-1
172 Disable Image Mirroring
173 -----------------------
175 To disable mirroring for a specific image with ``rbd``, specify the
176 ``mirror image disable`` command along with the pool and image name::
178 rbd mirror image disable {pool-name}/{image-name}
182 rbd --cluster local mirror image disable image-pool/image-1
184 Image Promotion and Demotion
185 ----------------------------
187 In a failover scenario where the primary designation needs to be moved to the
188 image in the peer Ceph cluster, access to the primary image should be stopped
189 (e.g. power down the VM or remove the associated drive from a VM), demote the
190 current primary image, promote the new primary image, and resume access to the
191 image on the alternate cluster.
193 .. note:: RBD only provides the necessary tools to facilitate an orderly
194 failover of an image. An external mechanism is required to coordinate the
195 full failover process (e.g. closing the image before demotion).
197 To demote a specific image to non-primary with ``rbd``, specify the
198 ``mirror image demote`` command along with the pool and image name::
200 rbd mirror image demote {pool-name}/{image-name}
204 rbd --cluster local mirror image demote image-pool/image-1
206 To demote all primary images within a pool to non-primary with ``rbd``, specify
207 the ``mirror pool demote`` command along with the pool name::
209 rbd mirror pool demote {pool-name}
213 rbd --cluster local mirror pool demote image-pool
215 To promote a specific image to primary with ``rbd``, specify the
216 ``mirror image promote`` command along with the pool and image name::
218 rbd mirror image promote [--force] {pool-name}/{image-name}
222 rbd --cluster remote mirror image promote image-pool/image-1
224 To promote all non-primary images within a pool to primary with ``rbd``, specify
225 the ``mirror pool promote`` command along with the pool name::
227 rbd mirror pool promote [--force] {pool-name}
231 rbd --cluster local mirror pool promote image-pool
233 .. tip:: Since the primary / non-primary status is per-image, it is possible to
234 have two clusters split the IO load and stage failover / failback.
236 .. note:: Promotion can be forced using the ``--force`` option. Forced
237 promotion is needed when the demotion cannot be propagated to the peer
238 Ceph cluster (e.g. Ceph cluster failure, communication outage). This will
239 result in a split-brain scenario between the two peers and the image will no
240 longer be in-sync until a `force resync command`_ is issued.
245 If a split-brain event is detected by the ``rbd-mirror`` daemon, it will not
246 attempt to mirror the affected image until corrected. To resume mirroring for an
247 image, first `demote the image`_ determined to be out-of-date and then request a
248 resync to the primary image. To request an image resync with ``rbd``, specify the
249 ``mirror image resync`` command along with the pool and image name::
251 rbd mirror image resync {pool-name}/{image-name}
255 rbd mirror image resync image-pool/image-1
257 .. note:: The ``rbd`` command only flags the image as requiring a resync. The
258 local cluster's ``rbd-mirror`` daemon process is responsible for performing
259 the resync asynchronously.
264 The peer cluster replication status is stored for every primary mirrored image.
265 This status can be retrieved using the ``mirror image status`` and
266 ``mirror pool status`` commands.
268 To request the mirror image status with ``rbd``, specify the
269 ``mirror image status`` command along with the pool and image name::
271 rbd mirror image status {pool-name}/{image-name}
275 rbd mirror image status image-pool/image-1
277 To request the mirror pool summary status with ``rbd``, specify the
278 ``mirror pool status`` command along with the pool name::
280 rbd mirror pool status {pool-name}
284 rbd mirror pool status image-pool
286 .. note:: Adding ``--verbose`` option to the ``mirror pool status`` command will
287 additionally output status details for every mirroring image in the pool.
292 The two ``rbd-mirror`` daemons are responsible for watching image journals on the
293 remote, peer cluster and replaying the journal events against the local
294 cluster. The RBD image journaling feature records all modifications to the
295 image in the order they occur. This ensures that a crash-consistent mirror of
296 the remote image is available locally.
298 The ``rbd-mirror`` daemon is available within the optional ``rbd-mirror``
299 distribution package.
301 .. important:: Each ``rbd-mirror`` daemon requires the ability to connect
302 to both clusters simultaneously.
303 .. warning:: Pre-Luminous releases: only run a single ``rbd-mirror`` daemon per
306 Each ``rbd-mirror`` daemon should use a unique Ceph user ID. To
307 `create a Ceph user`_, with ``ceph`` specify the ``auth get-or-create``
308 command, user name, monitor caps, and OSD caps::
310 ceph auth get-or-create client.rbd-mirror.{unique id} mon 'profile rbd' osd 'profile rbd'
312 The ``rbd-mirror`` daemon can be managed by ``systemd`` by specifying the user
313 ID as the daemon instance::
315 systemctl enable ceph-rbd-mirror@rbd-mirror.{unique id}
317 .. _rbd: ../../man/8/rbd
318 .. _ceph-conf: ../../rados/configuration/ceph-conf/#running-multiple-clusters
319 .. _explicitly enabled: #enable-image-mirroring
320 .. _force resync command: #force-image-resync
321 .. _demote the image: #image-promotion-and-demotion
322 .. _create a Ceph user: ../../rados/operations/user-management#add-a-user