5 .. index:: Ceph Block Device; mirroring
7 RBD images can be asynchronously mirrored between two Ceph clusters. This
8 capability uses the RBD journaling image feature to ensure crash-consistent
9 replication between clusters. Mirroring is configured on a per-pool basis
10 within peer clusters and can be configured to automatically mirror all
11 images within a pool or only a specific subset of images. Mirroring is
12 configured using the ``rbd`` command. The ``rbd-mirror`` daemon is responsible
13 for pulling image updates from the remote, peer cluster and applying them to
14 the image within the local cluster.
16 .. note:: RBD mirroring requires the Ceph Jewel release or later.
18 Depending on the desired needs for replication, RBD mirroring can be configured
19 for either one- or two-way replication:
21 * **One-way Replication**: When data is only mirrored from a primary cluster to
22 a secondary cluster, the ``rbd-mirror`` daemon runs only on the secondary
25 * **Two-way Replication**: When data is mirrored from primary images on one
26 cluster to non-primary images on another cluster (and vice-versa), the
27 ``rbd-mirror`` daemon runs on both clusters.
29 .. important:: Each instance of the ``rbd-mirror`` daemon must be able to
30 connect to both the local and remote Ceph clusters simultaneously (i.e.
31 all monitor and OSD hosts). Additionally, the network must have sufficient
32 bandwidth between the two data centers to handle mirroring workload.
37 The following procedures demonstrate how to perform the basic administrative
38 tasks to configure mirroring using the ``rbd`` command. Mirroring is
39 configured on a per-pool basis within the Ceph clusters.
41 The pool configuration steps should be performed on both peer clusters. These
42 procedures assume two clusters, named "local" and "remote", are accessible from
43 a single host for clarity.
45 See the `rbd`_ manpage for additional details of how to connect to different
48 .. note:: The cluster name in the following examples corresponds to a Ceph
49 configuration file of the same name (e.g. /etc/ceph/remote.conf). See the
50 `ceph-conf`_ documentation for how to configure multiple clusters.
55 To enable mirroring on a pool with ``rbd``, specify the ``mirror pool enable``
56 command, the pool name, and the mirroring mode::
58 rbd mirror pool enable {pool-name} {mode}
60 The mirroring mode can either be ``pool`` or ``image``:
62 * **pool**: When configured in ``pool`` mode, all images in the pool with the
63 journaling feature enabled are mirrored.
64 * **image**: When configured in ``image`` mode, mirroring needs to be
65 `explicitly enabled`_ on each image.
69 $ rbd --cluster local mirror pool enable image-pool pool
70 $ rbd --cluster remote mirror pool enable image-pool pool
75 To disable mirroring on a pool with ``rbd``, specify the ``mirror pool disable``
76 command and the pool name::
78 rbd mirror pool disable {pool-name}
80 When mirroring is disabled on a pool in this way, mirroring will also be
81 disabled on any images (within the pool) for which mirroring was enabled
86 $ rbd --cluster local mirror pool disable image-pool
87 $ rbd --cluster remote mirror pool disable image-pool
92 In order for the ``rbd-mirror`` daemon to discover its peer cluster, the peer
93 needs to be registered to the pool. To add a mirroring peer Ceph cluster with
94 ``rbd``, specify the ``mirror pool peer add`` command, the pool name, and a
95 cluster specification::
97 rbd mirror pool peer add {pool-name} {client-name}@{cluster-name}
101 $ rbd --cluster local mirror pool peer add image-pool client.remote@remote
102 $ rbd --cluster remote mirror pool peer add image-pool client.local@local
104 By default, the ``rbd-mirror`` daemon needs to have access to a Ceph
105 configuration file located at ``/etc/ceph/{cluster-name}.conf`` that provides
106 the addresses of the peer cluster's monitors, in addition to a keyring for
107 ``{client-name}`` located in the default or configured keyring search paths
108 (e.g. ``/etc/ceph/{cluster-name}.{client-name}.keyring``).
110 Alternatively, the peer cluster's monitor and/or client key can be securely
111 stored within the local Ceph monitor ``config-key`` store. To specify the
112 peer cluster connection attributes when adding a mirroring peer, use the
113 ``--remote-mon-host`` and ``--remote-key-file`` optionals. For example::
115 $ rbd --cluster local mirror pool peer add image-pool client.remote@remote --remote-mon-host 192.168.1.1,192.168.1.2 --remote-key-file <(echo 'AQAeuZdbMMoBChAAcj++/XUxNOLFaWdtTREEsw==')
116 $ rbd --cluster local mirror pool info image-pool --all
119 UUID NAME CLIENT MON_HOST KEY
120 587b08db-3d33-4f32-8af8-421e77abb081 remote client.remote 192.168.1.1,192.168.1.2 AQAeuZdbMMoBChAAcj++/XUxNOLFaWdtTREEsw==
125 To remove a mirroring peer Ceph cluster with ``rbd``, specify the
126 ``mirror pool peer remove`` command, the pool name, and the peer UUID
127 (available from the ``rbd mirror pool info`` command)::
129 rbd mirror pool peer remove {pool-name} {peer-uuid}
133 $ rbd --cluster local mirror pool peer remove image-pool 55672766-c02b-4729-8567-f13a66893445
134 $ rbd --cluster remote mirror pool peer remove image-pool 60c0e299-b38f-4234-91f6-eed0a367be08
139 When creating images in the destination cluster, ``rbd-mirror`` selects a data
142 #. If the destination cluster has a default data pool configured (with the
143 ``rbd_default_data_pool`` configuration option), it will be used.
144 #. Otherwise, if the source image uses a separate data pool, and a pool with the
145 same name exists on the destination cluster, that pool will be used.
146 #. If neither of the above is true, no data pool will be set.
151 Unlike pool configuration, image configuration only needs to be performed against
152 a single mirroring peer Ceph cluster.
154 Mirrored RBD images are designated as either primary or non-primary. This is a
155 property of the image and not the pool. Images that are designated as
156 non-primary cannot be modified.
158 Images are automatically promoted to primary when mirroring is first enabled on
159 an image (either implicitly if the pool mirror mode was **pool** and the image
160 has the journaling image feature enabled, or `explicitly enabled`_ by the
163 Enable Image Journaling Support
164 -------------------------------
166 RBD mirroring uses the RBD journaling feature to ensure that the replicated
167 image always remains crash-consistent. Before an image can be mirrored to
168 a peer cluster, the journaling feature must be enabled. The feature can be
169 enabled at image creation time by providing the
170 ``--image-feature exclusive-lock,journaling`` option to the ``rbd`` command.
172 Alternatively, the journaling feature can be dynamically enabled on
173 pre-existing RBD images. To enable journaling with ``rbd``, specify
174 the ``feature enable`` command, the pool and image name, and the feature name::
176 rbd feature enable {pool-name}/{image-name} {feature-name}
180 $ rbd --cluster local feature enable image-pool/image-1 journaling
182 .. note:: The journaling feature is dependent on the exclusive-lock feature. If
183 the exclusive-lock feature is not already enabled, it should be enabled prior
184 to enabling the journaling feature.
186 .. tip:: You can enable journaling on all new images by default by adding
187 ``rbd default features = 125`` to your Ceph configuration file.
189 Enable Image Mirroring
190 ----------------------
192 If the mirroring is configured in ``image`` mode for the image's pool, then it
193 is necessary to explicitly enable mirroring for each image within the pool.
194 To enable mirroring for a specific image with ``rbd``, specify the
195 ``mirror image enable`` command along with the pool and image name::
197 rbd mirror image enable {pool-name}/{image-name}
201 $ rbd --cluster local mirror image enable image-pool/image-1
203 Disable Image Mirroring
204 -----------------------
206 To disable mirroring for a specific image with ``rbd``, specify the
207 ``mirror image disable`` command along with the pool and image name::
209 rbd mirror image disable {pool-name}/{image-name}
213 $ rbd --cluster local mirror image disable image-pool/image-1
215 Image Promotion and Demotion
216 ----------------------------
218 In a failover scenario where the primary designation needs to be moved to the
219 image in the peer Ceph cluster, access to the primary image should be stopped
220 (e.g. power down the VM or remove the associated drive from a VM), demote the
221 current primary image, promote the new primary image, and resume access to the
222 image on the alternate cluster.
224 .. note:: RBD only provides the necessary tools to facilitate an orderly
225 failover of an image. An external mechanism is required to coordinate the
226 full failover process (e.g. closing the image before demotion).
228 To demote a specific image to non-primary with ``rbd``, specify the
229 ``mirror image demote`` command along with the pool and image name::
231 rbd mirror image demote {pool-name}/{image-name}
235 $ rbd --cluster local mirror image demote image-pool/image-1
237 To demote all primary images within a pool to non-primary with ``rbd``, specify
238 the ``mirror pool demote`` command along with the pool name::
240 rbd mirror pool demote {pool-name}
244 $ rbd --cluster local mirror pool demote image-pool
246 To promote a specific image to primary with ``rbd``, specify the
247 ``mirror image promote`` command along with the pool and image name::
249 rbd mirror image promote [--force] {pool-name}/{image-name}
253 $ rbd --cluster remote mirror image promote image-pool/image-1
255 To promote all non-primary images within a pool to primary with ``rbd``, specify
256 the ``mirror pool promote`` command along with the pool name::
258 rbd mirror pool promote [--force] {pool-name}
262 $ rbd --cluster local mirror pool promote image-pool
264 .. tip:: Since the primary / non-primary status is per-image, it is possible to
265 have two clusters split the IO load and stage failover / failback.
267 .. note:: Promotion can be forced using the ``--force`` option. Forced
268 promotion is needed when the demotion cannot be propagated to the peer
269 Ceph cluster (e.g. Ceph cluster failure, communication outage). This will
270 result in a split-brain scenario between the two peers and the image will no
271 longer be in-sync until a `force resync command`_ is issued.
276 If a split-brain event is detected by the ``rbd-mirror`` daemon, it will not
277 attempt to mirror the affected image until corrected. To resume mirroring for an
278 image, first `demote the image`_ determined to be out-of-date and then request a
279 resync to the primary image. To request an image resync with ``rbd``, specify the
280 ``mirror image resync`` command along with the pool and image name::
282 rbd mirror image resync {pool-name}/{image-name}
286 $ rbd mirror image resync image-pool/image-1
288 .. note:: The ``rbd`` command only flags the image as requiring a resync. The
289 local cluster's ``rbd-mirror`` daemon process is responsible for performing
290 the resync asynchronously.
295 The peer cluster replication status is stored for every primary mirrored image.
296 This status can be retrieved using the ``mirror image status`` and
297 ``mirror pool status`` commands.
299 To request the mirror image status with ``rbd``, specify the
300 ``mirror image status`` command along with the pool and image name::
302 rbd mirror image status {pool-name}/{image-name}
306 $ rbd mirror image status image-pool/image-1
308 To request the mirror pool summary status with ``rbd``, specify the
309 ``mirror pool status`` command along with the pool name::
311 rbd mirror pool status {pool-name}
315 $ rbd mirror pool status image-pool
317 .. note:: Adding ``--verbose`` option to the ``mirror pool status`` command will
318 additionally output status details for every mirroring image in the pool.
323 The two ``rbd-mirror`` daemons are responsible for watching image journals on the
324 remote, peer cluster and replaying the journal events against the local
325 cluster. The RBD image journaling feature records all modifications to the
326 image in the order they occur. This ensures that a crash-consistent mirror of
327 the remote image is available locally.
329 The ``rbd-mirror`` daemon is available within the optional ``rbd-mirror``
330 distribution package.
332 .. important:: Each ``rbd-mirror`` daemon requires the ability to connect
333 to both clusters simultaneously.
334 .. warning:: Pre-Luminous releases: only run a single ``rbd-mirror`` daemon per
337 Each ``rbd-mirror`` daemon should use a unique Ceph user ID. To
338 `create a Ceph user`_, with ``ceph`` specify the ``auth get-or-create``
339 command, user name, monitor caps, and OSD caps::
341 ceph auth get-or-create client.rbd-mirror.{unique id} mon 'profile rbd-mirror' osd 'profile rbd'
343 The ``rbd-mirror`` daemon can be managed by ``systemd`` by specifying the user
344 ID as the daemon instance::
346 systemctl enable ceph-rbd-mirror@rbd-mirror.{unique id}
348 The ``rbd-mirror`` can also be run in foreground by ``rbd-mirror`` command::
350 rbd-mirror -f --log-file={log_path}
352 .. _rbd: ../../man/8/rbd
353 .. _ceph-conf: ../../rados/configuration/ceph-conf/#running-multiple-clusters
354 .. _explicitly enabled: #enable-image-mirroring
355 .. _force resync command: #force-image-resync
356 .. _demote the image: #image-promotion-and-demotion
357 .. _create a Ceph user: ../../rados/operations/user-management#add-a-user