]>
Commit | Line | Data |
---|---|---|
7c673cae FG |
1 | =============== |
2 | RBD Mirroring | |
3 | =============== | |
4 | ||
5 | .. index:: Ceph Block Device; mirroring | |
6 | ||
7 | RBD images can be asynchronously mirrored between two Ceph clusters. This | |
9f95a23c TL |
8 | capability is available in two modes: |
9 | ||
10 | * **Journal-based**: This mode uses the RBD journaling image feature to ensure | |
11 | point-in-time, crash-consistent replication between clusters. Every write to | |
12 | the RBD image is first recorded to the associated journal before modifying the | |
13 | actual image. The remote cluster will read from this associated journal and | |
14 | replay the updates to its local copy of the image. Since each write to the | |
15 | RBD image will result in two writes to the Ceph cluster, expect write | |
f67539c2 | 16 | latencies to nearly double while using the RBD journaling image feature. |
9f95a23c TL |
17 | |
18 | * **Snapshot-based**: This mode uses periodically scheduled or manually | |
19 | created RBD image mirror-snapshots to replicate crash-consistent RBD images | |
20 | between clusters. The remote cluster will determine any data or metadata | |
21 | updates between two mirror-snapshots and copy the deltas to its local copy of | |
f67539c2 TL |
22 | the image. With the help of the RBD ``fast-diff`` image feature, updated data |
23 | blocks can be quickly determined without the need to scan the full RBD image. | |
24 | Since this mode is not as fine-grained as journaling, the complete delta | |
25 | between two snapshots will need to be synced prior to use during a failover | |
26 | scenario. Any partially applied set of deltas will be rolled back at moment | |
27 | of failover. | |
9f95a23c TL |
28 | |
29 | .. note:: journal-based mirroring requires the Ceph Jewel release or later; | |
30 | snapshot-based mirroring requires the Ceph Octopus release or later. | |
31 | ||
32 | Mirroring is configured on a per-pool basis within peer clusters and can be | |
f67539c2 TL |
33 | configured on a specific subset of images within the pool. You can also mirror |
34 | all images within a given pool when using journal-based | |
35 | mirroring. Mirroring is configured using the ``rbd`` command. The | |
36 | ``rbd-mirror`` daemon is responsible for pulling image updates from the remote | |
9f95a23c | 37 | peer cluster and applying them to the image within the local cluster. |
7c673cae | 38 | |
11fdf7f2 TL |
39 | Depending on the desired needs for replication, RBD mirroring can be configured |
40 | for either one- or two-way replication: | |
41 | ||
42 | * **One-way Replication**: When data is only mirrored from a primary cluster to | |
43 | a secondary cluster, the ``rbd-mirror`` daemon runs only on the secondary | |
44 | cluster. | |
45 | ||
46 | * **Two-way Replication**: When data is mirrored from primary images on one | |
47 | cluster to non-primary images on another cluster (and vice-versa), the | |
48 | ``rbd-mirror`` daemon runs on both clusters. | |
49 | ||
50 | .. important:: Each instance of the ``rbd-mirror`` daemon must be able to | |
51 | connect to both the local and remote Ceph clusters simultaneously (i.e. | |
52 | all monitor and OSD hosts). Additionally, the network must have sufficient | |
53 | bandwidth between the two data centers to handle mirroring workload. | |
7c673cae FG |
54 | |
55 | Pool Configuration | |
56 | ================== | |
57 | ||
58 | The following procedures demonstrate how to perform the basic administrative | |
59 | tasks to configure mirroring using the ``rbd`` command. Mirroring is | |
f67539c2 | 60 | configured on a per-pool basis. |
7c673cae | 61 | |
f67539c2 TL |
62 | These pool configuration steps should be performed on both peer clusters. These |
63 | procedures assume that both clusters, named "site-a" and "site-b", are accessible | |
64 | from a single host for clarity. | |
7c673cae FG |
65 | |
66 | See the `rbd`_ manpage for additional details of how to connect to different | |
67 | Ceph clusters. | |
68 | ||
69 | .. note:: The cluster name in the following examples corresponds to a Ceph | |
eafe8130 | 70 | configuration file of the same name (e.g. /etc/ceph/site-b.conf). See the |
f67539c2 TL |
71 | `ceph-conf`_ documentation for how to configure multiple clusters. Note |
72 | that ``rbd-mirror`` does **not** require the source and destination clusters | |
73 | to have unique internal names; both can and should call themselves ``ceph``. | |
74 | The config `files` that ``rbd-mirror`` needs for local and remote clusters | |
75 | can be named arbitrarily, and containerizing the daemon is one strategy | |
76 | for maintaining them outside of ``/etc/ceph`` to avoid confusion. | |
7c673cae FG |
77 | |
78 | Enable Mirroring | |
79 | ---------------- | |
80 | ||
f67539c2 | 81 | To enable mirroring on a pool with ``rbd``, issue the ``mirror pool enable`` |
20effc67 TL |
82 | subcommand with the pool name, the mirroring mode, and an optional friendly |
83 | site name to describe the local cluster:: | |
7c673cae | 84 | |
20effc67 | 85 | rbd mirror pool enable [--site-name {local-site-name}] {pool-name} {mode} |
7c673cae | 86 | |
9f95a23c | 87 | The mirroring mode can either be ``image`` or ``pool``: |
7c673cae | 88 | |
f67539c2 | 89 | * **image**: When configured in ``image`` mode, mirroring must |
7c673cae | 90 | `explicitly enabled`_ on each image. |
9f95a23c TL |
91 | * **pool** (default): When configured in ``pool`` mode, all images in the pool |
92 | with the journaling feature enabled are mirrored. | |
7c673cae FG |
93 | |
94 | For example:: | |
95 | ||
20effc67 TL |
96 | $ rbd --cluster site-a mirror pool enable --site-name site-a image-pool image |
97 | $ rbd --cluster site-b mirror pool enable --site-name site-b image-pool image | |
98 | ||
99 | The site name can also be specified when creating or importing a new | |
100 | `bootstrap token`_. | |
101 | ||
102 | The site name can be changed later using the same ``mirror pool enable`` | |
103 | subcommand but note that the local site name and the corresponding site name | |
104 | used by the remote cluster generally must match. | |
7c673cae FG |
105 | |
106 | Disable Mirroring | |
107 | ----------------- | |
108 | ||
109 | To disable mirroring on a pool with ``rbd``, specify the ``mirror pool disable`` | |
110 | command and the pool name:: | |
111 | ||
112 | rbd mirror pool disable {pool-name} | |
113 | ||
114 | When mirroring is disabled on a pool in this way, mirroring will also be | |
115 | disabled on any images (within the pool) for which mirroring was enabled | |
116 | explicitly. | |
117 | ||
118 | For example:: | |
119 | ||
eafe8130 TL |
120 | $ rbd --cluster site-a mirror pool disable image-pool |
121 | $ rbd --cluster site-b mirror pool disable image-pool | |
7c673cae | 122 | |
eafe8130 TL |
123 | Bootstrap Peers |
124 | --------------- | |
7c673cae FG |
125 | |
126 | In order for the ``rbd-mirror`` daemon to discover its peer cluster, the peer | |
f67539c2 | 127 | must be registered and a user account must be created. |
eafe8130 TL |
128 | This process can be automated with ``rbd`` and the |
129 | ``mirror pool peer bootstrap create`` and ``mirror pool peer bootstrap import`` | |
130 | commands. | |
131 | ||
f67539c2 TL |
132 | To manually create a new bootstrap token with ``rbd``, issue the |
133 | ``mirror pool peer bootstrap create`` subcommand, a pool name, and an | |
eafe8130 TL |
134 | optional friendly site name to describe the local cluster:: |
135 | ||
136 | rbd mirror pool peer bootstrap create [--site-name {local-site-name}] {pool-name} | |
137 | ||
138 | The output of ``mirror pool peer bootstrap create`` will be a token that should | |
139 | be provided to the ``mirror pool peer bootstrap import`` command. For example, | |
140 | on site-a:: | |
141 | ||
142 | $ rbd --cluster site-a mirror pool peer bootstrap create --site-name site-a image-pool | |
143 | eyJmc2lkIjoiOWY1MjgyZGItYjg5OS00NTk2LTgwOTgtMzIwYzFmYzM5NmYzIiwiY2xpZW50X2lkIjoicmJkLW1pcnJvci1wZWVyIiwia2V5IjoiQVFBUnczOWQwdkhvQmhBQVlMM1I4RmR5dHNJQU50bkFTZ0lOTVE9PSIsIm1vbl9ob3N0IjoiW3YyOjE5Mi4xNjguMS4zOjY4MjAsdjE6MTkyLjE2OC4xLjM6NjgyMV0ifQ== | |
144 | ||
145 | To manually import the bootstrap token created by another cluster with ``rbd``, | |
146 | specify the ``mirror pool peer bootstrap import`` command, the pool name, a file | |
147 | path to the created token (or '-' to read from standard input), along with an | |
148 | optional friendly site name to describe the local cluster and a mirroring | |
149 | direction (defaults to rx-tx for bidirectional mirroring, but can also be set | |
150 | to rx-only for unidirectional mirroring):: | |
151 | ||
152 | rbd mirror pool peer bootstrap import [--site-name {local-site-name}] [--direction {rx-only or rx-tx}] {pool-name} {token-path} | |
153 | ||
154 | For example, on site-b:: | |
155 | ||
156 | $ cat <<EOF > token | |
157 | eyJmc2lkIjoiOWY1MjgyZGItYjg5OS00NTk2LTgwOTgtMzIwYzFmYzM5NmYzIiwiY2xpZW50X2lkIjoicmJkLW1pcnJvci1wZWVyIiwia2V5IjoiQVFBUnczOWQwdkhvQmhBQVlMM1I4RmR5dHNJQU50bkFTZ0lOTVE9PSIsIm1vbl9ob3N0IjoiW3YyOjE5Mi4xNjguMS4zOjY4MjAsdjE6MTkyLjE2OC4xLjM6NjgyMV0ifQ== | |
158 | EOF | |
159 | $ rbd --cluster site-b mirror pool peer bootstrap import --site-name site-b image-pool token | |
160 | ||
161 | Add Cluster Peer Manually | |
162 | ------------------------- | |
163 | ||
164 | Cluster peers can be specified manually if desired or if the above bootstrap | |
165 | commands are not available with the currently installed Ceph release. | |
166 | ||
167 | The remote ``rbd-mirror`` daemon will need access to the local cluster to | |
168 | perform mirroring. A new local Ceph user should be created for the remote | |
169 | daemon to use. To `create a Ceph user`_, with ``ceph`` specify the | |
170 | ``auth get-or-create`` command, user name, monitor caps, and OSD caps:: | |
171 | ||
20effc67 | 172 | $ ceph auth get-or-create client.rbd-mirror-peer mon 'profile rbd-mirror-peer' osd 'profile rbd' |
eafe8130 TL |
173 | |
174 | The resulting keyring should be copied to the other cluster's ``rbd-mirror`` | |
175 | daemon hosts if not using the Ceph monitor ``config-key`` store described below. | |
176 | ||
177 | To manually add a mirroring peer Ceph cluster with ``rbd``, specify the | |
178 | ``mirror pool peer add`` command, the pool name, and a cluster specification:: | |
7c673cae FG |
179 | |
180 | rbd mirror pool peer add {pool-name} {client-name}@{cluster-name} | |
181 | ||
182 | For example:: | |
183 | ||
eafe8130 TL |
184 | $ rbd --cluster site-a mirror pool peer add image-pool client.rbd-mirror-peer@site-b |
185 | $ rbd --cluster site-b mirror pool peer add image-pool client.rbd-mirror-peer@site-a | |
11fdf7f2 TL |
186 | |
187 | By default, the ``rbd-mirror`` daemon needs to have access to a Ceph | |
188 | configuration file located at ``/etc/ceph/{cluster-name}.conf`` that provides | |
189 | the addresses of the peer cluster's monitors, in addition to a keyring for | |
190 | ``{client-name}`` located in the default or configured keyring search paths | |
191 | (e.g. ``/etc/ceph/{cluster-name}.{client-name}.keyring``). | |
192 | ||
193 | Alternatively, the peer cluster's monitor and/or client key can be securely | |
194 | stored within the local Ceph monitor ``config-key`` store. To specify the | |
195 | peer cluster connection attributes when adding a mirroring peer, use the | |
196 | ``--remote-mon-host`` and ``--remote-key-file`` optionals. For example:: | |
197 | ||
eafe8130 TL |
198 | $ cat <<EOF > remote-key-file |
199 | AQAeuZdbMMoBChAAcj++/XUxNOLFaWdtTREEsw== | |
200 | EOF | |
201 | $ rbd --cluster site-a mirror pool peer add image-pool client.rbd-mirror-peer@site-b --remote-mon-host 192.168.1.1,192.168.1.2 --remote-key-file remote-key-file | |
202 | $ rbd --cluster site-a mirror pool info image-pool --all | |
11fdf7f2 TL |
203 | Mode: pool |
204 | Peers: | |
eafe8130 TL |
205 | UUID NAME CLIENT MON_HOST KEY |
206 | 587b08db-3d33-4f32-8af8-421e77abb081 site-b client.rbd-mirror-peer 192.168.1.1,192.168.1.2 AQAeuZdbMMoBChAAcj++/XUxNOLFaWdtTREEsw== | |
7c673cae FG |
207 | |
208 | Remove Cluster Peer | |
209 | ------------------- | |
210 | ||
211 | To remove a mirroring peer Ceph cluster with ``rbd``, specify the | |
212 | ``mirror pool peer remove`` command, the pool name, and the peer UUID | |
213 | (available from the ``rbd mirror pool info`` command):: | |
214 | ||
215 | rbd mirror pool peer remove {pool-name} {peer-uuid} | |
216 | ||
217 | For example:: | |
218 | ||
eafe8130 TL |
219 | $ rbd --cluster site-a mirror pool peer remove image-pool 55672766-c02b-4729-8567-f13a66893445 |
220 | $ rbd --cluster site-b mirror pool peer remove image-pool 60c0e299-b38f-4234-91f6-eed0a367be08 | |
7c673cae | 221 | |
b32b8144 FG |
222 | Data Pools |
223 | ---------- | |
224 | ||
225 | When creating images in the destination cluster, ``rbd-mirror`` selects a data | |
226 | pool as follows: | |
227 | ||
228 | #. If the destination cluster has a default data pool configured (with the | |
229 | ``rbd_default_data_pool`` configuration option), it will be used. | |
230 | #. Otherwise, if the source image uses a separate data pool, and a pool with the | |
231 | same name exists on the destination cluster, that pool will be used. | |
232 | #. If neither of the above is true, no data pool will be set. | |
233 | ||
7c673cae FG |
234 | Image Configuration |
235 | =================== | |
236 | ||
9f95a23c TL |
237 | Unlike pool configuration, image configuration only needs to be performed |
238 | against a single mirroring peer Ceph cluster. | |
7c673cae FG |
239 | |
240 | Mirrored RBD images are designated as either primary or non-primary. This is a | |
241 | property of the image and not the pool. Images that are designated as | |
242 | non-primary cannot be modified. | |
243 | ||
244 | Images are automatically promoted to primary when mirroring is first enabled on | |
9f95a23c | 245 | an image (either implicitly if the pool mirror mode was ``pool`` and the image |
7c673cae | 246 | has the journaling image feature enabled, or `explicitly enabled`_ by the |
9f95a23c TL |
247 | ``rbd`` command if the pool mirror mode was ``image``). |
248 | ||
249 | Enable Image Mirroring | |
250 | ---------------------- | |
251 | ||
252 | If mirroring is configured in ``image`` mode for the image's pool, then it | |
253 | is necessary to explicitly enable mirroring for each image within the pool. | |
254 | To enable mirroring for a specific image with ``rbd``, specify the | |
255 | ``mirror image enable`` command along with the pool, image name, and mode:: | |
256 | ||
257 | rbd mirror image enable {pool-name}/{image-name} {mode} | |
258 | ||
259 | The mirror image mode can either be ``journal`` or ``snapshot``: | |
7c673cae | 260 | |
9f95a23c TL |
261 | * **journal** (default): When configured in ``journal`` mode, mirroring will |
262 | utilize the RBD journaling image feature to replicate the image contents. If | |
263 | the RBD journaling image feature is not yet enabled on the image, it will be | |
264 | automatically enabled. | |
265 | ||
266 | * **snapshot**: When configured in ``snapshot`` mode, mirroring will utilize | |
267 | RBD image mirror-snapshots to replicate the image contents. Once enabled, an | |
268 | initial mirror-snapshot will automatically be created. Additional RBD image | |
269 | `mirror-snapshots`_ can be created by the ``rbd`` command. | |
270 | ||
271 | For example:: | |
272 | ||
273 | $ rbd --cluster site-a mirror image enable image-pool/image-1 snapshot | |
274 | $ rbd --cluster site-a mirror image enable image-pool/image-2 journal | |
275 | ||
276 | Enable Image Journaling Feature | |
7c673cae FG |
277 | ------------------------------- |
278 | ||
9f95a23c TL |
279 | RBD journal-based mirroring uses the RBD image journaling feature to ensure that |
280 | the replicated image always remains crash-consistent. When using the ``image`` | |
281 | mirroring mode, the journaling feature will be automatically enabled when | |
282 | mirroring is enabled on the image. When using the ``pool`` mirroring mode, | |
283 | before an image can be mirrored to a peer cluster, the RBD image journaling | |
284 | feature must be enabled. The feature can be enabled at image creation time by | |
285 | providing the ``--image-feature exclusive-lock,journaling`` option to the | |
286 | ``rbd`` command. | |
7c673cae FG |
287 | |
288 | Alternatively, the journaling feature can be dynamically enabled on | |
289 | pre-existing RBD images. To enable journaling with ``rbd``, specify | |
290 | the ``feature enable`` command, the pool and image name, and the feature name:: | |
291 | ||
292 | rbd feature enable {pool-name}/{image-name} {feature-name} | |
293 | ||
294 | For example:: | |
295 | ||
eafe8130 | 296 | $ rbd --cluster site-a feature enable image-pool/image-1 journaling |
7c673cae FG |
297 | |
298 | .. note:: The journaling feature is dependent on the exclusive-lock feature. If | |
299 | the exclusive-lock feature is not already enabled, it should be enabled prior | |
300 | to enabling the journaling feature. | |
301 | ||
302 | .. tip:: You can enable journaling on all new images by default by adding | |
303 | ``rbd default features = 125`` to your Ceph configuration file. | |
304 | ||
f67539c2 TL |
305 | .. tip:: ``rbd-mirror`` tunables are set by default to values suitable for |
306 | mirroring an entire pool. When using ``rbd-mirror`` to migrate single | |
307 | volumes been clusters you may achieve substantial performance gains | |
308 | by setting ``rbd_mirror_journal_max_fetch_bytes=33554432`` and | |
309 | ``rbd_journal_max_payload_bytes=8388608`` within the ``[client]`` config | |
310 | section of the local or centralized configuration. Note that these | |
311 | settings may allow ``rbd-mirror`` to present a substantial write workload | |
312 | to the destination cluster: monitor cluster performance closely during | |
20effc67 | 313 | migrations and test carefully before running multiple migrations in parallel. |
f67539c2 | 314 | |
9f95a23c TL |
315 | Create Image Mirror-Snapshots |
316 | ----------------------------- | |
7c673cae | 317 | |
9f95a23c TL |
318 | When using snapshot-based mirroring, mirror-snapshots will need to be created |
319 | whenever it is desired to mirror the changed contents of the RBD image. To | |
320 | create a mirror-snapshot manually with ``rbd``, specify the | |
321 | ``mirror image snapshot`` command along with the pool and image name:: | |
322 | ||
323 | rbd mirror image snapshot {pool-name}/{image-name} | |
324 | ||
325 | For example:: | |
326 | ||
327 | $ rbd --cluster site-a mirror image snapshot image-pool/image-1 | |
328 | ||
2a845540 | 329 | By default up to ``5`` mirror-snapshots will be created per-image. The most |
9f95a23c TL |
330 | recent mirror-snapshot is automatically pruned if the limit is reached. |
331 | The limit can be overridden via the ``rbd_mirroring_max_mirroring_snapshots`` | |
332 | configuration option if required. Additionally, mirror-snapshots are | |
333 | automatically deleted when the image is removed or when mirroring is disabled. | |
334 | ||
335 | Mirror-snapshots can also be automatically created on a periodic basis if | |
336 | mirror-snapshot schedules are defined. The mirror-snapshot can be scheduled | |
337 | globally, per-pool, or per-image levels. Multiple mirror-snapshot schedules can | |
338 | be defined at any level, but only the most-specific snapshot schedules that | |
339 | match an individual mirrored image will run. | |
340 | ||
341 | To create a mirror-snapshot schedule with ``rbd``, specify the | |
342 | ``mirror snapshot schedule add`` command along with an optional pool or | |
343 | image name; interval; and optional start time:: | |
7c673cae | 344 | |
9f95a23c TL |
345 | rbd mirror snapshot schedule add [--pool {pool-name}] [--image {image-name}] {interval} [{start-time}] |
346 | ||
347 | The ``interval`` can be specified in days, hours, or minutes using ``d``, ``h``, | |
348 | ``m`` suffix respectively. The optional ``start-time`` can be specified using | |
349 | the ISO 8601 time format. For example:: | |
350 | ||
351 | $ rbd --cluster site-a mirror snapshot schedule add --pool image-pool 24h 14:00:00-05:00 | |
352 | $ rbd --cluster site-a mirror snapshot schedule add --pool image-pool --image image1 6h | |
353 | ||
354 | To remove a mirror-snapshot schedules with ``rbd``, specify the | |
355 | ``mirror snapshot schedule remove`` command with options that match the | |
356 | corresponding ``add`` schedule command. | |
357 | ||
358 | To list all snapshot schedules for a specific level (global, pool, or image) | |
359 | with ``rbd``, specify the ``mirror snapshot schedule ls`` command along with | |
360 | an optional pool or image name. Additionally, the ``--recursive`` option can | |
361 | be specified to list all schedules at the specified level and below. For | |
362 | example:: | |
363 | ||
20effc67 | 364 | $ rbd --cluster site-a mirror snapshot schedule ls --pool image-pool --recursive |
9f95a23c TL |
365 | POOL NAMESPACE IMAGE SCHEDULE |
366 | image-pool - - every 1d starting at 14:00:00-05:00 | |
367 | image-pool image1 every 6h | |
368 | ||
369 | To view the status for when the next snapshots will be created for | |
370 | snapshot-based mirroring RBD images with ``rbd``, specify the | |
371 | ``mirror snapshot schedule status`` command along with an optional pool or | |
372 | image name:: | |
373 | ||
374 | rbd mirror snapshot schedule status [--pool {pool-name}] [--image {image-name}] | |
7c673cae FG |
375 | |
376 | For example:: | |
377 | ||
20effc67 | 378 | $ rbd --cluster site-a mirror snapshot schedule status |
9f95a23c TL |
379 | SCHEDULE TIME IMAGE |
380 | 2020-02-26 18:00:00 image-pool/image1 | |
7c673cae FG |
381 | |
382 | Disable Image Mirroring | |
383 | ----------------------- | |
384 | ||
385 | To disable mirroring for a specific image with ``rbd``, specify the | |
386 | ``mirror image disable`` command along with the pool and image name:: | |
387 | ||
388 | rbd mirror image disable {pool-name}/{image-name} | |
389 | ||
390 | For example:: | |
391 | ||
eafe8130 | 392 | $ rbd --cluster site-a mirror image disable image-pool/image-1 |
7c673cae FG |
393 | |
394 | Image Promotion and Demotion | |
395 | ---------------------------- | |
396 | ||
397 | In a failover scenario where the primary designation needs to be moved to the | |
398 | image in the peer Ceph cluster, access to the primary image should be stopped | |
399 | (e.g. power down the VM or remove the associated drive from a VM), demote the | |
400 | current primary image, promote the new primary image, and resume access to the | |
401 | image on the alternate cluster. | |
402 | ||
403 | .. note:: RBD only provides the necessary tools to facilitate an orderly | |
404 | failover of an image. An external mechanism is required to coordinate the | |
405 | full failover process (e.g. closing the image before demotion). | |
406 | ||
407 | To demote a specific image to non-primary with ``rbd``, specify the | |
408 | ``mirror image demote`` command along with the pool and image name:: | |
409 | ||
410 | rbd mirror image demote {pool-name}/{image-name} | |
411 | ||
412 | For example:: | |
413 | ||
eafe8130 | 414 | $ rbd --cluster site-a mirror image demote image-pool/image-1 |
7c673cae FG |
415 | |
416 | To demote all primary images within a pool to non-primary with ``rbd``, specify | |
417 | the ``mirror pool demote`` command along with the pool name:: | |
418 | ||
419 | rbd mirror pool demote {pool-name} | |
420 | ||
421 | For example:: | |
422 | ||
eafe8130 | 423 | $ rbd --cluster site-a mirror pool demote image-pool |
7c673cae FG |
424 | |
425 | To promote a specific image to primary with ``rbd``, specify the | |
426 | ``mirror image promote`` command along with the pool and image name:: | |
427 | ||
428 | rbd mirror image promote [--force] {pool-name}/{image-name} | |
429 | ||
430 | For example:: | |
431 | ||
eafe8130 | 432 | $ rbd --cluster site-b mirror image promote image-pool/image-1 |
7c673cae FG |
433 | |
434 | To promote all non-primary images within a pool to primary with ``rbd``, specify | |
435 | the ``mirror pool promote`` command along with the pool name:: | |
436 | ||
437 | rbd mirror pool promote [--force] {pool-name} | |
438 | ||
439 | For example:: | |
440 | ||
eafe8130 | 441 | $ rbd --cluster site-a mirror pool promote image-pool |
7c673cae FG |
442 | |
443 | .. tip:: Since the primary / non-primary status is per-image, it is possible to | |
444 | have two clusters split the IO load and stage failover / failback. | |
445 | ||
446 | .. note:: Promotion can be forced using the ``--force`` option. Forced | |
447 | promotion is needed when the demotion cannot be propagated to the peer | |
448 | Ceph cluster (e.g. Ceph cluster failure, communication outage). This will | |
449 | result in a split-brain scenario between the two peers and the image will no | |
450 | longer be in-sync until a `force resync command`_ is issued. | |
451 | ||
452 | Force Image Resync | |
453 | ------------------ | |
454 | ||
455 | If a split-brain event is detected by the ``rbd-mirror`` daemon, it will not | |
456 | attempt to mirror the affected image until corrected. To resume mirroring for an | |
457 | image, first `demote the image`_ determined to be out-of-date and then request a | |
9f95a23c TL |
458 | resync to the primary image. To request an image resync with ``rbd``, specify |
459 | the ``mirror image resync`` command along with the pool and image name:: | |
7c673cae FG |
460 | |
461 | rbd mirror image resync {pool-name}/{image-name} | |
462 | ||
463 | For example:: | |
464 | ||
11fdf7f2 | 465 | $ rbd mirror image resync image-pool/image-1 |
7c673cae FG |
466 | |
467 | .. note:: The ``rbd`` command only flags the image as requiring a resync. The | |
468 | local cluster's ``rbd-mirror`` daemon process is responsible for performing | |
469 | the resync asynchronously. | |
470 | ||
471 | Mirror Status | |
472 | ============= | |
473 | ||
474 | The peer cluster replication status is stored for every primary mirrored image. | |
475 | This status can be retrieved using the ``mirror image status`` and | |
476 | ``mirror pool status`` commands. | |
477 | ||
478 | To request the mirror image status with ``rbd``, specify the | |
479 | ``mirror image status`` command along with the pool and image name:: | |
480 | ||
481 | rbd mirror image status {pool-name}/{image-name} | |
482 | ||
483 | For example:: | |
484 | ||
11fdf7f2 | 485 | $ rbd mirror image status image-pool/image-1 |
7c673cae FG |
486 | |
487 | To request the mirror pool summary status with ``rbd``, specify the | |
488 | ``mirror pool status`` command along with the pool name:: | |
489 | ||
490 | rbd mirror pool status {pool-name} | |
491 | ||
492 | For example:: | |
493 | ||
11fdf7f2 | 494 | $ rbd mirror pool status image-pool |
7c673cae FG |
495 | |
496 | .. note:: Adding ``--verbose`` option to the ``mirror pool status`` command will | |
497 | additionally output status details for every mirroring image in the pool. | |
498 | ||
499 | rbd-mirror Daemon | |
500 | ================= | |
501 | ||
9f95a23c TL |
502 | The two ``rbd-mirror`` daemons are responsible for watching image journals on |
503 | the remote, peer cluster and replaying the journal events against the local | |
7c673cae FG |
504 | cluster. The RBD image journaling feature records all modifications to the |
505 | image in the order they occur. This ensures that a crash-consistent mirror of | |
506 | the remote image is available locally. | |
507 | ||
508 | The ``rbd-mirror`` daemon is available within the optional ``rbd-mirror`` | |
509 | distribution package. | |
510 | ||
511 | .. important:: Each ``rbd-mirror`` daemon requires the ability to connect | |
512 | to both clusters simultaneously. | |
d2e6a577 FG |
513 | .. warning:: Pre-Luminous releases: only run a single ``rbd-mirror`` daemon per |
514 | Ceph cluster. | |
515 | ||
516 | Each ``rbd-mirror`` daemon should use a unique Ceph user ID. To | |
517 | `create a Ceph user`_, with ``ceph`` specify the ``auth get-or-create`` | |
518 | command, user name, monitor caps, and OSD caps:: | |
519 | ||
11fdf7f2 | 520 | ceph auth get-or-create client.rbd-mirror.{unique id} mon 'profile rbd-mirror' osd 'profile rbd' |
d2e6a577 FG |
521 | |
522 | The ``rbd-mirror`` daemon can be managed by ``systemd`` by specifying the user | |
523 | ID as the daemon instance:: | |
524 | ||
525 | systemctl enable ceph-rbd-mirror@rbd-mirror.{unique id} | |
7c673cae | 526 | |
11fdf7f2 TL |
527 | The ``rbd-mirror`` can also be run in foreground by ``rbd-mirror`` command:: |
528 | ||
529 | rbd-mirror -f --log-file={log_path} | |
530 | ||
7c673cae FG |
531 | .. _rbd: ../../man/8/rbd |
532 | .. _ceph-conf: ../../rados/configuration/ceph-conf/#running-multiple-clusters | |
533 | .. _explicitly enabled: #enable-image-mirroring | |
20effc67 | 534 | .. _bootstrap token: #bootstrap-peers |
7c673cae FG |
535 | .. _force resync command: #force-image-resync |
536 | .. _demote the image: #image-promotion-and-demotion | |
d2e6a577 | 537 | .. _create a Ceph user: ../../rados/operations/user-management#add-a-user |
9f95a23c | 538 | .. _mirror-snapshots: #create-image-mirror-snapshots |