]>
Commit | Line | Data |
---|---|---|
7c673cae FG |
1 | =============== |
2 | RBD Mirroring | |
3 | =============== | |
4 | ||
5 | .. index:: Ceph Block Device; mirroring | |
6 | ||
7 | RBD images can be asynchronously mirrored between two Ceph clusters. This | |
8 | capability uses the RBD journaling image feature to ensure crash-consistent | |
9 | replication between clusters. Mirroring is configured on a per-pool basis | |
10 | within peer clusters and can be configured to automatically mirror all | |
11 | images within a pool or only a specific subset of images. Mirroring is | |
12 | configured using the ``rbd`` command. The ``rbd-mirror`` daemon is responsible | |
13 | for pulling image updates from the remote, peer cluster and applying them to | |
14 | the image within the local cluster. | |
15 | ||
16 | .. note:: RBD mirroring requires the Ceph Jewel release or later. | |
17 | ||
18 | .. important:: To use RBD mirroring, you must have two Ceph clusters, each | |
19 | running the ``rbd-mirror`` daemon. | |
20 | ||
21 | Pool Configuration | |
22 | ================== | |
23 | ||
24 | The following procedures demonstrate how to perform the basic administrative | |
25 | tasks to configure mirroring using the ``rbd`` command. Mirroring is | |
26 | configured on a per-pool basis within the Ceph clusters. | |
27 | ||
28 | The pool configuration steps should be performed on both peer clusters. These | |
29 | procedures assume two clusters, named "local" and "remote", are accessible from | |
30 | a single host for clarity. | |
31 | ||
32 | See the `rbd`_ manpage for additional details of how to connect to different | |
33 | Ceph clusters. | |
34 | ||
35 | .. note:: The cluster name in the following examples corresponds to a Ceph | |
36 | configuration file of the same name (e.g. /etc/ceph/remote.conf). See the | |
37 | `ceph-conf`_ documentation for how to configure multiple clusters. | |
38 | ||
39 | Enable Mirroring | |
40 | ---------------- | |
41 | ||
42 | To enable mirroring on a pool with ``rbd``, specify the ``mirror pool enable`` | |
43 | command, the pool name, and the mirroring mode:: | |
44 | ||
45 | rbd mirror pool enable {pool-name} {mode} | |
46 | ||
47 | The mirroring mode can either be ``pool`` or ``image``: | |
48 | ||
49 | * **pool**: When configured in ``pool`` mode, all images in the pool with the | |
50 | journaling feature enabled are mirrored. | |
51 | * **image**: When configured in ``image`` mode, mirroring needs to be | |
52 | `explicitly enabled`_ on each image. | |
53 | ||
54 | For example:: | |
55 | ||
56 | rbd --cluster local mirror pool enable image-pool pool | |
57 | rbd --cluster remote mirror pool enable image-pool pool | |
58 | ||
59 | Disable Mirroring | |
60 | ----------------- | |
61 | ||
62 | To disable mirroring on a pool with ``rbd``, specify the ``mirror pool disable`` | |
63 | command and the pool name:: | |
64 | ||
65 | rbd mirror pool disable {pool-name} | |
66 | ||
67 | When mirroring is disabled on a pool in this way, mirroring will also be | |
68 | disabled on any images (within the pool) for which mirroring was enabled | |
69 | explicitly. | |
70 | ||
71 | For example:: | |
72 | ||
73 | rbd --cluster local mirror pool disable image-pool | |
74 | rbd --cluster remote mirror pool disable image-pool | |
75 | ||
76 | Add Cluster Peer | |
77 | ---------------- | |
78 | ||
79 | In order for the ``rbd-mirror`` daemon to discover its peer cluster, the peer | |
80 | needs to be registered to the pool. To add a mirroring peer Ceph cluster with | |
81 | ``rbd``, specify the ``mirror pool peer add`` command, the pool name, and a | |
82 | cluster specification:: | |
83 | ||
84 | rbd mirror pool peer add {pool-name} {client-name}@{cluster-name} | |
85 | ||
86 | For example:: | |
87 | ||
88 | rbd --cluster local mirror pool peer add image-pool client.remote@remote | |
89 | rbd --cluster remote mirror pool peer add image-pool client.local@local | |
90 | ||
91 | Remove Cluster Peer | |
92 | ------------------- | |
93 | ||
94 | To remove a mirroring peer Ceph cluster with ``rbd``, specify the | |
95 | ``mirror pool peer remove`` command, the pool name, and the peer UUID | |
96 | (available from the ``rbd mirror pool info`` command):: | |
97 | ||
98 | rbd mirror pool peer remove {pool-name} {peer-uuid} | |
99 | ||
100 | For example:: | |
101 | ||
102 | rbd --cluster local mirror pool peer remove image-pool 55672766-c02b-4729-8567-f13a66893445 | |
103 | rbd --cluster remote mirror pool peer remove image-pool 60c0e299-b38f-4234-91f6-eed0a367be08 | |
104 | ||
b32b8144 FG |
105 | Data Pools |
106 | ---------- | |
107 | ||
108 | When creating images in the destination cluster, ``rbd-mirror`` selects a data | |
109 | pool as follows: | |
110 | ||
111 | #. If the destination cluster has a default data pool configured (with the | |
112 | ``rbd_default_data_pool`` configuration option), it will be used. | |
113 | #. Otherwise, if the source image uses a separate data pool, and a pool with the | |
114 | same name exists on the destination cluster, that pool will be used. | |
115 | #. If neither of the above is true, no data pool will be set. | |
116 | ||
7c673cae FG |
117 | Image Configuration |
118 | =================== | |
119 | ||
120 | Unlike pool configuration, image configuration only needs to be performed against | |
121 | a single mirroring peer Ceph cluster. | |
122 | ||
123 | Mirrored RBD images are designated as either primary or non-primary. This is a | |
124 | property of the image and not the pool. Images that are designated as | |
125 | non-primary cannot be modified. | |
126 | ||
127 | Images are automatically promoted to primary when mirroring is first enabled on | |
128 | an image (either implicitly if the pool mirror mode was **pool** and the image | |
129 | has the journaling image feature enabled, or `explicitly enabled`_ by the | |
130 | ``rbd`` command). | |
131 | ||
132 | Enable Image Journaling Support | |
133 | ------------------------------- | |
134 | ||
135 | RBD mirroring uses the RBD journaling feature to ensure that the replicated | |
136 | image always remains crash-consistent. Before an image can be mirrored to | |
137 | a peer cluster, the journaling feature must be enabled. The feature can be | |
138 | enabled at image creation time by providing the | |
139 | ``--image-feature exclusive-lock,journaling`` option to the ``rbd`` command. | |
140 | ||
141 | Alternatively, the journaling feature can be dynamically enabled on | |
142 | pre-existing RBD images. To enable journaling with ``rbd``, specify | |
143 | the ``feature enable`` command, the pool and image name, and the feature name:: | |
144 | ||
145 | rbd feature enable {pool-name}/{image-name} {feature-name} | |
146 | ||
147 | For example:: | |
148 | ||
149 | rbd --cluster local feature enable image-pool/image-1 journaling | |
150 | ||
151 | .. note:: The journaling feature is dependent on the exclusive-lock feature. If | |
152 | the exclusive-lock feature is not already enabled, it should be enabled prior | |
153 | to enabling the journaling feature. | |
154 | ||
155 | .. tip:: You can enable journaling on all new images by default by adding | |
156 | ``rbd default features = 125`` to your Ceph configuration file. | |
157 | ||
158 | Enable Image Mirroring | |
159 | ---------------------- | |
160 | ||
161 | If the mirroring is configured in ``image`` mode for the image's pool, then it | |
162 | is necessary to explicitly enable mirroring for each image within the pool. | |
163 | To enable mirroring for a specific image with ``rbd``, specify the | |
164 | ``mirror image enable`` command along with the pool and image name:: | |
165 | ||
166 | rbd mirror image enable {pool-name}/{image-name} | |
167 | ||
168 | For example:: | |
169 | ||
170 | rbd --cluster local mirror image enable image-pool/image-1 | |
171 | ||
172 | Disable Image Mirroring | |
173 | ----------------------- | |
174 | ||
175 | To disable mirroring for a specific image with ``rbd``, specify the | |
176 | ``mirror image disable`` command along with the pool and image name:: | |
177 | ||
178 | rbd mirror image disable {pool-name}/{image-name} | |
179 | ||
180 | For example:: | |
181 | ||
182 | rbd --cluster local mirror image disable image-pool/image-1 | |
183 | ||
184 | Image Promotion and Demotion | |
185 | ---------------------------- | |
186 | ||
187 | In a failover scenario where the primary designation needs to be moved to the | |
188 | image in the peer Ceph cluster, access to the primary image should be stopped | |
189 | (e.g. power down the VM or remove the associated drive from a VM), demote the | |
190 | current primary image, promote the new primary image, and resume access to the | |
191 | image on the alternate cluster. | |
192 | ||
193 | .. note:: RBD only provides the necessary tools to facilitate an orderly | |
194 | failover of an image. An external mechanism is required to coordinate the | |
195 | full failover process (e.g. closing the image before demotion). | |
196 | ||
197 | To demote a specific image to non-primary with ``rbd``, specify the | |
198 | ``mirror image demote`` command along with the pool and image name:: | |
199 | ||
200 | rbd mirror image demote {pool-name}/{image-name} | |
201 | ||
202 | For example:: | |
203 | ||
204 | rbd --cluster local mirror image demote image-pool/image-1 | |
205 | ||
206 | To demote all primary images within a pool to non-primary with ``rbd``, specify | |
207 | the ``mirror pool demote`` command along with the pool name:: | |
208 | ||
209 | rbd mirror pool demote {pool-name} | |
210 | ||
211 | For example:: | |
212 | ||
213 | rbd --cluster local mirror pool demote image-pool | |
214 | ||
215 | To promote a specific image to primary with ``rbd``, specify the | |
216 | ``mirror image promote`` command along with the pool and image name:: | |
217 | ||
218 | rbd mirror image promote [--force] {pool-name}/{image-name} | |
219 | ||
220 | For example:: | |
221 | ||
222 | rbd --cluster remote mirror image promote image-pool/image-1 | |
223 | ||
224 | To promote all non-primary images within a pool to primary with ``rbd``, specify | |
225 | the ``mirror pool promote`` command along with the pool name:: | |
226 | ||
227 | rbd mirror pool promote [--force] {pool-name} | |
228 | ||
229 | For example:: | |
230 | ||
231 | rbd --cluster local mirror pool promote image-pool | |
232 | ||
233 | .. tip:: Since the primary / non-primary status is per-image, it is possible to | |
234 | have two clusters split the IO load and stage failover / failback. | |
235 | ||
236 | .. note:: Promotion can be forced using the ``--force`` option. Forced | |
237 | promotion is needed when the demotion cannot be propagated to the peer | |
238 | Ceph cluster (e.g. Ceph cluster failure, communication outage). This will | |
239 | result in a split-brain scenario between the two peers and the image will no | |
240 | longer be in-sync until a `force resync command`_ is issued. | |
241 | ||
242 | Force Image Resync | |
243 | ------------------ | |
244 | ||
245 | If a split-brain event is detected by the ``rbd-mirror`` daemon, it will not | |
246 | attempt to mirror the affected image until corrected. To resume mirroring for an | |
247 | image, first `demote the image`_ determined to be out-of-date and then request a | |
248 | resync to the primary image. To request an image resync with ``rbd``, specify the | |
249 | ``mirror image resync`` command along with the pool and image name:: | |
250 | ||
251 | rbd mirror image resync {pool-name}/{image-name} | |
252 | ||
253 | For example:: | |
254 | ||
255 | rbd mirror image resync image-pool/image-1 | |
256 | ||
257 | .. note:: The ``rbd`` command only flags the image as requiring a resync. The | |
258 | local cluster's ``rbd-mirror`` daemon process is responsible for performing | |
259 | the resync asynchronously. | |
260 | ||
261 | Mirror Status | |
262 | ============= | |
263 | ||
264 | The peer cluster replication status is stored for every primary mirrored image. | |
265 | This status can be retrieved using the ``mirror image status`` and | |
266 | ``mirror pool status`` commands. | |
267 | ||
268 | To request the mirror image status with ``rbd``, specify the | |
269 | ``mirror image status`` command along with the pool and image name:: | |
270 | ||
271 | rbd mirror image status {pool-name}/{image-name} | |
272 | ||
273 | For example:: | |
274 | ||
275 | rbd mirror image status image-pool/image-1 | |
276 | ||
277 | To request the mirror pool summary status with ``rbd``, specify the | |
278 | ``mirror pool status`` command along with the pool name:: | |
279 | ||
280 | rbd mirror pool status {pool-name} | |
281 | ||
282 | For example:: | |
283 | ||
284 | rbd mirror pool status image-pool | |
285 | ||
286 | .. note:: Adding ``--verbose`` option to the ``mirror pool status`` command will | |
287 | additionally output status details for every mirroring image in the pool. | |
288 | ||
289 | rbd-mirror Daemon | |
290 | ================= | |
291 | ||
292 | The two ``rbd-mirror`` daemons are responsible for watching image journals on the | |
293 | remote, peer cluster and replaying the journal events against the local | |
294 | cluster. The RBD image journaling feature records all modifications to the | |
295 | image in the order they occur. This ensures that a crash-consistent mirror of | |
296 | the remote image is available locally. | |
297 | ||
298 | The ``rbd-mirror`` daemon is available within the optional ``rbd-mirror`` | |
299 | distribution package. | |
300 | ||
301 | .. important:: Each ``rbd-mirror`` daemon requires the ability to connect | |
302 | to both clusters simultaneously. | |
d2e6a577 FG |
303 | .. warning:: Pre-Luminous releases: only run a single ``rbd-mirror`` daemon per |
304 | Ceph cluster. | |
305 | ||
306 | Each ``rbd-mirror`` daemon should use a unique Ceph user ID. To | |
307 | `create a Ceph user`_, with ``ceph`` specify the ``auth get-or-create`` | |
308 | command, user name, monitor caps, and OSD caps:: | |
309 | ||
310 | ceph auth get-or-create client.rbd-mirror.{unique id} mon 'profile rbd' osd 'profile rbd' | |
311 | ||
312 | The ``rbd-mirror`` daemon can be managed by ``systemd`` by specifying the user | |
313 | ID as the daemon instance:: | |
314 | ||
315 | systemctl enable ceph-rbd-mirror@rbd-mirror.{unique id} | |
7c673cae FG |
316 | |
317 | .. _rbd: ../../man/8/rbd | |
318 | .. _ceph-conf: ../../rados/configuration/ceph-conf/#running-multiple-clusters | |
319 | .. _explicitly enabled: #enable-image-mirroring | |
320 | .. _force resync command: #force-image-resync | |
321 | .. _demote the image: #image-promotion-and-demotion | |
d2e6a577 | 322 | .. _create a Ceph user: ../../rados/operations/user-management#add-a-user |
7c673cae | 323 |