]>
Commit | Line | Data |
---|---|---|
b3b6e05e TL |
1 | .. _cephfs-mirroring: |
2 | ||
3 | ========================= | |
4 | CephFS Snapshot Mirroring | |
5 | ========================= | |
6 | ||
7 | CephFS supports asynchronous replication of snapshots to a remote CephFS file system via | |
8 | `cephfs-mirror` tool. Snapshots are synchronized by mirroring snapshot data followed by | |
9 | creating a snapshot with the same name (for a given directory on the remote file system) as | |
10 | the snapshot being synchronized. | |
11 | ||
12 | Requirements | |
13 | ------------ | |
14 | ||
15 | The primary (local) and secondary (remote) Ceph clusters version should be Pacific or later. | |
16 | ||
17 | Creating Users | |
18 | -------------- | |
19 | ||
20 | Start by creating a user (on the primary/local cluster) for the mirror daemon. This user | |
21 | requires write capability on the metadata pool to create RADOS objects (index objects) | |
22 | for watch/notify operation and read capability on the data pool(s):: | |
23 | ||
24 | $ ceph auth get-or-create client.mirror mon 'profile cephfs-mirror' mds 'allow r' osd 'allow rw tag cephfs metadata=*, allow r tag cephfs data=*' mgr 'allow r' | |
25 | ||
26 | Create a user for each file system peer (on the secondary/remote cluster). This user needs | |
27 | to have full capabilities on the MDS (to take snapshots) and the OSDs:: | |
28 | ||
29 | $ ceph fs authorize <fs_name> client.mirror_remote / rwps | |
30 | ||
31 | This user should be used (as part of peer specification) when adding a peer. | |
32 | ||
33 | Starting Mirror Daemon | |
34 | ---------------------- | |
35 | ||
36 | Mirror daemon should be spawned using `systemctl(1)` unit files:: | |
37 | ||
38 | $ systemctl enable cephfs-mirror@mirror | |
39 | $ systemctl start cephfs-mirror@mirror | |
40 | ||
41 | `cephfs-mirror` daemon can be run in foreground using:: | |
42 | ||
43 | $ cephfs-mirror --id mirror --cluster site-a -f | |
44 | ||
45 | .. note:: User used here is `mirror` created in the `Creating Users` section. | |
46 | ||
47 | Interface | |
48 | --------- | |
49 | ||
50 | `Mirroring` module (manager plugin) provides interfaces for managing directory snapshot | |
51 | mirroring. Manager interfaces are (mostly) wrappers around monitor commands for managing | |
52 | file system mirroring and is the recommended control interface. | |
53 | ||
54 | Mirroring Module | |
55 | ---------------- | |
56 | ||
57 | The mirroring module is responsible for assigning directories to mirror daemons for | |
58 | synchronization. Multiple mirror daemons can be spawned to achieve concurrency in | |
59 | directory snapshot synchronization. When mirror daemons are spawned (or terminated) | |
60 | , the mirroring module discovers the modified set of mirror daemons and rebalances | |
61 | the directory assignment amongst the new set thus providing high-availability. | |
62 | ||
63 | .. note:: Multiple mirror daemons is currently untested. Only a single mirror daemon | |
64 | is recommended. | |
65 | ||
66 | Mirroring module is disabled by default. To enable mirroring use:: | |
67 | ||
68 | $ ceph mgr module enable mirroring | |
69 | ||
70 | Mirroring module provides a family of commands to control mirroring of directory | |
71 | snapshots. To add or remove directories, mirroring needs to be enabled for a given | |
72 | file system. To enable mirroring use:: | |
73 | ||
74 | $ ceph fs snapshot mirror enable <fs_name> | |
75 | ||
76 | .. note:: Mirroring module commands use `fs snapshot mirror` prefix as compared to | |
77 | the monitor commands which `fs mirror` prefix. Make sure to use module | |
78 | commands. | |
79 | ||
80 | To disable mirroring, use:: | |
81 | ||
82 | $ ceph fs snapshot mirror disable <fs_name> | |
83 | ||
84 | Once mirroring is enabled, add a peer to which directory snapshots are to be mirrored. | |
85 | Peers follow `<client>@<cluster>` specification and get assigned a unique-id (UUID) | |
86 | when added. See `Creating Users` section on how to create Ceph users for mirroring. | |
87 | ||
88 | To add a peer use:: | |
89 | ||
90 | $ ceph fs snapshot mirror peer_add <fs_name> <remote_cluster_spec> [<remote_fs_name>] [<remote_mon_host>] [<cephx_key>] | |
91 | ||
92 | `<remote_fs_name>` is optional, and defaults to `<fs_name>` (on the remote cluster). | |
93 | ||
94 | This requires the remote cluster ceph configuration and user keyring to be available in | |
95 | the primary cluster. See `Bootstrap Peers` section to avoid this. `peer_add` additionally | |
96 | supports passing the remote cluster monitor address and the user key. However, bootstrapping | |
97 | a peer is the recommended way to add a peer. | |
98 | ||
99 | .. note:: Only a single peer is supported right now. | |
100 | ||
101 | To remove a peer use:: | |
102 | ||
103 | $ ceph fs snapshot mirror peer_remove <fs_name> <peer_uuid> | |
104 | ||
105 | To list file system mirror peers use:: | |
106 | ||
107 | $ ceph fs snapshot mirror peer_list <fs_name> | |
108 | ||
109 | To configure a directory for mirroring, use:: | |
110 | ||
111 | $ ceph fs snapshot mirror add <fs_name> <path> | |
112 | ||
113 | To stop a mirroring directory snapshots use:: | |
114 | ||
115 | $ ceph fs snapshot mirror remove <fs_name> <path> | |
116 | ||
117 | Only absolute directory paths are allowed. Also, paths are normalized by the mirroring | |
20effc67 | 118 | module, therefore, `/a/b/../b` is equivalent to `/a/b`. |
b3b6e05e TL |
119 | |
120 | $ mkdir -p /d0/d1/d2 | |
121 | $ ceph fs snapshot mirror add cephfs /d0/d1/d2 | |
122 | {} | |
123 | $ ceph fs snapshot mirror add cephfs /d0/d1/../d1/d2 | |
124 | Error EEXIST: directory /d0/d1/d2 is already tracked | |
125 | ||
126 | Once a directory is added for mirroring, its subdirectory or ancestor directories are | |
20effc67 | 127 | disallowed to be added for mirroring:: |
b3b6e05e TL |
128 | |
129 | $ ceph fs snapshot mirror add cephfs /d0/d1 | |
130 | Error EINVAL: /d0/d1 is a ancestor of tracked path /d0/d1/d2 | |
131 | $ ceph fs snapshot mirror add cephfs /d0/d1/d2/d3 | |
132 | Error EINVAL: /d0/d1/d2/d3 is a subtree of tracked path /d0/d1/d2 | |
133 | ||
134 | Commands to check directory mapping (to mirror daemons) and directory distribution are | |
135 | detailed in `Mirroring Status` section. | |
136 | ||
137 | Bootstrap Peers | |
138 | --------------- | |
139 | ||
140 | Adding a peer (via `peer_add`) requires the peer cluster configuration and user keyring | |
141 | to be available in the primary cluster (manager host and hosts running the mirror daemon). | |
142 | This can be avoided by bootstrapping and importing a peer token. Peer bootstrap involves | |
143 | creating a bootstrap token on the peer cluster via:: | |
144 | ||
145 | $ ceph fs snapshot mirror peer_bootstrap create <fs_name> <client_entity> <site-name> | |
146 | ||
147 | e.g.:: | |
148 | ||
149 | $ ceph fs snapshot mirror peer_bootstrap create backup_fs client.mirror_remote site-remote | |
150 | {"token": "eyJmc2lkIjogIjBkZjE3MjE3LWRmY2QtNDAzMC05MDc5LTM2Nzk4NTVkNDJlZiIsICJmaWxlc3lzdGVtIjogImJhY2t1cF9mcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcGVlcl9ib290c3RyYXAiLCAic2l0ZV9uYW1lIjogInNpdGUtcmVtb3RlIiwgImtleSI6ICJBUUFhcDBCZ0xtRmpOeEFBVnNyZXozai9YYUV0T2UrbUJEZlJDZz09IiwgIm1vbl9ob3N0IjogIlt2MjoxOTIuMTY4LjAuNTo0MDkxOCx2MToxOTIuMTY4LjAuNTo0MDkxOV0ifQ=="} | |
151 | ||
152 | `site-name` refers to a user-defined string to identify the remote filesystem. In context | |
153 | of `peer_add` interface, `site-name` is the passed in `cluster` name from `remote_cluster_spec`. | |
154 | ||
155 | Import the bootstrap token in the primary cluster via:: | |
156 | ||
157 | $ ceph fs snapshot mirror peer_bootstrap import <fs_name> <token> | |
158 | ||
159 | e.g.:: | |
160 | ||
161 | $ ceph fs snapshot mirror peer_bootstrap import cephfs eyJmc2lkIjogIjBkZjE3MjE3LWRmY2QtNDAzMC05MDc5LTM2Nzk4NTVkNDJlZiIsICJmaWxlc3lzdGVtIjogImJhY2t1cF9mcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcGVlcl9ib290c3RyYXAiLCAic2l0ZV9uYW1lIjogInNpdGUtcmVtb3RlIiwgImtleSI6ICJBUUFhcDBCZ0xtRmpOeEFBVnNyZXozai9YYUV0T2UrbUJEZlJDZz09IiwgIm1vbl9ob3N0IjogIlt2MjoxOTIuMTY4LjAuNTo0MDkxOCx2MToxOTIuMTY4LjAuNTo0MDkxOV0ifQ== | |
162 | ||
163 | Mirroring Status | |
164 | ---------------- | |
165 | ||
166 | CephFS mirroring module provides `mirror daemon status` interface to check mirror daemon status:: | |
167 | ||
a4b75251 | 168 | $ ceph fs snapshot mirror daemon status |
b3b6e05e TL |
169 | [ |
170 | { | |
171 | "daemon_id": 284167, | |
172 | "filesystems": [ | |
173 | { | |
174 | "filesystem_id": 1, | |
175 | "name": "a", | |
176 | "directory_count": 1, | |
177 | "peers": [ | |
178 | { | |
179 | "uuid": "02117353-8cd1-44db-976b-eb20609aa160", | |
180 | "remote": { | |
181 | "client_name": "client.mirror_remote", | |
182 | "cluster_name": "ceph", | |
183 | "fs_name": "backup_fs" | |
184 | }, | |
185 | "stats": { | |
186 | "failure_count": 1, | |
187 | "recovery_count": 0 | |
188 | } | |
189 | } | |
190 | ] | |
191 | } | |
192 | ] | |
193 | } | |
194 | ] | |
195 | ||
196 | An entry per mirror daemon instance is displayed along with information such as configured | |
197 | peers and basic stats. For more detailed stats, use the admin socket interface as detailed | |
198 | below. | |
199 | ||
200 | CephFS mirror daemons provide admin socket commands for querying mirror status. To check | |
201 | available commands for mirror status use:: | |
202 | ||
203 | $ ceph --admin-daemon /path/to/mirror/daemon/admin/socket help | |
204 | { | |
205 | .... | |
206 | .... | |
207 | "fs mirror status cephfs@360": "get filesystem mirror status", | |
208 | .... | |
209 | .... | |
210 | } | |
211 | ||
212 | Commands with `fs mirror status` prefix provide mirror status for mirror enabled | |
213 | file systems. Note that `cephfs@360` is of format `filesystem-name@filesystem-id`. | |
214 | This format is required since mirror daemons get asynchronously notified regarding | |
215 | file system mirror status (A file system can be deleted and recreated with the same | |
216 | name). | |
217 | ||
218 | Right now, the command provides minimal information regarding mirror status:: | |
219 | ||
220 | $ ceph --admin-daemon /var/run/ceph/cephfs-mirror.asok fs mirror status cephfs@360 | |
221 | { | |
222 | "rados_inst": "192.168.0.5:0/1476644347", | |
223 | "peers": { | |
224 | "a2dc7784-e7a1-4723-b103-03ee8d8768f8": { | |
225 | "remote": { | |
226 | "client_name": "client.mirror_remote", | |
227 | "cluster_name": "site-a", | |
228 | "fs_name": "backup_fs" | |
229 | } | |
230 | } | |
231 | }, | |
232 | "snap_dirs": { | |
233 | "dir_count": 1 | |
234 | } | |
235 | } | |
236 | ||
237 | `Peers` section in the command output above shows the peer information such as unique | |
238 | peer-id (UUID) and specification. The peer-id is required to remove an existing peer | |
239 | as mentioned in the `Mirror Module and Interface` section. | |
240 | ||
241 | Command with `fs mirror peer status` prefix provide peer synchronization status. This | |
242 | command is of format `filesystem-name@filesystem-id peer-uuid`:: | |
243 | ||
244 | $ ceph --admin-daemon /var/run/ceph/cephfs-mirror.asok fs mirror peer status cephfs@360 a2dc7784-e7a1-4723-b103-03ee8d8768f8 | |
245 | { | |
246 | "/d0": { | |
247 | "state": "idle", | |
248 | "last_synced_snap": { | |
249 | "id": 120, | |
250 | "name": "snap1", | |
251 | "sync_duration": 0.079997898999999997, | |
252 | "sync_time_stamp": "274900.558797s" | |
253 | }, | |
254 | "snaps_synced": 2, | |
255 | "snaps_deleted": 0, | |
256 | "snaps_renamed": 0 | |
257 | } | |
258 | } | |
259 | ||
260 | Synchronization stats such as `snaps_synced`, `snaps_deleted` and `snaps_renamed` are reset | |
261 | on daemon restart and/or when a directory is reassigned to another mirror daemon (when | |
262 | multiple mirror daemons are deployed). | |
263 | ||
264 | A directory can be in one of the following states:: | |
265 | ||
266 | - `idle`: The directory is currently not being synchronized | |
267 | - `syncing`: The directory is currently being synchronized | |
268 | - `failed`: The directory has hit upper limit of consecutive failures | |
269 | ||
270 | When a directory hits a configured number of consecutive synchronization failures, the | |
271 | mirror daemon marks it as `failed`. Synchronization for these directories are retried. | |
272 | By default, the number of consecutive failures before a directory is marked as failed | |
273 | is controlled by `cephfs_mirror_max_consecutive_failures_per_directory` configuration | |
274 | option (default: 10) and the retry interval for failed directories is controlled via | |
275 | `cephfs_mirror_retry_failed_directories_interval` configuration option (default: 60s). | |
276 | ||
277 | E.g., adding a regular file for synchronization would result in failed status:: | |
278 | ||
279 | $ ceph fs snapshot mirror add cephfs /f0 | |
280 | $ ceph --admin-daemon /var/run/ceph/cephfs-mirror.asok fs mirror peer status cephfs@360 a2dc7784-e7a1-4723-b103-03ee8d8768f8 | |
281 | { | |
282 | "/d0": { | |
283 | "state": "idle", | |
284 | "last_synced_snap": { | |
285 | "id": 120, | |
286 | "name": "snap1", | |
287 | "sync_duration": 0.079997898999999997, | |
288 | "sync_time_stamp": "274900.558797s" | |
289 | }, | |
290 | "snaps_synced": 2, | |
291 | "snaps_deleted": 0, | |
292 | "snaps_renamed": 0 | |
293 | }, | |
294 | "/f0": { | |
295 | "state": "failed", | |
296 | "snaps_synced": 0, | |
297 | "snaps_deleted": 0, | |
298 | "snaps_renamed": 0 | |
299 | } | |
300 | } | |
301 | ||
302 | This allows a user to add a non-existent directory for synchronization. The mirror daemon | |
303 | would mark the directory as failed and retry (less frequently). When the directory comes | |
20effc67 | 304 | to existence, the mirror daemons would unmark the failed state upon successful snapshot |
b3b6e05e TL |
305 | synchronization. |
306 | ||
307 | When mirroring is disabled, the respective `fs mirror status` command for the file system | |
308 | will not show up in command help. | |
309 | ||
310 | Configuration Options | |
311 | --------------------- | |
312 | ||
20effc67 TL |
313 | .. confval:: cephfs_mirror_max_concurrent_directory_syncs |
314 | .. confval:: cephfs_mirror_action_update_interval | |
315 | .. confval:: cephfs_mirror_restart_mirror_on_blocklist_interval | |
316 | .. confval:: cephfs_mirror_max_snapshot_sync_per_cycle | |
317 | .. confval:: cephfs_mirror_directory_scan_interval | |
318 | .. confval:: cephfs_mirror_max_consecutive_failures_per_directory | |
319 | .. confval:: cephfs_mirror_retry_failed_directories_interval | |
320 | .. confval:: cephfs_mirror_restart_mirror_on_failure_interval | |
321 | .. confval:: cephfs_mirror_mount_timeout | |
b3b6e05e TL |
322 | |
323 | Re-adding Peers | |
324 | --------------- | |
325 | ||
326 | When re-adding (reassigning) a peer to a file system in another cluster, ensure that | |
327 | all mirror daemons have stopped synchronization to the peer. This can be checked | |
328 | via `fs mirror status` admin socket command (the `Peer UUID` should not show up | |
329 | in the command output). Also, it is recommended to purge synchronized directories | |
330 | from the peer before re-adding it to another file system (especially those directories | |
331 | which might exist in the new primary file system). This is not required if re-adding | |
332 | a peer to the same primary file system it was earlier synchronized from. |