ceph/doc/cephfs/cephfs-mirroring.rst

   1 .. _cephfs-mirroring:
   2
   3 =========================
   4 CephFS Snapshot Mirroring
   5 =========================
   6
   7 CephFS supports asynchronous replication of snapshots to a remote CephFS file system via
   8 `cephfs-mirror` tool. Snapshots are synchronized by mirroring snapshot data followed by
   9 creating a snapshot with the same name (for a given directory on the remote file system) as
  10 the snapshot being synchronized.
  11
  12 Requirements
  13 ------------
  14
  15 The primary (local) and secondary (remote) Ceph clusters version should be Pacific or later.
  16
  17 Creating Users
  18 --------------
  19
  20 Start by creating a user (on the primary/local cluster) for the mirror daemon. This user
  21 requires write capability on the metadata pool to create RADOS objects (index objects)
  22 for watch/notify operation and read capability on the data pool(s)::
  23
  24   $ ceph auth get-or-create client.mirror mon 'profile cephfs-mirror' mds 'allow r' osd 'allow rw tag cephfs metadata=*, allow r tag cephfs data=*' mgr 'allow r'
  25
  26 Create a user for each file system peer (on the secondary/remote cluster). This user needs
  27 to have full capabilities on the MDS (to take snapshots) and the OSDs::
  28
  29   $ ceph fs authorize <fs_name> client.mirror_remote / rwps
  30
  31 This user should be used (as part of peer specification) when adding a peer.
  32
  33 Starting Mirror Daemon
  34 ----------------------
  35
  36 Mirror daemon should be spawned using `systemctl(1)` unit files::
  37
  38   $ systemctl enable cephfs-mirror@mirror
  39   $ systemctl start cephfs-mirror@mirror
  40
  41 `cephfs-mirror` daemon can be run in foreground using::
  42
  43   $ cephfs-mirror --id mirror --cluster site-a -f
  44
  45 .. note:: User used here is `mirror` created in the `Creating Users` section.
  46
  47 Interface
  48 ---------
  49
  50 `Mirroring` module (manager plugin) provides interfaces for managing directory snapshot
  51 mirroring. Manager interfaces are (mostly) wrappers around monitor commands for managing
  52 file system mirroring and is the recommended control interface.
  53
  54 Mirroring Module
  55 ----------------
  56
  57 The mirroring module is responsible for assigning directories to mirror daemons for
  58 synchronization. Multiple mirror daemons can be spawned to achieve concurrency in
  59 directory snapshot synchronization. When mirror daemons are spawned (or terminated)
  60 , the mirroring module discovers the modified set of mirror daemons and rebalances
  61 the directory assignment amongst the new set thus providing high-availability.
  62
  63 .. note:: Multiple mirror daemons is currently untested. Only a single mirror daemon
  64           is recommended.
  65
  66 Mirroring module is disabled by default. To enable mirroring use::
  67
  68   $ ceph mgr module enable mirroring
  69
  70 Mirroring module provides a family of commands to control mirroring of directory
  71 snapshots. To add or remove directories, mirroring needs to be enabled for a given
  72 file system. To enable mirroring use::
  73
  74   $ ceph fs snapshot mirror enable <fs_name>
  75
  76 .. note:: Mirroring module commands use `fs snapshot mirror` prefix as compared to
  77           the monitor commands which `fs mirror` prefix. Make sure to use module
  78           commands.
  79
  80 To disable mirroring, use::
  81
  82   $ ceph fs snapshot mirror disable <fs_name>
  83
  84 Once mirroring is enabled, add a peer to which directory snapshots are to be mirrored.
  85 Peers follow `<client>@<cluster>` specification and get assigned a unique-id (UUID)
  86 when added. See `Creating Users` section on how to create Ceph users for mirroring.
  87
  88 To add a peer use::
  89
  90   $ ceph fs snapshot mirror peer_add <fs_name> <remote_cluster_spec> [<remote_fs_name>] [<remote_mon_host>] [<cephx_key>]
  91
  92 `<remote_fs_name>` is optional, and defaults to `<fs_name>` (on the remote cluster).
  93
  94 This requires the remote cluster ceph configuration and user keyring to be available in
  95 the primary cluster. See `Bootstrap Peers` section to avoid this. `peer_add` additionally
  96 supports passing the remote cluster monitor address and the user key. However, bootstrapping
  97 a peer is the recommended way to add a peer.
  98
  99 .. note:: Only a single peer is supported right now.
 100
 101 To remove a peer use::
 102
 103   $ ceph fs snapshot mirror peer_remove <fs_name> <peer_uuid>
 104
 105 To list file system mirror peers use::
 106
 107   $ ceph fs snapshot mirror peer_list <fs_name>
 108
 109 To configure a directory for mirroring, use::
 110
 111   $ ceph fs snapshot mirror add <fs_name> <path>
 112
 113 To stop a mirroring directory snapshots use::
 114
 115   $ ceph fs snapshot mirror remove <fs_name> <path>
 116
 117 Only absolute directory paths are allowed. Also, paths are normalized by the mirroring
 118 module, therefore, `/a/b/../b` is equivalent to `/a/b`.
 119
 120   $ mkdir -p /d0/d1/d2
 121   $ ceph fs snapshot mirror add cephfs /d0/d1/d2
 122   {}
 123   $ ceph fs snapshot mirror add cephfs /d0/d1/../d1/d2
 124   Error EEXIST: directory /d0/d1/d2 is already tracked
 125
 126 Once a directory is added for mirroring, its subdirectory or ancestor directories are
 127 disallowed to be added for mirroring::
 128
 129   $ ceph fs snapshot mirror add cephfs /d0/d1
 130   Error EINVAL: /d0/d1 is a ancestor of tracked path /d0/d1/d2
 131   $ ceph fs snapshot mirror add cephfs /d0/d1/d2/d3
 132   Error EINVAL: /d0/d1/d2/d3 is a subtree of tracked path /d0/d1/d2
 133
 134 Commands to check directory mapping (to mirror daemons) and directory distribution are
 135 detailed in `Mirroring Status` section.
 136
 137 Bootstrap Peers
 138 ---------------
 139
 140 Adding a peer (via `peer_add`) requires the peer cluster configuration and user keyring
 141 to be available in the primary cluster (manager host and hosts running the mirror daemon).
 142 This can be avoided by bootstrapping and importing a peer token. Peer bootstrap involves
 143 creating a bootstrap token on the peer cluster via::
 144
 145   $ ceph fs snapshot mirror peer_bootstrap create <fs_name> <client_entity> <site-name>
 146
 147 e.g.::
 148
 149   $ ceph fs snapshot mirror peer_bootstrap create backup_fs client.mirror_remote site-remote
 150   {"token": "eyJmc2lkIjogIjBkZjE3MjE3LWRmY2QtNDAzMC05MDc5LTM2Nzk4NTVkNDJlZiIsICJmaWxlc3lzdGVtIjogImJhY2t1cF9mcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcGVlcl9ib290c3RyYXAiLCAic2l0ZV9uYW1lIjogInNpdGUtcmVtb3RlIiwgImtleSI6ICJBUUFhcDBCZ0xtRmpOeEFBVnNyZXozai9YYUV0T2UrbUJEZlJDZz09IiwgIm1vbl9ob3N0IjogIlt2MjoxOTIuMTY4LjAuNTo0MDkxOCx2MToxOTIuMTY4LjAuNTo0MDkxOV0ifQ=="}
 151
 152 `site-name` refers to a user-defined string to identify the remote filesystem. In context
 153 of `peer_add` interface, `site-name` is the passed in `cluster` name from `remote_cluster_spec`.
 154
 155 Import the bootstrap token in the primary cluster via::
 156
 157   $ ceph fs snapshot mirror peer_bootstrap import <fs_name> <token>
 158
 159 e.g.::
 160
 161   $ ceph fs snapshot mirror peer_bootstrap import cephfs eyJmc2lkIjogIjBkZjE3MjE3LWRmY2QtNDAzMC05MDc5LTM2Nzk4NTVkNDJlZiIsICJmaWxlc3lzdGVtIjogImJhY2t1cF9mcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcGVlcl9ib290c3RyYXAiLCAic2l0ZV9uYW1lIjogInNpdGUtcmVtb3RlIiwgImtleSI6ICJBUUFhcDBCZ0xtRmpOeEFBVnNyZXozai9YYUV0T2UrbUJEZlJDZz09IiwgIm1vbl9ob3N0IjogIlt2MjoxOTIuMTY4LjAuNTo0MDkxOCx2MToxOTIuMTY4LjAuNTo0MDkxOV0ifQ==
 162
 163 Mirroring Status
 164 ----------------
 165
 166 CephFS mirroring module provides `mirror daemon status` interface to check mirror daemon status::
 167
 168   $ ceph fs snapshot mirror daemon status
 169   [
 170     {
 171       "daemon_id": 284167,
 172       "filesystems": [
 173         {
 174           "filesystem_id": 1,
 175           "name": "a",
 176           "directory_count": 1,
 177           "peers": [
 178             {
 179               "uuid": "02117353-8cd1-44db-976b-eb20609aa160",
 180               "remote": {
 181                 "client_name": "client.mirror_remote",
 182                 "cluster_name": "ceph",
 183                 "fs_name": "backup_fs"
 184               },
 185               "stats": {
 186                 "failure_count": 1,
 187                 "recovery_count": 0
 188               }
 189             }
 190           ]
 191         }
 192       ]
 193     }
 194   ]
 195
 196 An entry per mirror daemon instance is displayed along with information such as configured
 197 peers and basic stats. For more detailed stats, use the admin socket interface as detailed
 198 below.
 199
 200 CephFS mirror daemons provide admin socket commands for querying mirror status. To check
 201 available commands for mirror status use::
 202
 203   $ ceph --admin-daemon /path/to/mirror/daemon/admin/socket help
 204   {
 205       ....
 206       ....
 207       "fs mirror status cephfs@360": "get filesystem mirror status",
 208       ....
 209       ....
 210   }
 211
 212 Commands with `fs mirror status` prefix provide mirror status for mirror enabled
 213 file systems. Note that `cephfs@360` is of format `filesystem-name@filesystem-id`.
 214 This format is required since mirror daemons get asynchronously notified regarding
 215 file system mirror status (A file system can be deleted and recreated with the same
 216 name).
 217
 218 Right now, the command provides minimal information regarding mirror status::
 219
 220   $ ceph --admin-daemon /var/run/ceph/cephfs-mirror.asok fs mirror status cephfs@360
 221   {
 222     "rados_inst": "192.168.0.5:0/1476644347",
 223     "peers": {
 224         "a2dc7784-e7a1-4723-b103-03ee8d8768f8": {
 225             "remote": {
 226                 "client_name": "client.mirror_remote",
 227                 "cluster_name": "site-a",
 228                 "fs_name": "backup_fs"
 229             }
 230         }
 231     },
 232     "snap_dirs": {
 233         "dir_count": 1
 234     }
 235   }
 236
 237 `Peers` section in the command output above shows the peer information such as unique
 238 peer-id (UUID) and specification. The peer-id is required to remove an existing peer
 239 as mentioned in the `Mirror Module and Interface` section.
 240
 241 Command with `fs mirror peer status` prefix provide peer synchronization status. This
 242 command is of format `filesystem-name@filesystem-id peer-uuid`::
 243
 244   $ ceph --admin-daemon /var/run/ceph/cephfs-mirror.asok fs mirror peer status cephfs@360 a2dc7784-e7a1-4723-b103-03ee8d8768f8
 245   {
 246     "/d0": {
 247         "state": "idle",
 248         "last_synced_snap": {
 249             "id": 120,
 250             "name": "snap1",
 251             "sync_duration": 0.079997898999999997,
 252             "sync_time_stamp": "274900.558797s"
 253         },
 254         "snaps_synced": 2,
 255         "snaps_deleted": 0,
 256         "snaps_renamed": 0
 257     }
 258   }
 259
 260 Synchronization stats such as `snaps_synced`, `snaps_deleted` and `snaps_renamed` are reset
 261 on daemon restart and/or when a directory is reassigned to another mirror daemon (when
 262 multiple mirror daemons are deployed).
 263
 264 A directory can be in one of the following states::
 265
 266   - `idle`: The directory is currently not being synchronized
 267   - `syncing`: The directory is currently being synchronized
 268   - `failed`: The directory has hit upper limit of consecutive failures
 269
 270 When a directory hits a configured number of consecutive synchronization failures, the
 271 mirror daemon marks it as `failed`. Synchronization for these directories are retried.
 272 By default, the number of consecutive failures before a directory is marked as failed
 273 is controlled by `cephfs_mirror_max_consecutive_failures_per_directory` configuration
 274 option (default: 10) and the retry interval for failed directories is controlled via
 275 `cephfs_mirror_retry_failed_directories_interval` configuration option (default: 60s).
 276
 277 E.g., adding a regular file for synchronization would result in failed status::
 278
 279   $ ceph fs snapshot mirror add cephfs /f0
 280   $ ceph --admin-daemon /var/run/ceph/cephfs-mirror.asok fs mirror peer status cephfs@360 a2dc7784-e7a1-4723-b103-03ee8d8768f8
 281   {
 282     "/d0": {
 283         "state": "idle",
 284         "last_synced_snap": {
 285             "id": 120,
 286             "name": "snap1",
 287             "sync_duration": 0.079997898999999997,
 288             "sync_time_stamp": "274900.558797s"
 289         },
 290         "snaps_synced": 2,
 291         "snaps_deleted": 0,
 292         "snaps_renamed": 0
 293     },
 294     "/f0": {
 295         "state": "failed",
 296         "snaps_synced": 0,
 297         "snaps_deleted": 0,
 298         "snaps_renamed": 0
 299     }
 300   }
 301
 302 This allows a user to add a non-existent directory for synchronization. The mirror daemon
 303 would mark the directory as failed and retry (less frequently). When the directory comes
 304 to existence, the mirror daemons would unmark the failed state upon successful snapshot
 305 synchronization.
 306
 307 When mirroring is disabled, the respective `fs mirror status` command for the file system
 308 will not show up in command help.
 309
 310 Configuration Options
 311 ---------------------
 312
 313 .. confval:: cephfs_mirror_max_concurrent_directory_syncs
 314 .. confval:: cephfs_mirror_action_update_interval
 315 .. confval:: cephfs_mirror_restart_mirror_on_blocklist_interval
 316 .. confval:: cephfs_mirror_max_snapshot_sync_per_cycle
 317 .. confval:: cephfs_mirror_directory_scan_interval
 318 .. confval:: cephfs_mirror_max_consecutive_failures_per_directory
 319 .. confval:: cephfs_mirror_retry_failed_directories_interval
 320 .. confval:: cephfs_mirror_restart_mirror_on_failure_interval
 321 .. confval:: cephfs_mirror_mount_timeout
 322
 323 Re-adding Peers
 324 ---------------
 325
 326 When re-adding (reassigning) a peer to a file system in another cluster, ensure that
 327 all mirror daemons have stopped synchronization to the peer. This can be checked
 328 via `fs mirror status` admin socket command (the `Peer UUID` should not show up
 329 in the command output). Also, it is recommended to purge synchronized directories
 330 from the peer  before re-adding it to another file system (especially those directories
 331 which might exist in the new primary file system). This is not required if re-adding
 332 a peer to the same primary file system it was earlier synchronized from.