ceph/doc/cephfs/cephfs-mirroring.rst

   1 .. _cephfs-mirroring:
   2
   3 =========================
   4 CephFS Snapshot Mirroring
   5 =========================
   6
   7 CephFS supports asynchronous replication of snapshots to a remote CephFS file system via
   8 the `cephfs-mirror` tool. Snapshots are synchronized by mirroring snapshot data followed by
   9 creating a remote snapshot with the same name (for a given directory on the remote file system) as
  10 the source snapshot.
  11
  12 Requirements
  13 ------------
  14
  15 The primary (local) and secondary (remote) Ceph clusters version should be Pacific or later.
  16
  17 .. _cephfs_mirroring_creating_users:
  18
  19 Creating Users
  20 --------------
  21
  22 Start by creating a Ceph user (on the primary/local cluster) for the `cephfs-mirror` daemon. This user
  23 requires write capability on the metadata pool to create RADOS objects (index objects)
  24 for watch/notify operation and read capability on the data pool(s)::
  25
  26   $ ceph auth get-or-create client.mirror mon 'profile cephfs-mirror' mds 'allow r' osd 'allow rw tag cephfs metadata=*, allow r tag cephfs data=*' mgr 'allow r'
  27
  28 Create a Ceph user for each file system peer (on the secondary/remote cluster). This user needs
  29 to have full capabilities on the MDS (to take snapshots) and the OSDs::
  30
  31   $ ceph fs authorize <fs_name> client.mirror_remote / rwps
  32
  33 This user will be supplied as part of the peer specification when adding a peer.
  34
  35 Starting Mirror Daemon
  36 ----------------------
  37
  38 The mirror daemon should be spawned using `systemctl(1)` unit files::
  39
  40   $ systemctl enable cephfs-mirror@mirror
  41   $ systemctl start cephfs-mirror@mirror
  42
  43 `cephfs-mirror` daemon can be run in foreground using::
  44
  45   $ cephfs-mirror --id mirror --cluster site-a -f
  46
  47 .. note:: The user specified here is `mirror`, the creation of which is
  48    described in the :ref:`Creating Users<cephfs_mirroring_creating_users>`
  49    section.
  50
  51 Multiple ``cephfs-mirror`` daemons may be deployed for concurrent
  52 synchronization and high availability. Mirror daemons share the synchronization
  53 load using a simple ``M/N`` policy, where ``M`` is the number of directories
  54 and ``N`` is the number of ``cephfs-mirror`` daemons.
  55
  56 When ``cephadm`` is used to manage a Ceph cluster, ``cephfs-mirror`` daemons can be
  57 deployed by running the following command:
  58
  59 .. prompt:: bash $
  60
  61    ceph orch apply cephfs-mirror
  62
  63 To deploy multiple mirror daemons, run a command of the following form:
  64
  65 .. prompt:: bash $
  66
  67    ceph orch apply cephfs-mirror --placement=<placement-spec>
  68
  69 For example, to deploy 3 `cephfs-mirror` daemons on different hosts, run a command of the following form:
  70
  71 .. prompt:: bash $
  72
  73   $ ceph orch apply cephfs-mirror --placement="3 host1,host2,host3"
  74
  75 Interface
  76 ---------
  77
  78 The `Mirroring` module (manager plugin) provides interfaces for managing
  79 directory snapshot mirroring. These are (mostly) wrappers around monitor
  80 commands for managing file system mirroring and is the recommended control
  81 interface.
  82
  83 Mirroring Module
  84 ----------------
  85
  86 The mirroring module is responsible for assigning directories to mirror daemons
  87 for synchronization. Multiple mirror daemons can be spawned to achieve
  88 concurrency in directory snapshot synchronization. When mirror daemons are
  89 spawned (or terminated), the mirroring module discovers the modified set of
  90 mirror daemons and rebalances directory assignments across the new set, thus
  91 providing high-availability.
  92
  93 .. note:: Deploying a single mirror daemon is recommended. Running multiple
  94    daemons is untested.
  95
  96 The mirroring module is disabled by default. To enable the mirroring module,
  97 run the following command:
  98
  99 .. prompt:: bash $
 100
 101    ceph mgr module enable mirroring
 102
 103 The mirroring module provides a family of commands that can be used to control
 104 the mirroring of directory snapshots. To add or remove directories, mirroring
 105 must be enabled for a given file system. To enable mirroring for a given file
 106 system, run a command of the following form:
 107
 108 .. prompt:: bash $
 109
 110    ceph fs snapshot mirror enable <fs_name>
 111
 112 .. note:: "Mirroring module" commands are prefixed with ``fs snapshot mirror``.
 113    This distinguishes them from "monitor commands", which are prefixed with ``fs
 114    mirror``. Be sure (in this context) to use module commands.
 115
 116 To disable mirroring for a given file system, run a command of the following form:
 117
 118 .. prompt:: bash $
 119
 120    ceph fs snapshot mirror disable <fs_name>
 121
 122 After mirroring is enabled, add a peer to which directory snapshots are to be
 123 mirrored. Peers are specified by the ``<client>@<cluster>`` format, which is
 124 referred to elsewhere in this document as the ``remote_cluster_spec``. Peers
 125 are assigned a unique-id (UUID) when added. See the :ref:`Creating
 126 Users<cephfs_mirroring_creating_users>` section for instructions that describe
 127 how to create Ceph users for mirroring.
 128
 129 To add a peer, run a command of the following form:
 130
 131 .. prompt:: bash $
 132
 133    ceph fs snapshot mirror peer_add <fs_name> <remote_cluster_spec> [<remote_fs_name>] [<remote_mon_host>] [<cephx_key>]
 134
 135 ``<remote_cluster_spec>`` is of the format ``client.<id>@<cluster_name>``.
 136
 137 ``<remote_fs_name>`` is optional, and defaults to `<fs_name>` (on the remote
 138 cluster).
 139
 140 For this command to succeed, the remote cluster's Ceph configuration and user
 141 keyring must be available in the primary cluster. For example, if a user named
 142 ``client_mirror`` is created on the remote cluster which has ``rwps``
 143 permissions for the remote file system named ``remote_fs`` (see `Creating
 144 Users`) and the remote cluster is named ``remote_ceph`` (that is, the remote
 145 cluster configuration file is named ``remote_ceph.conf`` on the primary
 146 cluster), run the following command to add the remote filesystem as a peer to
 147 the primary filesystem ``primary_fs``:
 148
 149 .. prompt:: bash $
 150
 151    ceph fs snapshot mirror peer_add primary_fs client.mirror_remote@remote_ceph remote_fs
 152
 153 To avoid having to maintain the remote cluster configuration file and remote
 154 ceph user keyring in the primary cluster, users can bootstrap a peer (which
 155 stores the relevant remote cluster details in the monitor config store on the
 156 primary cluster). See the :ref:`Bootstrap
 157 Peers<cephfs_mirroring_bootstrap_peers>` section.
 158
 159 The ``peer_add`` command supports passing the remote cluster monitor address
 160 and the user key. However, bootstrapping a peer is the recommended way to add a
 161 peer.
 162
 163 .. note:: Only a single peer is currently supported.
 164
 165 To remove a peer, run a command of the following form:
 166
 167 .. prompt:: bash $
 168
 169    ceph fs snapshot mirror peer_remove <fs_name> <peer_uuid>
 170
 171 To list file system mirror peers, run a command of the following form:
 172
 173 .. prompt:: bash $
 174
 175    ceph fs snapshot mirror peer_list <fs_name>
 176
 177 To configure a directory for mirroring, run a command of the following form:
 178
 179 .. prompt:: bash $
 180
 181    ceph fs snapshot mirror add <fs_name> <path>
 182
 183 To stop mirroring directory snapshots, run a command of the following form:
 184
 185 .. prompt:: bash $
 186
 187    ceph fs snapshot mirror remove <fs_name> <path>
 188
 189 Only absolute directory paths are allowed.
 190
 191 Paths are normalized by the mirroring module. This means that ``/a/b/../b`` is
 192 equivalent to ``/a/b``. Paths always start from the CephFS file-system root and
 193 not from the host system mount point.
 194
 195 For example::
 196
 197   $ mkdir -p /d0/d1/d2
 198   $ ceph fs snapshot mirror add cephfs /d0/d1/d2
 199   {}
 200   $ ceph fs snapshot mirror add cephfs /d0/d1/../d1/d2
 201   Error EEXIST: directory /d0/d1/d2 is already tracked
 202
 203 After a directory is added for mirroring, the additional mirroring of
 204 subdirectories or ancestor directories is disallowed::
 205
 206   $ ceph fs snapshot mirror add cephfs /d0/d1
 207   Error EINVAL: /d0/d1 is a ancestor of tracked path /d0/d1/d2
 208   $ ceph fs snapshot mirror add cephfs /d0/d1/d2/d3
 209   Error EINVAL: /d0/d1/d2/d3 is a subtree of tracked path /d0/d1/d2
 210
 211 The :ref:`Mirroring Status<cephfs_mirroring_mirroring_status>` section contains
 212 information about the commands for checking the directory mapping (to mirror
 213 daemons) and for checking the directory distribution.
 214
 215 .. _cephfs_mirroring_bootstrap_peers:
 216
 217 Bootstrap Peers
 218 ---------------
 219
 220 Adding a peer (via `peer_add`) requires the peer cluster configuration and user keyring
 221 to be available in the primary cluster (manager host and hosts running the mirror daemon).
 222 This can be avoided by bootstrapping and importing a peer token. Peer bootstrap involves
 223 creating a bootstrap token on the peer cluster via::
 224
 225   $ ceph fs snapshot mirror peer_bootstrap create <fs_name> <client_entity> <site-name>
 226
 227 e.g.::
 228
 229   $ ceph fs snapshot mirror peer_bootstrap create backup_fs client.mirror_remote site-remote
 230   {"token": "eyJmc2lkIjogIjBkZjE3MjE3LWRmY2QtNDAzMC05MDc5LTM2Nzk4NTVkNDJlZiIsICJmaWxlc3lzdGVtIjogImJhY2t1cF9mcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcGVlcl9ib290c3RyYXAiLCAic2l0ZV9uYW1lIjogInNpdGUtcmVtb3RlIiwgImtleSI6ICJBUUFhcDBCZ0xtRmpOeEFBVnNyZXozai9YYUV0T2UrbUJEZlJDZz09IiwgIm1vbl9ob3N0IjogIlt2MjoxOTIuMTY4LjAuNTo0MDkxOCx2MToxOTIuMTY4LjAuNTo0MDkxOV0ifQ=="}
 231
 232 `site-name` refers to a user-defined string to identify the remote filesystem. In context
 233 of `peer_add` interface, `site-name` is the passed in `cluster` name from `remote_cluster_spec`.
 234
 235 Import the bootstrap token in the primary cluster via::
 236
 237   $ ceph fs snapshot mirror peer_bootstrap import <fs_name> <token>
 238
 239 e.g.::
 240
 241   $ ceph fs snapshot mirror peer_bootstrap import cephfs eyJmc2lkIjogIjBkZjE3MjE3LWRmY2QtNDAzMC05MDc5LTM2Nzk4NTVkNDJlZiIsICJmaWxlc3lzdGVtIjogImJhY2t1cF9mcyIsICJ1c2VyIjogImNsaWVudC5taXJyb3JfcGVlcl9ib290c3RyYXAiLCAic2l0ZV9uYW1lIjogInNpdGUtcmVtb3RlIiwgImtleSI6ICJBUUFhcDBCZ0xtRmpOeEFBVnNyZXozai9YYUV0T2UrbUJEZlJDZz09IiwgIm1vbl9ob3N0IjogIlt2MjoxOTIuMTY4LjAuNTo0MDkxOCx2MToxOTIuMTY4LjAuNTo0MDkxOV0ifQ==
 242
 243
 244 .. _cephfs_mirroring_mirroring_status:
 245
 246 Mirroring Status
 247 ----------------
 248
 249 CephFS mirroring module provides `mirror daemon status` interface to check mirror daemon status::
 250
 251   $ ceph fs snapshot mirror daemon status
 252   [
 253     {
 254       "daemon_id": 284167,
 255       "filesystems": [
 256         {
 257           "filesystem_id": 1,
 258           "name": "a",
 259           "directory_count": 1,
 260           "peers": [
 261             {
 262               "uuid": "02117353-8cd1-44db-976b-eb20609aa160",
 263               "remote": {
 264                 "client_name": "client.mirror_remote",
 265                 "cluster_name": "ceph",
 266                 "fs_name": "backup_fs"
 267               },
 268               "stats": {
 269                 "failure_count": 1,
 270                 "recovery_count": 0
 271               }
 272             }
 273           ]
 274         }
 275       ]
 276     }
 277   ]
 278
 279 An entry per mirror daemon instance is displayed along with information such as configured
 280 peers and basic stats. For more detailed stats, use the admin socket interface as detailed
 281 below.
 282
 283 CephFS mirror daemons provide admin socket commands for querying mirror status. To check
 284 available commands for mirror status use::
 285
 286   $ ceph --admin-daemon /path/to/mirror/daemon/admin/socket help
 287   {
 288       ....
 289       ....
 290       "fs mirror status cephfs@360": "get filesystem mirror status",
 291       ....
 292       ....
 293   }
 294
 295 Commands prefixed with`fs mirror status` provide mirror status for mirror enabled
 296 file systems. Note that `cephfs@360` is of format `filesystem-name@filesystem-id`.
 297 This format is required since mirror daemons get asynchronously notified regarding
 298 file system mirror status (A file system can be deleted and recreated with the same
 299 name).
 300
 301 This command currently provides minimal information regarding mirror status::
 302
 303   $ ceph --admin-daemon /var/run/ceph/cephfs-mirror.asok fs mirror status cephfs@360
 304   {
 305     "rados_inst": "192.168.0.5:0/1476644347",
 306     "peers": {
 307         "a2dc7784-e7a1-4723-b103-03ee8d8768f8": {
 308             "remote": {
 309                 "client_name": "client.mirror_remote",
 310                 "cluster_name": "site-a",
 311                 "fs_name": "backup_fs"
 312             }
 313         }
 314     },
 315     "snap_dirs": {
 316         "dir_count": 1
 317     }
 318   }
 319
 320 The `Peers` section in the command output above shows the peer information including the unique
 321 peer-id (UUID) and specification. The peer-id is required when removing an existing peer
 322 as mentioned in the `Mirror Module and Interface` section.
 323
 324 Commands prefixed with `fs mirror peer status` provide peer synchronization status. This
 325 command is of format `filesystem-name@filesystem-id peer-uuid`::
 326
 327   $ ceph --admin-daemon /var/run/ceph/cephfs-mirror.asok fs mirror peer status cephfs@360 a2dc7784-e7a1-4723-b103-03ee8d8768f8
 328   {
 329     "/d0": {
 330         "state": "idle",
 331         "last_synced_snap": {
 332             "id": 120,
 333             "name": "snap1",
 334             "sync_duration": 0.079997898999999997,
 335             "sync_time_stamp": "274900.558797s"
 336         },
 337         "snaps_synced": 2,
 338         "snaps_deleted": 0,
 339         "snaps_renamed": 0
 340     }
 341   }
 342
 343 Synchronization stats including `snaps_synced`, `snaps_deleted` and `snaps_renamed` are reset
 344 on daemon restart and/or when a directory is reassigned to another mirror daemon (when
 345 multiple mirror daemons are deployed).
 346
 347 A directory can be in one of the following states::
 348
 349   - `idle`: The directory is currently not being synchronized
 350   - `syncing`: The directory is currently being synchronized
 351   - `failed`: The directory has hit upper limit of consecutive failures
 352
 353 When a directory experiences a configured number of consecutive synchronization failures, the
 354 mirror daemon marks it as `failed`. Synchronization for these directories is retried.
 355 By default, the number of consecutive failures before a directory is marked as failed
 356 is controlled by `cephfs_mirror_max_consecutive_failures_per_directory` configuration
 357 option (default: 10) and the retry interval for failed directories is controlled via
 358 `cephfs_mirror_retry_failed_directories_interval` configuration option (default: 60s).
 359
 360 E.g., adding a regular file for synchronization would result in failed status::
 361
 362   $ ceph fs snapshot mirror add cephfs /f0
 363   $ ceph --admin-daemon /var/run/ceph/cephfs-mirror.asok fs mirror peer status cephfs@360 a2dc7784-e7a1-4723-b103-03ee8d8768f8
 364   {
 365     "/d0": {
 366         "state": "idle",
 367         "last_synced_snap": {
 368             "id": 120,
 369             "name": "snap1",
 370             "sync_duration": 0.079997898999999997,
 371             "sync_time_stamp": "274900.558797s"
 372         },
 373         "snaps_synced": 2,
 374         "snaps_deleted": 0,
 375         "snaps_renamed": 0
 376     },
 377     "/f0": {
 378         "state": "failed",
 379         "snaps_synced": 0,
 380         "snaps_deleted": 0,
 381         "snaps_renamed": 0
 382     }
 383   }
 384
 385 This allows a user to add a non-existent directory for synchronization. The mirror daemon
 386 will mark such a directory as failed and retry (less frequently). When the directory is
 387 created, the mirror daemon will clear the failed state upon successful synchronization.
 388
 389 When mirroring is disabled, the respective `fs mirror status` command for the file system
 390 will not show up in command help.
 391
 392 Configuration Options
 393 ---------------------
 394
 395 .. confval:: cephfs_mirror_max_concurrent_directory_syncs
 396 .. confval:: cephfs_mirror_action_update_interval
 397 .. confval:: cephfs_mirror_restart_mirror_on_blocklist_interval
 398 .. confval:: cephfs_mirror_max_snapshot_sync_per_cycle
 399 .. confval:: cephfs_mirror_directory_scan_interval
 400 .. confval:: cephfs_mirror_max_consecutive_failures_per_directory
 401 .. confval:: cephfs_mirror_retry_failed_directories_interval
 402 .. confval:: cephfs_mirror_restart_mirror_on_failure_interval
 403 .. confval:: cephfs_mirror_mount_timeout
 404
 405 Re-adding Peers
 406 ---------------
 407
 408 When re-adding (reassigning) a peer to a file system in another cluster, ensure that
 409 all mirror daemons have stopped synchronization to the peer. This can be checked
 410 via `fs mirror status` admin socket command (the `Peer UUID` should not show up
 411 in the command output). Also, it is recommended to purge synchronized directories
 412 from the peer  before re-adding it to another file system (especially those directories
 413 which might exist in the new primary file system). This is not required if re-adding
 414 a peer to the same primary file system it was earlier synchronized from.