ceph/doc/cephfs/fs-volumes.rst

   1 .. _fs-volumes-and-subvolumes:
   2
   3 FS volumes and subvolumes
   4 =========================
   5
   6 The volumes module of the :term:`Ceph Manager` daemon (ceph-mgr) provides a
   7 single source of truth for CephFS exports. The OpenStack shared file system
   8 service (manila_) and the Ceph Container Storage Interface (CSI_) storage
   9 administrators use the common CLI provided by the ceph-mgr ``volumes`` module
  10 to manage CephFS exports.
  11
  12 The ceph-mgr ``volumes`` module implements the following file system export
  13 abstractions:
  14
  15 * FS volumes, an abstraction for CephFS file systems
  16
  17 * FS subvolumes, an abstraction for independent CephFS directory trees
  18
  19 * FS subvolume groups, an abstraction for a directory level higher than FS
  20   subvolumes. Used to effect policies (e.g., :doc:`/cephfs/file-layouts`)
  21   across a set of subvolumes
  22
  23 Some possible use-cases for the export abstractions:
  24
  25 * FS subvolumes used as Manila shares or CSI volumes
  26
  27 * FS subvolume groups used as Manila share groups
  28
  29 Requirements
  30 ------------
  31
  32 * Nautilus (14.2.x) or later Ceph release
  33
  34 * Cephx client user (see :doc:`/rados/operations/user-management`) with
  35   at least the following capabilities::
  36
  37     mon 'allow r'
  38     mgr 'allow rw'
  39
  40 FS Volumes
  41 ----------
  42
  43 Create a volume by running the following command:
  44
  45     $ ceph fs volume create <vol_name> [<placement>]
  46
  47 This creates a CephFS file system and its data and metadata pools. It can also
  48 deploy MDS daemons for the filesystem using a ceph-mgr orchestrator module (for
  49 example Rook). See :doc:`/mgr/orchestrator`.
  50
  51 ``<vol_name>`` is the volume name (an arbitrary string). ``<placement>`` is an
  52 optional string that specifies the hosts that should have an MDS running on
  53 them and, optionally, the total number of MDS daemons that the cluster should
  54 have. For example, the following placement string means "deploy MDS on nodes
  55 ``host1`` and ``host2`` (one MDS per host)::
  56
  57     "host1,host2"
  58
  59 The following placement specification means "deploy two MDS daemons on each of
  60 nodes ``host1`` and ``host2`` (for a total of four MDS daemons in the
  61 cluster)"::
  62
  63     "4 host1,host2"
  64
  65 See :ref:`orchestrator-cli-service-spec` for more on placement specification.
  66 Specifying placement via a YAML file is not supported.
  67
  68 To remove a volume, run the following command:
  69
  70     $ ceph fs volume rm <vol_name> [--yes-i-really-mean-it]
  71
  72 This removes a file system and its data and metadata pools. It also tries to
  73 remove MDS daemons using the enabled ceph-mgr orchestrator module.
  74
  75 List volumes by running the following command:
  76
  77     $ ceph fs volume ls
  78
  79 Rename a volume by running the following command:
  80
  81     $ ceph fs volume rename <vol_name> <new_vol_name> [--yes-i-really-mean-it]
  82
  83 Renaming a volume can be an expensive operation that requires the following:
  84
  85 - Renaming the orchestrator-managed MDS service to match the <new_vol_name>.
  86   This involves launching a MDS service with ``<new_vol_name>`` and bringing
  87   down the MDS service with ``<vol_name>``.
  88 - Renaming the file system matching ``<vol_name>`` to ``<new_vol_name>``.
  89 - Changing the application tags on the data and metadata pools of the file system
  90   to ``<new_vol_name>``.
  91 - Renaming the metadata and data pools of the file system.
  92
  93 The CephX IDs that are authorized for ``<vol_name>`` must be reauthorized for
  94 ``<new_vol_name>``. Any ongoing operations of the clients using these IDs may
  95 be disrupted. Ensure that mirroring is disabled on the volume.
  96
  97 To fetch the information of a CephFS volume, run the following command:
  98
  99     $ ceph fs volume info vol_name [--human_readable]
 100
 101 The ``--human_readable`` flag shows used and available pool capacities in KB/MB/GB.
 102
 103 The output format is JSON and contains fields as follows:
 104
 105 * ``pools``: Attributes of data and metadata pools
 106         * ``avail``: The amount of free space available in bytes
 107         * ``used``: The amount of storage consumed in bytes
 108         * ``name``: Name of the pool
 109 * ``mon_addrs``: List of Ceph monitor addresses
 110 * ``used_size``: Current used size of the CephFS volume in bytes
 111 * ``pending_subvolume_deletions``: Number of subvolumes pending deletion
 112
 113 Sample output of the ``volume info`` command::
 114
 115   $ ceph fs volume info vol_name
 116   {
 117       "mon_addrs": [
 118           "192.168.1.7:40977"
 119       ],
 120       "pending_subvolume_deletions": 0,
 121       "pools": {
 122           "data": [
 123               {
 124                   "avail": 106288709632,
 125                   "name": "cephfs.vol_name.data",
 126                   "used": 4096
 127               }
 128           ],
 129           "metadata": [
 130               {
 131                   "avail": 106288709632,
 132                   "name": "cephfs.vol_name.meta",
 133                   "used": 155648
 134               }
 135           ]
 136       },
 137       "used_size": 0
 138   }
 139
 140 FS Subvolume groups
 141 -------------------
 142
 143 Create a subvolume group by running the following command:
 144
 145     $ ceph fs subvolumegroup create <vol_name> <group_name> [--size <size_in_bytes>] [--pool_layout <data_pool_name>] [--uid <uid>] [--gid <gid>] [--mode <octal_mode>]
 146
 147 The command succeeds even if the subvolume group already exists.
 148
 149 When creating a subvolume group you can specify its data pool layout (see
 150 :doc:`/cephfs/file-layouts`), uid, gid, file mode in octal numerals, and
 151 size in bytes. The size of the subvolume group is specified by setting
 152 a quota on it (see :doc:`/cephfs/quota`). By default, the subvolume group
 153 is created with octal file mode ``755``, uid ``0``, gid ``0`` and the data pool
 154 layout of its parent directory.
 155
 156 Remove a subvolume group by running a command of the following form:
 157
 158     $ ceph fs subvolumegroup rm <vol_name> <group_name> [--force]
 159
 160 The removal of a subvolume group fails if the subvolume group is not empty or
 161 is non-existent. The ``--force`` flag allows the non-existent "subvolume group remove
 162 command" to succeed.
 163
 164 Fetch the absolute path of a subvolume group by running a command of the
 165 following form:
 166
 167     $ ceph fs subvolumegroup getpath <vol_name> <group_name>
 168
 169 List subvolume groups by running a command of the following form:
 170
 171     $ ceph fs subvolumegroup ls <vol_name>
 172
 173 .. note:: Subvolume group snapshot feature is no longer supported in mainline
 174    CephFS (existing group snapshots can still be listed and deleted)
 175
 176 Fetch the metadata of a subvolume group by running a command of the following form:
 177
 178 .. prompt:: bash #
 179
 180    ceph fs subvolumegroup info <vol_name> <group_name>
 181
 182 The output format is JSON and contains fields as follows:
 183
 184 * ``atime``: access time of the subvolume group path in the format "YYYY-MM-DD HH:MM:SS"
 185 * ``mtime``: modification time of the subvolume group path in the format "YYYY-MM-DD HH:MM:SS"
 186 * ``ctime``: change time of the subvolume group path in the format "YYYY-MM-DD HH:MM:SS"
 187 * ``uid``: uid of the subvolume group path
 188 * ``gid``: gid of the subvolume group path
 189 * ``mode``: mode of the subvolume group path
 190 * ``mon_addrs``: list of monitor addresses
 191 * ``bytes_pcent``: quota used in percentage if quota is set, else displays "undefined"
 192 * ``bytes_quota``: quota size in bytes if quota is set, else displays "infinite"
 193 * ``bytes_used``: current used size of the subvolume group in bytes
 194 * ``created_at``: creation time of the subvolume group in the format "YYYY-MM-DD HH:MM:SS"
 195 * ``data_pool``: data pool to which the subvolume group belongs
 196
 197 Check the presence of any subvolume group by running a command of the following
 198 form:
 199
 200 .. prompt:: bash $
 201
 202    ceph fs subvolumegroup exist <vol_name>
 203
 204 The ``exist`` command outputs:
 205
 206 * "subvolumegroup exists": if any subvolumegroup is present
 207 * "no subvolumegroup exists": if no subvolumegroup is present
 208
 209 .. note:: This command checks for the presence of custom groups and not
 210    presence of the default one. To validate the emptiness of the volume, a
 211    subvolumegroup existence check alone is not sufficient. Subvolume existence
 212    also needs to be checked as there might be subvolumes in the default group.
 213
 214 Resize a subvolume group by running a command of the following form:
 215
 216 .. prompt:: bash $
 217
 218    ceph fs subvolumegroup resize <vol_name> <group_name> <new_size> [--no_shrink]
 219
 220 The command resizes the subvolume group quota, using the size specified by
 221 ``new_size``.  The ``--no_shrink`` flag prevents the subvolume group from
 222 shrinking below the current used size.
 223
 224 The subvolume group may be resized to an infinite size by passing ``inf`` or
 225 ``infinite`` as the ``new_size``.
 226
 227 Remove a snapshot of a subvolume group by running a command of the following form:
 228
 229 .. prompt:: bash $
 230
 231    ceph fs subvolumegroup snapshot rm <vol_name> <group_name> <snap_name> [--force]
 232
 233 Supplying the ``--force`` flag allows the command to succeed when it would
 234 otherwise fail due to the nonexistence of the snapshot.
 235
 236 List snapshots of a subvolume group by running a command of the following form:
 237
 238 .. prompt:: bash $
 239
 240    ceph fs subvolumegroup snapshot ls <vol_name> <group_name>
 241
 242
 243 FS Subvolumes
 244 -------------
 245
 246 Create a subvolume using::
 247
 248     $ ceph fs subvolume create <vol_name> <subvol_name> [--size <size_in_bytes>] [--group_name <subvol_group_name>] [--pool_layout <data_pool_name>] [--uid <uid>] [--gid <gid>] [--mode <octal_mode>] [--namespace-isolated]
 249
 250
 251 The command succeeds even if the subvolume already exists.
 252
 253 When creating a subvolume you can specify its subvolume group, data pool layout,
 254 uid, gid, file mode in octal numerals, and size in bytes. The size of the subvolume is
 255 specified by setting a quota on it (see :doc:`/cephfs/quota`). The subvolume can be
 256 created in a separate RADOS namespace by specifying --namespace-isolated option. By
 257 default a subvolume is created within the default subvolume group, and with an octal file
 258 mode '755', uid of its subvolume group, gid of its subvolume group, data pool layout of
 259 its parent directory and no size limit.
 260
 261 Remove a subvolume using::
 262
 263     $ ceph fs subvolume rm <vol_name> <subvol_name> [--group_name <subvol_group_name>] [--force] [--retain-snapshots]
 264
 265 The command removes the subvolume and its contents. It does this in two steps.
 266 First, it moves the subvolume to a trash folder, and then asynchronously purges
 267 its contents.
 268
 269 The removal of a subvolume fails if it has snapshots, or is non-existent.
 270 '--force' flag allows the non-existent subvolume remove command to succeed.
 271
 272 A subvolume can be removed retaining existing snapshots of the subvolume using the
 273 '--retain-snapshots' option. If snapshots are retained, the subvolume is considered
 274 empty for all operations not involving the retained snapshots.
 275
 276 .. note:: Snapshot retained subvolumes can be recreated using 'ceph fs subvolume create'
 277
 278 .. note:: Retained snapshots can be used as a clone source to recreate the subvolume, or clone to a newer subvolume.
 279
 280 Resize a subvolume using::
 281
 282     $ ceph fs subvolume resize <vol_name> <subvol_name> <new_size> [--group_name <subvol_group_name>] [--no_shrink]
 283
 284 The command resizes the subvolume quota using the size specified by ``new_size``.
 285 The `--no_shrink`` flag prevents the subvolume from shrinking below the current  used size of the subvolume.
 286
 287 The subvolume can be resized to an unlimited (but sparse) logical size by passing ``inf`` or ``infinite`` as `` new_size``.
 288
 289 Authorize cephx auth IDs, the read/read-write access to fs subvolumes::
 290
 291     $ ceph fs subvolume authorize <vol_name> <sub_name> <auth_id> [--group_name=<group_name>] [--access_level=<access_level>]
 292
 293 The ``access_level`` takes ``r`` or ``rw`` as value.
 294
 295 Deauthorize cephx auth IDs, the read/read-write access to fs subvolumes::
 296
 297     $ ceph fs subvolume deauthorize <vol_name> <sub_name> <auth_id> [--group_name=<group_name>]
 298
 299 List cephx auth IDs authorized to access fs subvolume::
 300
 301     $ ceph fs subvolume authorized_list <vol_name> <sub_name> [--group_name=<group_name>]
 302
 303 Evict fs clients based on auth ID and subvolume mounted::
 304
 305     $ ceph fs subvolume evict <vol_name> <sub_name> <auth_id> [--group_name=<group_name>]
 306
 307 Fetch the absolute path of a subvolume using::
 308
 309     $ ceph fs subvolume getpath <vol_name> <subvol_name> [--group_name <subvol_group_name>]
 310
 311 Fetch the information of a subvolume using::
 312
 313     $ ceph fs subvolume info <vol_name> <subvol_name> [--group_name <subvol_group_name>]
 314
 315 The output format is JSON and contains fields as follows.
 316
 317 * ``atime``: access time of the subvolume path in the format "YYYY-MM-DD HH:MM:SS"
 318 * ``mtime``: modification time of the subvolume path in the format "YYYY-MM-DD HH:MM:SS"
 319 * ``ctime``: change time of the subvolume path in the format "YYYY-MM-DD HH:MM:SS"
 320 * ``uid``: uid of the subvolume path
 321 * ``gid``: gid of the subvolume path
 322 * ``mode``: mode of the subvolume path
 323 * ``mon_addrs``: list of monitor addresses
 324 * ``bytes_pcent``: quota used in percentage if quota is set, else displays ``undefined``
 325 * ``bytes_quota``: quota size in bytes if quota is set, else displays ``infinite``
 326 * ``bytes_used``: current used size of the subvolume in bytes
 327 * ``created_at``: creation time of the subvolume in the format "YYYY-MM-DD HH:MM:SS"
 328 * ``data_pool``: data pool to which the subvolume belongs
 329 * ``path``: absolute path of a subvolume
 330 * ``type``: subvolume type indicating whether it's clone or subvolume
 331 * ``pool_namespace``: RADOS namespace of the subvolume
 332 * ``features``: features supported by the subvolume
 333 * ``state``: current state of the subvolume
 334
 335 If a subvolume has been removed retaining its snapshots, the output contains only fields as follows.
 336
 337 * ``type``: subvolume type indicating whether it's clone or subvolume
 338 * ``features``: features supported by the subvolume
 339 * ``state``: current state of the subvolume
 340
 341 A subvolume's ``features`` are based on the internal version of the subvolume and are
 342 a subset of the following:
 343
 344 * ``snapshot-clone``: supports cloning using a subvolumes snapshot as the source
 345 * ``snapshot-autoprotect``: supports automatically protecting snapshots, that are active clone sources, from deletion
 346 * ``snapshot-retention``: supports removing subvolume contents, retaining any existing snapshots
 347
 348 A subvolume's ``state`` is based on the current state of the subvolume and contains one of the following values.
 349
 350 * ``complete``: subvolume is ready for all operations
 351 * ``snapshot-retained``: subvolume is removed but its snapshots are retained
 352
 353 List subvolumes using::
 354
 355     $ ceph fs subvolume ls <vol_name> [--group_name <subvol_group_name>]
 356
 357 .. note:: subvolumes that are removed but have snapshots retained, are also listed.
 358
 359 Check the presence of any subvolume using::
 360
 361     $ ceph fs subvolume exist <vol_name> [--group_name <subvol_group_name>]
 362
 363 These are the possible results of the ``exist`` command:
 364
 365 * ``subvolume exists``: if any subvolume of given group_name is present
 366 * ``no subvolume exists``: if no subvolume of given group_name is present
 367
 368 Set custom metadata on the subvolume as a key-value pair using::
 369
 370     $ ceph fs subvolume metadata set <vol_name> <subvol_name> <key_name> <value> [--group_name <subvol_group_name>]
 371
 372 .. note:: If the key_name already exists then the old value will get replaced by the new value.
 373
 374 .. note:: key_name and value should be a string of ASCII characters (as specified in python's string.printable). key_name is case-insensitive and always stored in lower case.
 375
 376 .. note:: Custom metadata on a subvolume is not preserved when snapshotting the subvolume, and hence, is also not preserved when cloning the subvolume snapshot.
 377
 378 Get custom metadata set on the subvolume using the metadata key::
 379
 380     $ ceph fs subvolume metadata get <vol_name> <subvol_name> <key_name> [--group_name <subvol_group_name>]
 381
 382 List custom metadata (key-value pairs) set on the subvolume using::
 383
 384     $ ceph fs subvolume metadata ls <vol_name> <subvol_name> [--group_name <subvol_group_name>]
 385
 386 Remove custom metadata set on the subvolume using the metadata key::
 387
 388     $ ceph fs subvolume metadata rm <vol_name> <subvol_name> <key_name> [--group_name <subvol_group_name>] [--force]
 389
 390 Using the ``--force`` flag allows the command to succeed that would otherwise
 391 fail if the metadata key did not exist.
 392
 393 Create a snapshot of a subvolume using::
 394
 395     $ ceph fs subvolume snapshot create <vol_name> <subvol_name> <snap_name> [--group_name <subvol_group_name>]
 396
 397 Remove a snapshot of a subvolume using::
 398
 399     $ ceph fs subvolume snapshot rm <vol_name> <subvol_name> <snap_name> [--group_name <subvol_group_name>] [--force]
 400
 401 Using the ``--force`` flag allows the command to succeed that would otherwise
 402 fail if the snapshot did not exist.
 403
 404 .. note:: if the last snapshot within a snapshot retained subvolume is removed, the subvolume is also removed
 405
 406 List snapshots of a subvolume using::
 407
 408     $ ceph fs subvolume snapshot ls <vol_name> <subvol_name> [--group_name <subvol_group_name>]
 409
 410 Fetch the information of a snapshot using::
 411
 412     $ ceph fs subvolume snapshot info <vol_name> <subvol_name> <snap_name> [--group_name <subvol_group_name>]
 413
 414 The output format is JSON and contains fields as follows.
 415
 416 * ``created_at``: creation time of the snapshot in the format "YYYY-MM-DD HH:MM:SS:ffffff"
 417 * ``data_pool``: data pool to which the snapshot belongs
 418 * ``has_pending_clones``: ``yes`` if snapshot clone is in progress, otherwise ``no``
 419 * ``pending_clones``: list of in-progress or pending clones and their target group if any exist, otherwise this field is not shown
 420 * ``orphan_clones_count``: count of orphan clones if the snapshot has orphan clones, otherwise this field is not shown
 421
 422 Sample output when snapshot clones are in progress or pending::
 423
 424   $ ceph fs subvolume snapshot info cephfs subvol snap
 425   {
 426       "created_at": "2022-06-14 13:54:58.618769",
 427       "data_pool": "cephfs.cephfs.data",
 428       "has_pending_clones": "yes",
 429       "pending_clones": [
 430           {
 431               "name": "clone_1",
 432               "target_group": "target_subvol_group"
 433           },
 434           {
 435               "name": "clone_2"
 436           },
 437           {
 438               "name": "clone_3",
 439               "target_group": "target_subvol_group"
 440           }
 441       ]
 442   }
 443
 444 Sample output when no snapshot clone is in progress or pending::
 445
 446   $ ceph fs subvolume snapshot info cephfs subvol snap
 447   {
 448       "created_at": "2022-06-14 13:54:58.618769",
 449       "data_pool": "cephfs.cephfs.data",
 450       "has_pending_clones": "no"
 451   }
 452
 453 Set custom key-value metadata on the snapshot by running::
 454
 455     $ ceph fs subvolume snapshot metadata set <vol_name> <subvol_name> <snap_name> <key_name> <value> [--group_name <subvol_group_name>]
 456
 457 .. note:: If the key_name already exists then the old value will get replaced by the new value.
 458
 459 .. note:: The key_name and value should be a strings of ASCII characters (as specified in Python's ``string.printable``). The key_name is case-insensitive and always stored in lowercase.
 460
 461 .. note:: Custom metadata on a snapshot is not preserved when snapshotting the subvolume, and hence is also not preserved when cloning the subvolume snapshot.
 462
 463 Get custom metadata set on the snapshot using the metadata key::
 464
 465     $ ceph fs subvolume snapshot metadata get <vol_name> <subvol_name> <snap_name> <key_name> [--group_name <subvol_group_name>]
 466
 467 List custom metadata (key-value pairs) set on the snapshot using::
 468
 469     $ ceph fs subvolume snapshot metadata ls <vol_name> <subvol_name> <snap_name> [--group_name <subvol_group_name>]
 470
 471 Remove custom metadata set on the snapshot using the metadata key::
 472
 473     $ ceph fs subvolume snapshot metadata rm <vol_name> <subvol_name> <snap_name> <key_name> [--group_name <subvol_group_name>] [--force]
 474
 475 Using the ``--force`` flag allows the command to succeed that would otherwise
 476 fail if the metadata key did not exist.
 477
 478 Cloning Snapshots
 479 -----------------
 480
 481 Subvolumes can be created by cloning subvolume snapshots. Cloning is an asynchronous operation that copies
 482 data from a snapshot to a subvolume. Due to this bulk copying, cloning is inefficient for very large
 483 data sets.
 484
 485 .. note:: Removing a snapshot (source subvolume) would fail if there are pending or in progress clone operations.
 486
 487 Protecting snapshots prior to cloning was a prerequisite in the Nautilus release, and the commands to protect/unprotect
 488 snapshots were introduced for this purpose. This prerequisite, and hence the commands to protect/unprotect, is being
 489 deprecated and may be removed from a future release.
 490
 491 The commands being deprecated are:
 492
 493 .. prompt:: bash #
 494
 495    ceph fs subvolume snapshot protect <vol_name> <subvol_name> <snap_name> [--group_name <subvol_group_name>]
 496    ceph fs subvolume snapshot unprotect <vol_name> <subvol_name> <snap_name> [--group_name <subvol_group_name>]
 497
 498 .. note:: Using the above commands will not result in an error, but they have no useful purpose.
 499
 500 .. note:: Use the ``subvolume info`` command to fetch subvolume metadata regarding supported ``features`` to help decide if protect/unprotect of snapshots is required, based on the availability of the ``snapshot-autoprotect`` feature.
 501
 502 To initiate a clone operation use::
 503
 504   $ ceph fs subvolume snapshot clone <vol_name> <subvol_name> <snap_name> <target_subvol_name>
 505
 506 If a snapshot (source subvolume) is a part of non-default group, the group name needs to be specified::
 507
 508   $ ceph fs subvolume snapshot clone <vol_name> <subvol_name> <snap_name> <target_subvol_name> --group_name <subvol_group_name>
 509
 510 Cloned subvolumes can be a part of a different group than the source snapshot (by default, cloned subvolumes are created in default group). To clone to a particular group use::
 511
 512   $ ceph fs subvolume snapshot clone <vol_name> <subvol_name> <snap_name> <target_subvol_name> --target_group_name <subvol_group_name>
 513
 514 Similar to specifying a pool layout when creating a subvolume, pool layout can be specified when creating a cloned subvolume. To create a cloned subvolume with a specific pool layout use::
 515
 516   $ ceph fs subvolume snapshot clone <vol_name> <subvol_name> <snap_name> <target_subvol_name> --pool_layout <pool_layout>
 517
 518 Configure the maximum number of concurrent clones. The default is 4::
 519
 520   $ ceph config set mgr mgr/volumes/max_concurrent_clones <value>
 521
 522 To check the status of a clone operation use::
 523
 524   $ ceph fs clone status <vol_name> <clone_name> [--group_name <group_name>]
 525
 526 A clone can be in one of the following states:
 527
 528 #. ``pending``     : Clone operation has not started
 529 #. ``in-progress`` : Clone operation is in progress
 530 #. ``complete``    : Clone operation has successfully finished
 531 #. ``failed``      : Clone operation has failed
 532 #. ``canceled``    : Clone operation is cancelled by user
 533
 534 The reason for a clone failure is shown as below:
 535
 536 #. ``errno``     : error number
 537 #. ``error_msg`` : failure error string
 538
 539 Here is an example of an ``in-progress`` clone::
 540
 541   $ ceph fs subvolume snapshot clone cephfs subvol1 snap1 clone1
 542   $ ceph fs clone status cephfs clone1
 543   {
 544     "status": {
 545       "state": "in-progress",
 546       "source": {
 547         "volume": "cephfs",
 548         "subvolume": "subvol1",
 549         "snapshot": "snap1"
 550       }
 551     }
 552   }
 553
 554 .. note:: The ``failure`` section will be shown only if the clone's state is ``failed`` or ``cancelled``
 555
 556 Here is an example of a ``failed`` clone::
 557
 558   $ ceph fs subvolume snapshot clone cephfs subvol1 snap1 clone1
 559   $ ceph fs clone status cephfs clone1
 560   {
 561     "status": {
 562       "state": "failed",
 563       "source": {
 564         "volume": "cephfs",
 565         "subvolume": "subvol1",
 566         "snapshot": "snap1"
 567         "size": "104857600"
 568       },
 569       "failure": {
 570         "errno": "122",
 571         "errstr": "Disk quota exceeded"
 572       }
 573     }
 574   }
 575
 576 (NOTE: since ``subvol1`` is in the default group, the ``source`` object's  ``clone status`` does not include the group name)
 577
 578 .. note:: Cloned subvolumes are accessible only after the clone operation has successfully completed.
 579
 580 After a successful clone operation, ``clone status`` will look like the below::
 581
 582   $ ceph fs clone status cephfs clone1
 583   {
 584     "status": {
 585       "state": "complete"
 586     }
 587   }
 588
 589 If a clone operation is unsuccessful, the ``state`` value will be  ``failed``.
 590
 591 To retry a failed clone operation, the incomplete clone must be deleted and the clone operation must be issued again.
 592 To delete a partial clone use::
 593
 594   $ ceph fs subvolume rm <vol_name> <clone_name> [--group_name <group_name>] --force
 595
 596 .. note:: Cloning synchronizes only directories, regular files and symbolic links. Inode timestamps (access and
 597           modification times) are synchronized up to seconds granularity.
 598
 599 An ``in-progress`` or a ``pending`` clone operation may be canceled. To cancel a clone operation use the ``clone cancel`` command::
 600
 601   $ ceph fs clone cancel <vol_name> <clone_name> [--group_name <group_name>]
 602
 603 On successful cancellation, the cloned subvolume is moved to the ``canceled`` state::
 604
 605   $ ceph fs subvolume snapshot clone cephfs subvol1 snap1 clone1
 606   $ ceph fs clone cancel cephfs clone1
 607   $ ceph fs clone status cephfs clone1
 608   {
 609     "status": {
 610       "state": "canceled",
 611       "source": {
 612         "volume": "cephfs",
 613         "subvolume": "subvol1",
 614         "snapshot": "snap1"
 615       }
 616     }
 617   }
 618
 619 .. note:: The canceled cloned may be deleted by supplying the ``--force`` option to the `fs subvolume rm` command.
 620
 621
 622 .. _subvol-pinning:
 623
 624 Pinning Subvolumes and Subvolume Groups
 625 ---------------------------------------
 626
 627 Subvolumes and subvolume groups may be automatically pinned to ranks according
 628 to policies. This can distribute load across MDS ranks in predictable and
 629 stable ways.  Review :ref:`cephfs-pinning` and :ref:`cephfs-ephemeral-pinning`
 630 for details on how pinning works.
 631
 632 Pinning is configured by::
 633
 634   $ ceph fs subvolumegroup pin <vol_name> <group_name> <pin_type> <pin_setting>
 635
 636 or for subvolumes::
 637
 638   $ ceph fs subvolume pin <vol_name> <group_name> <pin_type> <pin_setting>
 639
 640 Typically you will want to set subvolume group pins. The ``pin_type`` may be
 641 one of ``export``, ``distributed``, or ``random``. The ``pin_setting``
 642 corresponds to the extended attributed "value" as in the pinning documentation
 643 referenced above.
 644
 645 So, for example, setting a distributed pinning strategy on a subvolume group::
 646
 647   $ ceph fs subvolumegroup pin cephfilesystem-a csi distributed 1
 648
 649 Will enable distributed subtree partitioning policy for the "csi" subvolume
 650 group.  This will cause every subvolume within the group to be automatically
 651 pinned to one of the available ranks on the file system.
 652
 653
 654 .. _manila: https://github.com/openstack/manila
 655 .. _CSI: https://github.com/ceph/ceph-csi