1 .. _fs-volumes-and-subvolumes:
3 FS volumes and subvolumes
4 =========================
6 The volumes module of the :term:`Ceph Manager` daemon (ceph-mgr) provides a
7 single source of truth for CephFS exports. The OpenStack shared file system
8 service (manila_) and the Ceph Container Storage Interface (CSI_) storage
9 administrators use the common CLI provided by the ceph-mgr ``volumes`` module
10 to manage CephFS exports.
12 The ceph-mgr ``volumes`` module implements the following file system export
15 * FS volumes, an abstraction for CephFS file systems
17 * FS subvolumes, an abstraction for independent CephFS directory trees
19 * FS subvolume groups, an abstraction for a directory level higher than FS
20 subvolumes. Used to effect policies (e.g., :doc:`/cephfs/file-layouts`)
21 across a set of subvolumes
23 Some possible use-cases for the export abstractions:
25 * FS subvolumes used as Manila shares or CSI volumes
27 * FS subvolume groups used as Manila share groups
32 * Nautilus (14.2.x) or later Ceph release
34 * Cephx client user (see :doc:`/rados/operations/user-management`) with
35 at least the following capabilities::
43 Create a volume by running the following command:
47 ceph fs volume create <vol_name> [placement]
49 This creates a CephFS file system and its data and metadata pools. It can also
50 deploy MDS daemons for the filesystem using a ceph-mgr orchestrator module (for
51 example Rook). See :doc:`/mgr/orchestrator`.
53 ``<vol_name>`` is the volume name (an arbitrary string). ``[placement]`` is an
54 optional string that specifies the :ref:`orchestrator-cli-placement-spec` for
55 the MDS. See also :ref:`orchestrator-cli-cephfs` for more examples on
58 .. note:: Specifying placement via a YAML file is not supported through the
61 To remove a volume, run the following command:
63 $ ceph fs volume rm <vol_name> [--yes-i-really-mean-it]
65 This removes a file system and its data and metadata pools. It also tries to
66 remove MDS daemons using the enabled ceph-mgr orchestrator module.
68 .. note:: After volume deletion, it is recommended to restart `ceph-mgr`
69 if a new file system is created on the same cluster and subvolume interface
70 is being used. Please see https://tracker.ceph.com/issues/49605#note-5
73 List volumes by running the following command:
77 Rename a volume by running the following command:
79 $ ceph fs volume rename <vol_name> <new_vol_name> [--yes-i-really-mean-it]
81 Renaming a volume can be an expensive operation that requires the following:
83 - Renaming the orchestrator-managed MDS service to match the <new_vol_name>.
84 This involves launching a MDS service with ``<new_vol_name>`` and bringing
85 down the MDS service with ``<vol_name>``.
86 - Renaming the file system matching ``<vol_name>`` to ``<new_vol_name>``.
87 - Changing the application tags on the data and metadata pools of the file system
88 to ``<new_vol_name>``.
89 - Renaming the metadata and data pools of the file system.
91 The CephX IDs that are authorized for ``<vol_name>`` must be reauthorized for
92 ``<new_vol_name>``. Any ongoing operations of the clients using these IDs may
93 be disrupted. Ensure that mirroring is disabled on the volume.
95 To fetch the information of a CephFS volume, run the following command:
97 $ ceph fs volume info vol_name [--human_readable]
99 The ``--human_readable`` flag shows used and available pool capacities in KB/MB/GB.
101 The output format is JSON and contains fields as follows:
103 * ``pools``: Attributes of data and metadata pools
104 * ``avail``: The amount of free space available in bytes
105 * ``used``: The amount of storage consumed in bytes
106 * ``name``: Name of the pool
107 * ``mon_addrs``: List of Ceph monitor addresses
108 * ``used_size``: Current used size of the CephFS volume in bytes
109 * ``pending_subvolume_deletions``: Number of subvolumes pending deletion
111 Sample output of the ``volume info`` command::
113 $ ceph fs volume info vol_name
118 "pending_subvolume_deletions": 0,
122 "avail": 106288709632,
123 "name": "cephfs.vol_name.data",
129 "avail": 106288709632,
130 "name": "cephfs.vol_name.meta",
141 Create a subvolume group by running the following command:
143 $ ceph fs subvolumegroup create <vol_name> <group_name> [--size <size_in_bytes>] [--pool_layout <data_pool_name>] [--uid <uid>] [--gid <gid>] [--mode <octal_mode>]
145 The command succeeds even if the subvolume group already exists.
147 When creating a subvolume group you can specify its data pool layout (see
148 :doc:`/cephfs/file-layouts`), uid, gid, file mode in octal numerals, and
149 size in bytes. The size of the subvolume group is specified by setting
150 a quota on it (see :doc:`/cephfs/quota`). By default, the subvolume group
151 is created with octal file mode ``755``, uid ``0``, gid ``0`` and the data pool
152 layout of its parent directory.
154 Remove a subvolume group by running a command of the following form:
156 $ ceph fs subvolumegroup rm <vol_name> <group_name> [--force]
158 The removal of a subvolume group fails if the subvolume group is not empty or
159 is non-existent. The ``--force`` flag allows the non-existent "subvolume group remove
162 Fetch the absolute path of a subvolume group by running a command of the
165 $ ceph fs subvolumegroup getpath <vol_name> <group_name>
167 List subvolume groups by running a command of the following form:
169 $ ceph fs subvolumegroup ls <vol_name>
171 .. note:: Subvolume group snapshot feature is no longer supported in mainline
172 CephFS (existing group snapshots can still be listed and deleted)
174 Fetch the metadata of a subvolume group by running a command of the following form:
178 ceph fs subvolumegroup info <vol_name> <group_name>
180 The output format is JSON and contains fields as follows:
182 * ``atime``: access time of the subvolume group path in the format "YYYY-MM-DD HH:MM:SS"
183 * ``mtime``: modification time of the subvolume group path in the format "YYYY-MM-DD HH:MM:SS"
184 * ``ctime``: change time of the subvolume group path in the format "YYYY-MM-DD HH:MM:SS"
185 * ``uid``: uid of the subvolume group path
186 * ``gid``: gid of the subvolume group path
187 * ``mode``: mode of the subvolume group path
188 * ``mon_addrs``: list of monitor addresses
189 * ``bytes_pcent``: quota used in percentage if quota is set, else displays "undefined"
190 * ``bytes_quota``: quota size in bytes if quota is set, else displays "infinite"
191 * ``bytes_used``: current used size of the subvolume group in bytes
192 * ``created_at``: creation time of the subvolume group in the format "YYYY-MM-DD HH:MM:SS"
193 * ``data_pool``: data pool to which the subvolume group belongs
195 Check the presence of any subvolume group by running a command of the following
200 ceph fs subvolumegroup exist <vol_name>
202 The ``exist`` command outputs:
204 * "subvolumegroup exists": if any subvolumegroup is present
205 * "no subvolumegroup exists": if no subvolumegroup is present
207 .. note:: This command checks for the presence of custom groups and not
208 presence of the default one. To validate the emptiness of the volume, a
209 subvolumegroup existence check alone is not sufficient. Subvolume existence
210 also needs to be checked as there might be subvolumes in the default group.
212 Resize a subvolume group by running a command of the following form:
216 ceph fs subvolumegroup resize <vol_name> <group_name> <new_size> [--no_shrink]
218 The command resizes the subvolume group quota, using the size specified by
219 ``new_size``. The ``--no_shrink`` flag prevents the subvolume group from
220 shrinking below the current used size.
222 The subvolume group may be resized to an infinite size by passing ``inf`` or
223 ``infinite`` as the ``new_size``.
225 Remove a snapshot of a subvolume group by running a command of the following form:
229 ceph fs subvolumegroup snapshot rm <vol_name> <group_name> <snap_name> [--force]
231 Supplying the ``--force`` flag allows the command to succeed when it would
232 otherwise fail due to the nonexistence of the snapshot.
234 List snapshots of a subvolume group by running a command of the following form:
238 ceph fs subvolumegroup snapshot ls <vol_name> <group_name>
244 Create a subvolume using::
246 $ ceph fs subvolume create <vol_name> <subvol_name> [--size <size_in_bytes>] [--group_name <subvol_group_name>] [--pool_layout <data_pool_name>] [--uid <uid>] [--gid <gid>] [--mode <octal_mode>] [--namespace-isolated]
249 The command succeeds even if the subvolume already exists.
251 When creating a subvolume you can specify its subvolume group, data pool layout,
252 uid, gid, file mode in octal numerals, and size in bytes. The size of the subvolume is
253 specified by setting a quota on it (see :doc:`/cephfs/quota`). The subvolume can be
254 created in a separate RADOS namespace by specifying --namespace-isolated option. By
255 default a subvolume is created within the default subvolume group, and with an octal file
256 mode '755', uid of its subvolume group, gid of its subvolume group, data pool layout of
257 its parent directory and no size limit.
259 Remove a subvolume using::
261 $ ceph fs subvolume rm <vol_name> <subvol_name> [--group_name <subvol_group_name>] [--force] [--retain-snapshots]
263 The command removes the subvolume and its contents. It does this in two steps.
264 First, it moves the subvolume to a trash folder, and then asynchronously purges
267 The removal of a subvolume fails if it has snapshots, or is non-existent.
268 '--force' flag allows the non-existent subvolume remove command to succeed.
270 A subvolume can be removed retaining existing snapshots of the subvolume using the
271 '--retain-snapshots' option. If snapshots are retained, the subvolume is considered
272 empty for all operations not involving the retained snapshots.
274 .. note:: Snapshot retained subvolumes can be recreated using 'ceph fs subvolume create'
276 .. note:: Retained snapshots can be used as a clone source to recreate the subvolume, or clone to a newer subvolume.
278 Resize a subvolume using::
280 $ ceph fs subvolume resize <vol_name> <subvol_name> <new_size> [--group_name <subvol_group_name>] [--no_shrink]
282 The command resizes the subvolume quota using the size specified by ``new_size``.
283 The `--no_shrink`` flag prevents the subvolume from shrinking below the current used size of the subvolume.
285 The subvolume can be resized to an unlimited (but sparse) logical size by passing ``inf`` or ``infinite`` as `` new_size``.
287 Authorize cephx auth IDs, the read/read-write access to fs subvolumes::
289 $ ceph fs subvolume authorize <vol_name> <sub_name> <auth_id> [--group_name=<group_name>] [--access_level=<access_level>]
291 The ``access_level`` takes ``r`` or ``rw`` as value.
293 Deauthorize cephx auth IDs, the read/read-write access to fs subvolumes::
295 $ ceph fs subvolume deauthorize <vol_name> <sub_name> <auth_id> [--group_name=<group_name>]
297 List cephx auth IDs authorized to access fs subvolume::
299 $ ceph fs subvolume authorized_list <vol_name> <sub_name> [--group_name=<group_name>]
301 Evict fs clients based on auth ID and subvolume mounted::
303 $ ceph fs subvolume evict <vol_name> <sub_name> <auth_id> [--group_name=<group_name>]
305 Fetch the absolute path of a subvolume using::
307 $ ceph fs subvolume getpath <vol_name> <subvol_name> [--group_name <subvol_group_name>]
309 Fetch the information of a subvolume using::
311 $ ceph fs subvolume info <vol_name> <subvol_name> [--group_name <subvol_group_name>]
313 The output format is JSON and contains fields as follows.
315 * ``atime``: access time of the subvolume path in the format "YYYY-MM-DD HH:MM:SS"
316 * ``mtime``: modification time of the subvolume path in the format "YYYY-MM-DD HH:MM:SS"
317 * ``ctime``: change time of the subvolume path in the format "YYYY-MM-DD HH:MM:SS"
318 * ``uid``: uid of the subvolume path
319 * ``gid``: gid of the subvolume path
320 * ``mode``: mode of the subvolume path
321 * ``mon_addrs``: list of monitor addresses
322 * ``bytes_pcent``: quota used in percentage if quota is set, else displays ``undefined``
323 * ``bytes_quota``: quota size in bytes if quota is set, else displays ``infinite``
324 * ``bytes_used``: current used size of the subvolume in bytes
325 * ``created_at``: creation time of the subvolume in the format "YYYY-MM-DD HH:MM:SS"
326 * ``data_pool``: data pool to which the subvolume belongs
327 * ``path``: absolute path of a subvolume
328 * ``type``: subvolume type indicating whether it's clone or subvolume
329 * ``pool_namespace``: RADOS namespace of the subvolume
330 * ``features``: features supported by the subvolume
331 * ``state``: current state of the subvolume
333 If a subvolume has been removed retaining its snapshots, the output contains only fields as follows.
335 * ``type``: subvolume type indicating whether it's clone or subvolume
336 * ``features``: features supported by the subvolume
337 * ``state``: current state of the subvolume
339 A subvolume's ``features`` are based on the internal version of the subvolume and are
340 a subset of the following:
342 * ``snapshot-clone``: supports cloning using a subvolumes snapshot as the source
343 * ``snapshot-autoprotect``: supports automatically protecting snapshots, that are active clone sources, from deletion
344 * ``snapshot-retention``: supports removing subvolume contents, retaining any existing snapshots
346 A subvolume's ``state`` is based on the current state of the subvolume and contains one of the following values.
348 * ``complete``: subvolume is ready for all operations
349 * ``snapshot-retained``: subvolume is removed but its snapshots are retained
351 List subvolumes using::
353 $ ceph fs subvolume ls <vol_name> [--group_name <subvol_group_name>]
355 .. note:: subvolumes that are removed but have snapshots retained, are also listed.
357 Check the presence of any subvolume using::
359 $ ceph fs subvolume exist <vol_name> [--group_name <subvol_group_name>]
361 These are the possible results of the ``exist`` command:
363 * ``subvolume exists``: if any subvolume of given group_name is present
364 * ``no subvolume exists``: if no subvolume of given group_name is present
366 Set custom metadata on the subvolume as a key-value pair using::
368 $ ceph fs subvolume metadata set <vol_name> <subvol_name> <key_name> <value> [--group_name <subvol_group_name>]
370 .. note:: If the key_name already exists then the old value will get replaced by the new value.
372 .. note:: key_name and value should be a string of ASCII characters (as specified in python's string.printable). key_name is case-insensitive and always stored in lower case.
374 .. note:: Custom metadata on a subvolume is not preserved when snapshotting the subvolume, and hence, is also not preserved when cloning the subvolume snapshot.
376 Get custom metadata set on the subvolume using the metadata key::
378 $ ceph fs subvolume metadata get <vol_name> <subvol_name> <key_name> [--group_name <subvol_group_name>]
380 List custom metadata (key-value pairs) set on the subvolume using::
382 $ ceph fs subvolume metadata ls <vol_name> <subvol_name> [--group_name <subvol_group_name>]
384 Remove custom metadata set on the subvolume using the metadata key::
386 $ ceph fs subvolume metadata rm <vol_name> <subvol_name> <key_name> [--group_name <subvol_group_name>] [--force]
388 Using the ``--force`` flag allows the command to succeed that would otherwise
389 fail if the metadata key did not exist.
391 Create a snapshot of a subvolume using::
393 $ ceph fs subvolume snapshot create <vol_name> <subvol_name> <snap_name> [--group_name <subvol_group_name>]
395 Remove a snapshot of a subvolume using::
397 $ ceph fs subvolume snapshot rm <vol_name> <subvol_name> <snap_name> [--group_name <subvol_group_name>] [--force]
399 Using the ``--force`` flag allows the command to succeed that would otherwise
400 fail if the snapshot did not exist.
402 .. note:: if the last snapshot within a snapshot retained subvolume is removed, the subvolume is also removed
404 List snapshots of a subvolume using::
406 $ ceph fs subvolume snapshot ls <vol_name> <subvol_name> [--group_name <subvol_group_name>]
408 Fetch the information of a snapshot using::
410 $ ceph fs subvolume snapshot info <vol_name> <subvol_name> <snap_name> [--group_name <subvol_group_name>]
412 The output format is JSON and contains fields as follows.
414 * ``created_at``: creation time of the snapshot in the format "YYYY-MM-DD HH:MM:SS:ffffff"
415 * ``data_pool``: data pool to which the snapshot belongs
416 * ``has_pending_clones``: ``yes`` if snapshot clone is in progress, otherwise ``no``
417 * ``pending_clones``: list of in-progress or pending clones and their target group if any exist, otherwise this field is not shown
418 * ``orphan_clones_count``: count of orphan clones if the snapshot has orphan clones, otherwise this field is not shown
420 Sample output when snapshot clones are in progress or pending::
422 $ ceph fs subvolume snapshot info cephfs subvol snap
424 "created_at": "2022-06-14 13:54:58.618769",
425 "data_pool": "cephfs.cephfs.data",
426 "has_pending_clones": "yes",
430 "target_group": "target_subvol_group"
437 "target_group": "target_subvol_group"
442 Sample output when no snapshot clone is in progress or pending::
444 $ ceph fs subvolume snapshot info cephfs subvol snap
446 "created_at": "2022-06-14 13:54:58.618769",
447 "data_pool": "cephfs.cephfs.data",
448 "has_pending_clones": "no"
451 Set custom key-value metadata on the snapshot by running::
453 $ ceph fs subvolume snapshot metadata set <vol_name> <subvol_name> <snap_name> <key_name> <value> [--group_name <subvol_group_name>]
455 .. note:: If the key_name already exists then the old value will get replaced by the new value.
457 .. note:: The key_name and value should be a strings of ASCII characters (as specified in Python's ``string.printable``). The key_name is case-insensitive and always stored in lowercase.
459 .. note:: Custom metadata on a snapshot is not preserved when snapshotting the subvolume, and hence is also not preserved when cloning the subvolume snapshot.
461 Get custom metadata set on the snapshot using the metadata key::
463 $ ceph fs subvolume snapshot metadata get <vol_name> <subvol_name> <snap_name> <key_name> [--group_name <subvol_group_name>]
465 List custom metadata (key-value pairs) set on the snapshot using::
467 $ ceph fs subvolume snapshot metadata ls <vol_name> <subvol_name> <snap_name> [--group_name <subvol_group_name>]
469 Remove custom metadata set on the snapshot using the metadata key::
471 $ ceph fs subvolume snapshot metadata rm <vol_name> <subvol_name> <snap_name> <key_name> [--group_name <subvol_group_name>] [--force]
473 Using the ``--force`` flag allows the command to succeed that would otherwise
474 fail if the metadata key did not exist.
479 Subvolumes can be created by cloning subvolume snapshots. Cloning is an asynchronous operation that copies
480 data from a snapshot to a subvolume. Due to this bulk copying, cloning is inefficient for very large
483 .. note:: Removing a snapshot (source subvolume) would fail if there are pending or in progress clone operations.
485 Protecting snapshots prior to cloning was a prerequisite in the Nautilus release, and the commands to protect/unprotect
486 snapshots were introduced for this purpose. This prerequisite, and hence the commands to protect/unprotect, is being
487 deprecated and may be removed from a future release.
489 The commands being deprecated are:
493 ceph fs subvolume snapshot protect <vol_name> <subvol_name> <snap_name> [--group_name <subvol_group_name>]
494 ceph fs subvolume snapshot unprotect <vol_name> <subvol_name> <snap_name> [--group_name <subvol_group_name>]
496 .. note:: Using the above commands will not result in an error, but they have no useful purpose.
498 .. note:: Use the ``subvolume info`` command to fetch subvolume metadata regarding supported ``features`` to help decide if protect/unprotect of snapshots is required, based on the availability of the ``snapshot-autoprotect`` feature.
500 To initiate a clone operation use::
502 $ ceph fs subvolume snapshot clone <vol_name> <subvol_name> <snap_name> <target_subvol_name>
504 .. note:: ``subvolume snapshot clone`` command depends upon the above mentioned config option ``snapshot_clone_no_wait``
506 If a snapshot (source subvolume) is a part of non-default group, the group name needs to be specified::
508 $ ceph fs subvolume snapshot clone <vol_name> <subvol_name> <snap_name> <target_subvol_name> --group_name <subvol_group_name>
510 If a snapshot (source subvolume) is a part of non-default group, the group name needs to be specified:
512 Cloned subvolumes can be a part of a different group than the source snapshot (by default, cloned subvolumes are created in default group). To clone to a particular group use::
514 $ ceph fs subvolume snapshot clone <vol_name> <subvol_name> <snap_name> <target_subvol_name> --target_group_name <subvol_group_name>
516 Similar to specifying a pool layout when creating a subvolume, pool layout can be specified when creating a cloned subvolume. To create a cloned subvolume with a specific pool layout use::
518 $ ceph fs subvolume snapshot clone <vol_name> <subvol_name> <snap_name> <target_subvol_name> --pool_layout <pool_layout>
520 To check the status of a clone operation use::
522 ceph fs subvolume snapshot clone <vol_name> <subvol_name> <snap_name> <target_subvol_name> --pool_layout <pool_layout>
524 To check the status of a clone operation use:
528 ceph fs clone status <vol_name> <clone_name> [--group_name <group_name>]
530 A clone can be in one of the following states:
532 #. ``pending`` : Clone operation has not started
533 #. ``in-progress`` : Clone operation is in progress
534 #. ``complete`` : Clone operation has successfully finished
535 #. ``failed`` : Clone operation has failed
536 #. ``canceled`` : Clone operation is cancelled by user
538 The reason for a clone failure is shown as below:
540 #. ``errno`` : error number
541 #. ``error_msg`` : failure error string
543 Here is an example of an ``in-progress`` clone::
545 $ ceph fs subvolume snapshot clone cephfs subvol1 snap1 clone1
546 $ ceph fs clone status cephfs clone1
549 "state": "in-progress",
552 "subvolume": "subvol1",
558 .. note:: The ``failure`` section will be shown only if the clone's state is ``failed`` or ``cancelled``
560 Here is an example of a ``failed`` clone::
562 $ ceph fs subvolume snapshot clone cephfs subvol1 snap1 clone1
563 $ ceph fs clone status cephfs clone1
569 "subvolume": "subvol1",
575 "errstr": "Disk quota exceeded"
580 (NOTE: since ``subvol1`` is in the default group, the ``source`` object's ``clone status`` does not include the group name)
582 .. note:: Cloned subvolumes are accessible only after the clone operation has successfully completed.
584 After a successful clone operation, ``clone status`` will look like the below::
586 $ ceph fs clone status cephfs clone1
593 If a clone operation is unsuccessful, the ``state`` value will be ``failed``.
595 To retry a failed clone operation, the incomplete clone must be deleted and the clone operation must be issued again.
596 To delete a partial clone use::
598 $ ceph fs subvolume rm <vol_name> <clone_name> [--group_name <group_name>] --force
600 .. note:: Cloning synchronizes only directories, regular files and symbolic links. Inode timestamps (access and
601 modification times) are synchronized up to seconds granularity.
603 An ``in-progress`` or a ``pending`` clone operation may be canceled. To cancel a clone operation use the ``clone cancel`` command::
605 $ ceph fs clone cancel <vol_name> <clone_name> [--group_name <group_name>]
607 On successful cancellation, the cloned subvolume is moved to the ``canceled`` state::
609 $ ceph fs subvolume snapshot clone cephfs subvol1 snap1 clone1
610 $ ceph fs clone cancel cephfs clone1
611 $ ceph fs clone status cephfs clone1
617 "subvolume": "subvol1",
623 .. note:: The canceled cloned may be deleted by supplying the ``--force`` option to the `fs subvolume rm` command.
628 Configure the maximum number of concurrent clone operations. The default is 4:
632 ceph config set mgr mgr/volumes/max_concurrent_clones <value>
634 Configure the snapshot_clone_no_wait option :
636 The ``snapshot_clone_no_wait`` config option is used to reject clone creation requests when cloner threads
637 (which can be configured using above option i.e. ``max_concurrent_clones``) are not available.
638 It is enabled by default i.e. the value set is True, whereas it can be configured by using below command.
642 ceph config set mgr mgr/volumes/snapshot_clone_no_wait <bool>
644 The current value of ``snapshot_clone_no_wait`` can be fetched by using below command.
648 ceph config get mgr mgr/volumes/snapshot_clone_no_wait
653 Pinning Subvolumes and Subvolume Groups
654 ---------------------------------------
656 Subvolumes and subvolume groups may be automatically pinned to ranks according
657 to policies. This can distribute load across MDS ranks in predictable and
658 stable ways. Review :ref:`cephfs-pinning` and :ref:`cephfs-ephemeral-pinning`
659 for details on how pinning works.
661 Pinning is configured by::
663 $ ceph fs subvolumegroup pin <vol_name> <group_name> <pin_type> <pin_setting>
667 $ ceph fs subvolume pin <vol_name> <group_name> <pin_type> <pin_setting>
669 Typically you will want to set subvolume group pins. The ``pin_type`` may be
670 one of ``export``, ``distributed``, or ``random``. The ``pin_setting``
671 corresponds to the extended attributed "value" as in the pinning documentation
674 So, for example, setting a distributed pinning strategy on a subvolume group::
676 $ ceph fs subvolumegroup pin cephfilesystem-a csi distributed 1
678 Will enable distributed subtree partitioning policy for the "csi" subvolume
679 group. This will cause every subvolume within the group to be automatically
680 pinned to one of the available ranks on the file system.
683 .. _manila: https://github.com/openstack/manila
684 .. _CSI: https://github.com/ceph/ceph-csi