1 .. _fs-volumes-and-subvolumes:
3 FS volumes and subvolumes
4 =========================
6 A single source of truth for CephFS exports is implemented in the volumes
7 module of the :term:`Ceph Manager` daemon (ceph-mgr). The OpenStack shared
8 file system service (manila_), Ceph Container Storage Interface (CSI_),
9 storage administrators among others can use the common CLI provided by the
10 ceph-mgr volumes module to manage the CephFS exports.
12 The ceph-mgr volumes module implements the following file system export
15 * FS volumes, an abstraction for CephFS file systems
17 * FS subvolumes, an abstraction for independent CephFS directory trees
19 * FS subvolume groups, an abstraction for a directory level higher than FS
20 subvolumes to effect policies (e.g., :doc:`/cephfs/file-layouts`) across a
23 Some possible use-cases for the export abstractions:
25 * FS subvolumes used as manila shares or CSI volumes
27 * FS subvolume groups used as manila share groups
32 * Nautilus (14.2.x) or a later version of Ceph
34 * Cephx client user (see :doc:`/rados/operations/user-management`) with
35 the following minimum capabilities::
44 Create a volume using::
46 $ ceph fs volume create <vol_name> [<placement>]
48 This creates a CephFS file system and its data and metadata pools. It also tries
49 to create MDSes for the filesystem using the enabled ceph-mgr orchestrator
50 module (see :doc:`/mgr/orchestrator`) , e.g., rook.
52 Remove a volume using::
54 $ ceph fs volume rm <vol_name> [--yes-i-really-mean-it]
56 This removes a file system and its data and metadata pools. It also tries to
57 remove MDSes using the enabled ceph-mgr orchestrator module.
66 Create a subvolume group using::
68 $ ceph fs subvolumegroup create <vol_name> <group_name> [--pool_layout <data_pool_name> --uid <uid> --gid <gid> --mode <octal_mode>]
70 The command succeeds even if the subvolume group already exists.
72 When creating a subvolume group you can specify its data pool layout (see
73 :doc:`/cephfs/file-layouts`), uid, gid, and file mode in octal numerals. By default, the
74 subvolume group is created with an octal file mode '755', uid '0', gid '0' and data pool
75 layout of its parent directory.
78 Remove a subvolume group using::
80 $ ceph fs subvolumegroup rm <vol_name> <group_name> [--force]
82 The removal of a subvolume group fails if it is not empty or non-existent.
83 '--force' flag allows the non-existent subvolume group remove command to succeed.
86 Fetch the absolute path of a subvolume group using::
88 $ ceph fs subvolumegroup getpath <vol_name> <group_name>
90 List subvolume groups using::
92 $ ceph fs subvolumegroup ls <vol_name>
94 Create a snapshot (see :doc:`/cephfs/experimental-features`) of a
95 subvolume group using::
97 $ ceph fs subvolumegroup snapshot create <vol_name> <group_name> <snap_name>
99 This implicitly snapshots all the subvolumes under the subvolume group.
101 Remove a snapshot of a subvolume group using::
103 $ ceph fs subvolumegroup snapshot rm <vol_name> <group_name> <snap_name> [--force]
105 Using the '--force' flag allows the command to succeed that would otherwise
106 fail if the snapshot did not exist.
108 List snapshots of a subvolume group using::
110 $ ceph fs subvolumegroup snapshot ls <vol_name> <group_name>
116 Create a subvolume using::
118 $ ceph fs subvolume create <vol_name> <subvol_name> [--size <size_in_bytes> --group_name <subvol_group_name> --pool_layout <data_pool_name> --uid <uid> --gid <gid> --mode <octal_mode> --namespace-isolated]
121 The command succeeds even if the subvolume already exists.
123 When creating a subvolume you can specify its subvolume group, data pool layout,
124 uid, gid, file mode in octal numerals, and size in bytes. The size of the subvolume is
125 specified by setting a quota on it (see :doc:`/cephfs/quota`). The subvolume can be
126 created in a separate RADOS namespace by specifying --namespace-isolated option. By
127 default a subvolume is created within the default subvolume group, and with an octal file
128 mode '755', uid of its subvolume group, gid of its subvolume group, data pool layout of
129 its parent directory and no size limit.
131 Remove a subvolume using::
133 $ ceph fs subvolume rm <vol_name> <subvol_name> [--group_name <subvol_group_name> --force]
136 The command removes the subvolume and its contents. It does this in two steps.
137 First, it move the subvolume to a trash folder, and then asynchronously purges
140 The removal of a subvolume fails if it has snapshots, or is non-existent.
141 '--force' flag allows the non-existent subvolume remove command to succeed.
143 Resize a subvolume using::
145 $ ceph fs subvolume resize <vol_name> <subvol_name> <new_size> [--group_name <subvol_group_name>] [--no_shrink]
147 The command resizes the subvolume quota using the size specified by 'new_size'.
148 '--no_shrink' flag prevents the subvolume to shrink below the current used size of the subvolume.
150 The subvolume can be resized to an infinite size by passing 'inf' or 'infinite' as the new_size.
152 Fetch the absolute path of a subvolume using::
154 $ ceph fs subvolume getpath <vol_name> <subvol_name> [--group_name <subvol_group_name>]
156 Fetch the metadata of a subvolume using::
158 $ ceph fs subvolume info <vol_name> <subvol_name> [--group_name <subvol_group_name>]
160 The output format is json and contains fields as follows.
162 * atime: access time of subvolume path in the format "YYYY-MM-DD HH:MM:SS"
163 * mtime: modification time of subvolume path in the format "YYYY-MM-DD HH:MM:SS"
164 * ctime: change time of subvolume path in the format "YYYY-MM-DD HH:MM:SS"
165 * uid: uid of subvolume path
166 * gid: gid of subvolume path
167 * mode: mode of subvolume path
168 * mon_addrs: list of monitor addresses
169 * bytes_pcent: quota used in percentage if quota is set, else displays "undefined"
170 * bytes_quota: quota size in bytes if quota is set, else displays "infinite"
171 * bytes_used: current used size of the subvolume in bytes
172 * created_at: time of creation of subvolume in the format "YYYY-MM-DD HH:MM:SS"
173 * data_pool: data pool the subvolume belongs to
174 * path: absolute path of a subvolume
175 * type: subvolume type indicating whether it's clone or subvolume
176 * pool_namespace: RADOS namespace of the subvolume
178 List subvolumes using::
180 $ ceph fs subvolume ls <vol_name> [--group_name <subvol_group_name>]
182 Create a snapshot of a subvolume using::
184 $ ceph fs subvolume snapshot create <vol_name> <subvol_name> <snap_name> [--group_name <subvol_group_name>]
187 Remove a snapshot of a subvolume using::
189 $ ceph fs subvolume snapshot rm <vol_name> <subvol_name> <snap_name> [--group_name <subvol_group_name> --force]
191 Using the '--force' flag allows the command to succeed that would otherwise
192 fail if the snapshot did not exist.
194 List snapshots of a subvolume using::
196 $ ceph fs subvolume snapshot ls <vol_name> <subvol_name> [--group_name <subvol_group_name>]
198 Fetch the metadata of a snapshot using::
200 $ ceph fs subvolume snapshot info <vol_name> <subvol_name> <snap_name> [--group_name <subvol_group_name>]
202 The output format is json and contains fields as follows.
204 * created_at: time of creation of snapshot in the format "YYYY-MM-DD HH:MM:SS:ffffff"
205 * data_pool: data pool the snapshot belongs to
206 * has_pending_clones: "yes" if snapshot clone is in progress otherwise "no"
207 * protected: "yes" if snapshot is protected otherwise "no"
208 * size: snapshot size in bytes
213 Subvolumes can be created by cloning subvolume snapshots. Cloning is an asynchronous operation involving copying
214 data from a snapshot to a subvolume. Due to this bulk copy nature, cloning is currently inefficient for very huge
217 Before starting a clone operation, the snapshot should be protected. Protecting a snapshot ensures that the snapshot
218 cannot be deleted when a clone operation is in progress. Snapshots can be protected using::
220 $ ceph fs subvolume snapshot protect <vol_name> <subvol_name> <snap_name> [--group_name <subvol_group_name>]
222 To initiate a clone operation use::
224 $ ceph fs subvolume snapshot clone <vol_name> <subvol_name> <snap_name> <target_subvol_name>
226 If a snapshot (source subvolume) is a part of non-default group, the group name needs to be specified as per::
228 $ ceph fs subvolume snapshot clone <vol_name> <subvol_name> <snap_name> <target_subvol_name> --group_name <subvol_group_name>
230 Cloned subvolumes can be a part of a different group than the source snapshot (by default, cloned subvolumes are created in default group). To clone to a particular group use::
232 $ ceph fs subvolume snapshot clone <vol_name> <subvol_name> <snap_name> <target_subvol_name> --target_group_name <subvol_group_name>
234 Similar to specifying a pool layout when creating a subvolume, pool layout can be specified when creating a cloned subvolume. To create a cloned subvolume with a specific pool layout use::
236 $ ceph fs subvolume snapshot clone <vol_name> <subvol_name> <snap_name> <target_subvol_name> --pool_layout <pool_layout>
238 To check the status of a clone operation use::
240 $ ceph fs clone status <vol_name> <clone_name> [--group_name <group_name>]
242 A clone can be in one of the following states:
244 #. `pending` : Clone operation has not started
245 #. `in-progress` : Clone operation is in progress
246 #. `complete` : Clone operation has sucessfully finished
247 #. `failed` : Clone operation has failed
249 Sample output from an `in-progress` clone operation::
251 $ ceph fs subvolume snapshot protect cephfs subvol1 snap1
252 $ ceph fs subvolume snapshot clone cephfs subvol1 snap1 clone1
253 $ ceph fs clone status cephfs clone1
256 "state": "in-progress",
259 "subvolume": "subvol1",
265 (NOTE: since `subvol1` is in default group, `source` section in `clone status` does not include group name)
267 .. note:: Cloned subvolumes are accessible only after the clone operation has successfully completed.
269 For a successsful clone operation, `clone status` would look like so::
271 $ ceph fs clone status cephfs clone1
278 or `failed` state when clone is unsuccessful.
280 On failure of a clone operation, the partial clone needs to be deleted and the clone operation needs to be retriggered.
281 To delete a partial clone use::
283 $ ceph fs subvolume rm <vol_name> <clone_name> [--group_name <group_name>] --force
285 When no clone operations are in progress or scheduled, the snaphot can be unprotected. To unprotect a snapshot use::
287 $ ceph fs subvolume snapshot unprotect <vol_name> <subvol_name> <snap_name> [--group_name <subvol_group_name>]
289 Note that unprotecting a snapshot would fail if there are pending or in progress clone operations. Also note that,
290 only unprotected snapshots can be removed. This guarantees that a snapshot cannot be deleted when clones are pending
293 .. note:: Cloning only synchronizes directories, regular files and symbolic links. Also, inode timestamps (access and
294 modification times) are synchronized upto seconds granularity.
296 An `in-progress` or a `pending` clone operation can be canceled. To cancel a clone operation use the `clone cancel` command::
298 $ ceph fs clone cancel <vol_name> <clone_name> [--group_name <group_name>]
300 On successful cancelation, the cloned subvolume is moved to `canceled` state::
302 $ ceph fs subvolume snapshot protect cephfs subvol1 snap1
303 $ ceph fs subvolume snapshot clone cephfs subvol1 snap1 clone1
304 $ ceph fs clone cancel cephfs clone1
305 $ ceph fs clone status cephfs clone1
311 "subvolume": "subvol1",
317 .. note:: The canceled cloned can be deleted by using --force option in `fs subvolume rm` command.
319 .. _manila: https://github.com/openstack/manila
320 .. _CSI: https://github.com/ceph/ceph-csi