]>
Commit | Line | Data |
---|---|---|
7c673cae FG |
1 | Quotas |
2 | ====== | |
3 | ||
4 | CephFS allows quotas to be set on any directory in the system. The | |
5 | quota can restrict the number of *bytes* or the number of *files* | |
6 | stored beneath that point in the directory hierarchy. | |
7 | ||
39ae355f TL |
8 | Like most other things in CephFS, quotas are configured using virtual |
9 | extended attributes: | |
10 | ||
11 | * ``ceph.quota.max_files`` -- file limit | |
12 | * ``ceph.quota.max_bytes`` -- byte limit | |
13 | ||
14 | If the extended attributes appear on a directory that means a quota is | |
15 | configured there. If they are not present then no quota is set on that | |
16 | directory (although one may still be configured on a parent directory). | |
17 | ||
18 | To set a quota, set the extended attribute on a CephFS directory with a | |
19 | value:: | |
20 | ||
21 | setfattr -n ceph.quota.max_bytes -v 100000000 /some/dir # 100 MB | |
22 | setfattr -n ceph.quota.max_files -v 10000 /some/dir # 10,000 files | |
23 | ||
24 | To view quota limit:: | |
25 | ||
26 | $ getfattr -n ceph.quota.max_bytes /some/dir | |
27 | # file: dir1/ | |
28 | ceph.quota.max_bytes="100000000" | |
29 | $ | |
30 | $ getfattr -n ceph.quota.max_files /some/dir | |
31 | # file: dir1/ | |
32 | ceph.quota.max_files="10000" | |
33 | ||
34 | .. note:: Running ``getfattr /some/dir -d -m -`` for a CephFS directory will | |
35 | print none of the CephFS extended attributes. This is because the CephFS | |
36 | kernel and FUSE clients hide this information from the ``listxattr(2)`` | |
37 | system call. Instead, a specific CephFS extended attribute can be viewed by | |
38 | running ``getfattr /some/dir -n ceph.<some-xattr>``. | |
39 | ||
40 | To remove a quota, set the value of extended attribute to ``0``:: | |
41 | ||
42 | $ setfattr -n ceph.quota.max_bytes -v 0 /some/dir | |
43 | $ getfattr /some/dir -n ceph.quota.max_bytes | |
44 | dir1/: ceph.quota.max_bytes: No such attribute | |
45 | $ | |
46 | $ setfattr -n ceph.quota.max_files -v 0 /some/dir | |
47 | $ getfattr dir1/ -n ceph.quota.max_files | |
48 | dir1/: ceph.quota.max_files: No such attribute | |
49 | ||
50 | Space Usage Reporting and CephFS Quotas | |
51 | --------------------------------------- | |
52 | When the root directory of the CephFS mount has quota set on it, the available | |
53 | space on the CephFS reported by space usage report tools (like ``df``) is | |
54 | based on quota limit. That is, ``available space = quota limit - used space`` | |
55 | instead of ``available space = total space - used space``. | |
56 | ||
57 | This behaviour can be disabled by setting following option in client section | |
58 | of ``ceph.conf``:: | |
59 | ||
60 | client quota df = false | |
61 | ||
7c673cae FG |
62 | Limitations |
63 | ----------- | |
64 | ||
65 | #. *Quotas are cooperative and non-adversarial.* CephFS quotas rely on | |
66 | the cooperation of the client who is mounting the file system to | |
67 | stop writers when a limit is reached. A modified or adversarial | |
68 | client cannot be prevented from writing as much data as it needs. | |
69 | Quotas should not be relied on to prevent filling the system in | |
70 | environments where the clients are fully untrusted. | |
71 | ||
72 | #. *Quotas are imprecise.* Processes that are writing to the file | |
73 | system will be stopped a short time after the quota limit is | |
74 | reached. They will inevitably be allowed to write some amount of | |
75 | data over the configured limit. How far over the quota they are | |
76 | able to go depends primarily on the amount of time, not the amount | |
77 | of data. Generally speaking writers will be stopped within 10s of | |
78 | seconds of crossing the configured limit. | |
79 | ||
11fdf7f2 TL |
80 | #. *Quotas are implemented in the kernel client 4.17 and higher.* |
81 | Quotas are supported by the userspace client (libcephfs, ceph-fuse). | |
82 | Linux kernel clients >= 4.17 support CephFS quotas but only on | |
83 | mimic+ clusters. Kernel clients (even recent versions) will fail | |
84 | to handle quotas on older clusters, even if they may be able to set | |
85 | the quotas extended attributes. | |
7c673cae FG |
86 | |
87 | #. *Quotas must be configured carefully when used with path-based | |
88 | mount restrictions.* The client needs to have access to the | |
89 | directory inode on which quotas are configured in order to enforce | |
90 | them. If the client has restricted access to a specific path | |
91 | (e.g., ``/home/user``) based on the MDS capability, and a quota is | |
92 | configured on an ancestor directory they do not have access to | |
93 | (e.g., ``/home``), the client will not enforce it. When using | |
94 | path-based access restrictions be sure to configure the quota on | |
95 | the directory the client is restricted too (e.g., ``/home/user``) | |
96 | or something nested beneath it. | |
97 | ||
2a845540 TL |
98 | In case of a kernel client, it needs to have access to the parent |
99 | of the directory inode on which quotas are configured in order to | |
100 | enforce them. If quota is configured on a directory path | |
101 | (e.g., ``/home/volumes/group``), the kclient needs to have access | |
102 | to the parent (e.g., ``/home/volumes``). | |
103 | ||
104 | An example command to create such an user is as below:: | |
105 | ||
106 | $ ceph auth get-or-create client.guest mds 'allow r path=/home/volumes, allow rw path=/home/volumes/group' mgr 'allow rw' osd 'allow rw tag cephfs metadata=*' mon 'allow r' | |
107 | ||
108 | See also: https://tracker.ceph.com/issues/55090 | |
109 | ||
11fdf7f2 TL |
110 | #. *Snapshot file data which has since been deleted or changed does not count |
111 | towards the quota.* See also: http://tracker.ceph.com/issues/24284 | |
112 | ||
7c673cae FG |
113 | Configuration |
114 | ------------- | |
115 | ||
116 | Like most other things in CephFS, quotas are configured using virtual | |
117 | extended attributes: | |
118 | ||
119 | * ``ceph.quota.max_files`` -- file limit | |
120 | * ``ceph.quota.max_bytes`` -- byte limit | |
121 | ||
122 | If the attributes appear on a directory inode that means a quota is | |
123 | configured there. If they are not present then no quota is set on | |
124 | that directory (although one may still be configured on a parent directory). | |
125 | ||
126 | To set a quota:: | |
127 | ||
128 | setfattr -n ceph.quota.max_bytes -v 100000000 /some/dir # 100 MB | |
129 | setfattr -n ceph.quota.max_files -v 10000 /some/dir # 10,000 files | |
130 | ||
131 | To view quota settings:: | |
132 | ||
133 | getfattr -n ceph.quota.max_bytes /some/dir | |
134 | getfattr -n ceph.quota.max_files /some/dir | |
135 | ||
136 | Note that if the value of the extended attribute is ``0`` that means | |
137 | the quota is not set. | |
138 | ||
139 | To remove a quota:: | |
140 | ||
141 | setfattr -n ceph.quota.max_bytes -v 0 /some/dir | |
142 | setfattr -n ceph.quota.max_files -v 0 /some/dir | |
39ae355f TL |
143 | |
144 | ||
145 | .. note:: In cases where CephFS extended attributes are set on a CephFS | |
146 | directory (for example, ``/some/dir``), running ``getfattr /some/dir -d -m | |
147 | -`` will not print those CephFS extended attributes. This is because CephFS | |
148 | kernel and FUSE clients hide this information from the ``listxattr(2)`` | |
149 | system call. You can access a specific CephFS extended attribute by running | |
150 | ``getfattr /some/dir -n ceph.<some-xattr>`` instead. |