5 .. versionadded:: Jewel
7 .. note:: Only the NFSv4 protocol is supported when using a cephadm or rook based deployment.
9 Ceph Object Gateway namespaces can be exported over the file-based
10 NFSv4 protocols, alongside traditional HTTP access
11 protocols (S3 and Swift).
13 In particular, the Ceph Object Gateway can now be configured to
14 provide file-based access when embedded in the NFS-Ganesha NFS server.
16 The simplest and preferred way of managing nfs-ganesha clusters and rgw exports
17 is using ``ceph nfs ...`` commands. See :doc:`/mgr/nfs` for more details.
22 The librgw.so shared library (Unix) provides a loadable interface to
23 Ceph Object Gateway services, and instantiates a full Ceph Object Gateway
24 instance on initialization.
26 In turn, librgw.so exports rgw_file, a stateful API for file-oriented
27 access to RGW buckets and objects. The API is general, but its design
28 is strongly influenced by the File System Abstraction Layer (FSAL) API
29 of NFS-Ganesha, for which it has been primarily designed.
31 A set of Python bindings is also provided.
36 The implementation conforms to Amazon Web Services (AWS) hierarchical
37 namespace conventions which map UNIX-style path names onto S3 buckets
40 The top level of the attached namespace consists of S3 buckets,
41 represented as NFS directories. Files and directories subordinate to
42 buckets are each represented as objects, following S3 prefix and
43 delimiter conventions, with '/' being the only supported path
46 For example, if an NFS client has mounted an RGW namespace at "/nfs",
47 then a file "/nfs/mybucket/www/index.html" in the NFS namespace
48 corresponds to an RGW object "www/index.html" in a bucket/container
51 Although it is generally invisible to clients, the NFS namespace is
52 assembled through concatenation of the corresponding paths implied by
53 the objects in the namespace. Leaf objects, whether files or
54 directories, will always be materialized in an RGW object of the
55 corresponding key name, "<name>" if a file, "<name>/" if a directory.
56 Non-leaf directories (e.g., "www" above) might only be implied by
57 their appearance in the names of one or more leaf objects. Directories
58 created within NFS or directly operated on by an NFS client (e.g., via
59 an attribute-setting operation such as chown or chmod) always have a
60 leaf object representation used to store materialized attributes such
61 as Unix ownership and permissions.
66 The RGW NFS interface supports most operations on files and
67 directories, with the following restrictions:
69 - Links, including symlinks, are not supported.
70 - NFS ACLs are not supported.
72 + Unix user and group ownership and permissions *are* supported.
74 - Directories may not be moved/renamed.
76 + Files may be moved between directories.
78 - Only full, sequential *write* I/O is supported
80 + i.e., write operations are constrained to be **uploads**.
81 + Many typical I/O operations such as editing files in place will necessarily fail as they perform non-sequential stores.
82 + Some file utilities *apparently* writing sequentially (e.g., some versions of GNU tar) may fail due to infrequent non-sequential stores.
83 + When mounting via NFS, sequential application I/O can generally be constrained to be written sequentially to the NFS server via a synchronous mount option (e.g. -osync in Linux).
84 + NFS clients which cannot mount synchronously (e.g., MS Windows) will not be able to upload files.
89 The RGW NFS interface provides a hybrid security model with the
90 following characteristics:
92 - NFS protocol security is provided by the NFS-Ganesha server, as negotiated by the NFS server and clients
94 + e.g., clients can by trusted (AUTH_SYS), or required to present Kerberos user credentials (RPCSEC_GSS)
95 + RPCSEC_GSS wire security can be integrity only (krb5i) or integrity and privacy (encryption, krb5p)
96 + various NFS-specific security and permission rules are available
98 * e.g., root-squashing
100 - a set of RGW/S3 security credentials (unknown to NFS) is associated with each RGW NFS mount (i.e., NFS-Ganesha EXPORT)
102 + all RGW object operations performed via the NFS server will be performed by the RGW user associated with the credentials stored in the export being accessed (currently only RGW and RGW LDAP credentials are supported)
104 * additional RGW authentication types such as Keystone are not currently supported
106 Manually configuring an NFS-Ganesha Instance
107 ============================================
109 Each NFS RGW instance is an NFS-Ganesha server instance *embeddding*
110 a full Ceph RGW instance.
112 Therefore, the RGW NFS configuration includes Ceph and Ceph Object
113 Gateway-specific configuration in a local ceph.conf, as well as
114 NFS-Ganesha-specific configuration in the NFS-Ganesha config file,
120 Required ceph.conf configuration for RGW NFS includes:
122 * valid [client.radosgw.{instance-name}] section
123 * valid values for minimal instance configuration, in particular, an installed and correct ``keyring``
125 Other config variables are optional, front-end-specific and front-end
126 selection variables (e.g., ``rgw data`` and ``rgw frontends``) are
127 optional and in some cases ignored.
129 A small number of config variables (e.g., ``rgw_nfs_namespace_expire_secs``)
130 are unique to RGW NFS.
135 A strictly minimal ganesha.conf for use with RGW NFS includes one
136 EXPORT block with embedded FSAL block of type RGW::
140 Export_ID={numeric-id};
146 Transport_Protocols = TCP;
148 # optional, permit unsquashed access by client "root" user
149 #Squash = No_Root_Squash;
153 User_Id = {s3-user-id};
154 Access_Key_Id ="{s3-access-key}";
155 Secret_Access_Key = "{s3-secret}";
159 ``Export_ID`` must have an integer value, e.g., "77"
161 ``Path`` (for RGW) should be "/"
163 ``Pseudo`` defines an NFSv4 pseudo root name (NFSv4 only)
165 ``SecType = sys;`` allows clients to attach without Kerberos
168 ``Squash = No_Root_Squash;`` enables the client root user to override
169 permissions (Unix convention). When root-squashing is enabled,
170 operations attempted by the root user are performed as if by the local
171 "nobody" (and "nogroup") user on the NFS-Ganesha server
173 The RGW FSAL additionally supports RGW-specific configuration
174 variables in the RGW config section::
177 cluster = "{cluster name, default 'ceph'}";
178 name = "client.rgw.{instance-name}";
179 ceph_conf = "/opt/ceph-rgw/etc/ceph/ceph.conf";
180 init_args = "-d --debug-rgw=16";
183 ``cluster`` sets a Ceph cluster name (must match the cluster being exported)
185 ``name`` sets an RGW instance name (must match the cluster being exported)
187 ``ceph_conf`` gives a path to a non-default ceph.conf file to use
190 Other useful NFS-Ganesha configuration:
191 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
193 Any EXPORT block which should support NFSv3 should include version 3
194 in the NFS_Protocols setting. Additionally, NFSv3 is the last major
195 version to support the UDP transport. To enable UDP, include it in the
196 Transport_Protocols setting. For example::
201 Transport_Protocols = UDP,TCP;
205 One important family of options pertains to interaction with the Linux
206 idmapping service, which is used to normalize user and group names
207 across systems. Details of idmapper integration are not provided here.
209 With Linux NFS clients, NFS-Ganesha can be configured
210 to accept client-supplied numeric user and group identifiers with
211 NFSv4, which by default stringifies these--this may be useful in small
212 setups and for experimentation::
215 Allow_Numeric_Owners = true;
216 Only_Numeric_Owners = true;
222 NFS-Ganesha configuration problems are usually debugged by running the
223 server with debugging options, controlled by the LOG config section.
225 NFS-Ganesha log messages are grouped into various components, logging
226 can be enabled separately for each component. Valid values for
227 component logging include::
229 *FATAL* critical errors only
230 *WARN* unusual condition
231 *DEBUG* mildly verbose trace output
232 *FULL_DEBUG* verbose trace output
247 CACHE_INODE_LRU = FATAL;
249 HASHTABLE_CACHE = FATAL;
270 # optional: redirect log output
273 # destination = "/tmp/ganesha-rgw.log";
278 Running Multiple NFS Gateways
279 =============================
281 Each NFS-Ganesha instance acts as a full gateway endpoint, with the
282 limitation that currently an NFS-Ganesha instance cannot be configured
283 to export HTTP services. As with ordinary gateway instances, any
284 number of NFS-Ganesha instances can be started, exporting the same or
285 different resources from the cluster. This enables the clustering of
286 NFS-Ganesha instances. However, this does not imply high availability.
288 When regular gateway instances and NFS-Ganesha instances overlap the
289 same data resources, they will be accessible from both the standard S3
290 API and through the NFS-Ganesha instance as exported. You can
291 co-locate the NFS-Ganesha instance with a Ceph Object Gateway instance
297 Exporting an NFS namespace and other RGW namespaces (e.g., S3 or Swift
298 via the Civetweb HTTP front-end) from the same program instance is
299 currently not supported.
301 When adding objects and buckets outside of NFS, those objects will
302 appear in the NFS namespace in the time set by
303 ``rgw_nfs_namespace_expire_secs``, which defaults to 300 seconds (5 minutes).
304 Override the default value for ``rgw_nfs_namespace_expire_secs`` in the
305 Ceph configuration file to change the refresh rate.
307 If exporting Swift containers that do not conform to valid S3 bucket
308 naming requirements, set ``rgw_relaxed_s3_bucket_names`` to true in the
309 [client.radosgw] section of the Ceph configuration file. For example,
310 if a Swift container name contains underscores, it is not a valid S3
311 bucket name and will be rejected unless ``rgw_relaxed_s3_bucket_names``
314 Configuring NFSv4 clients
315 =========================
317 To access the namespace, mount the configured NFS-Ganesha export(s)
318 into desired locations in the local POSIX namespace. As noted, this
319 implementation has a few unique restrictions:
321 - NFS 4.1 and higher protocol flavors are preferred
323 + NFSv4 OPEN and CLOSE operations are used to track upload transactions
325 - To upload data successfully, clients must preserve write ordering
327 + on Linux and many Unix NFS clients, use the -osync mount option
329 Conventions for mounting NFS resources are platform-specific. The
330 following conventions work on Linux and some Unix platforms:
332 From the command line::
334 mount -t nfs -o nfsvers=4.1,noauto,soft,sync,proto=tcp <ganesha-host-name>:/ <mount-point>
338 <ganesha-host-name>:/ <mount-point> nfs noauto,soft,nfsvers=4.1,sync,proto=tcp 0 0
340 Specify the NFS-Ganesha host name and the path to the mount point on
343 Configuring NFSv3 Clients
344 =========================
346 Linux clients can be configured to mount with NFSv3 by supplying
347 ``nfsvers=3`` and ``noacl`` as mount options. To use UDP as the
348 transport, add ``proto=udp`` to the mount options. However, TCP is the
349 preferred transport::
351 <ganesha-host-name>:/ <mount-point> nfs noauto,noacl,soft,nfsvers=3,sync,proto=tcp 0 0
353 Configure the NFS Ganesha EXPORT block Protocols setting with version
354 3 and the Transports setting with UDP if the mount will use version 3 with UDP.
359 Since NFSv3 does not communicate client OPEN and CLOSE operations to
360 file servers, RGW NFS cannot use these operations to mark the
361 beginning and ending of file upload transactions. Instead, RGW NFS
362 starts a new upload when the first write is sent to a file at offset
363 0, and finalizes the upload when no new writes to the file have been
364 seen for a period of time, by default, 10 seconds. To change this
365 timeout, set an alternate value for ``rgw_nfs_write_completion_interval_s``
366 in the RGW section(s) of the Ceph configuration file.
371 .. [#] http://docs.aws.amazon.com/AmazonS3/latest/dev/ListingKeysHierarchy.html