ceph/doc/radosgw/nfs.rst

   1 ===
   2 NFS
   3 ===
   4
   5 .. versionadded:: Jewel
   6
   7 .. note:: Only the NFSv4 protocol is supported when using a cephadm or rook based deployment.
   8
   9 Ceph Object Gateway namespaces can be exported over the file-based
  10 NFSv4 protocols, alongside traditional HTTP access
  11 protocols (S3 and Swift).
  12
  13 In particular, the Ceph Object Gateway can now be configured to
  14 provide file-based access when embedded in the NFS-Ganesha NFS server.
  15
  16 The simplest and preferred way of managing nfs-ganesha clusters and rgw exports
  17 is using ``ceph nfs ...`` commands. See :doc:`/mgr/nfs` for more details.
  18
  19 librgw
  20 ======
  21
  22 The librgw.so shared library (Unix) provides a loadable interface to
  23 Ceph Object Gateway services, and instantiates a full Ceph Object Gateway
  24 instance on initialization.
  25
  26 In turn, librgw.so exports rgw_file, a stateful API for file-oriented
  27 access to RGW buckets and objects.  The API is general, but its design
  28 is strongly influenced by the File System Abstraction Layer (FSAL) API
  29 of NFS-Ganesha, for which it has been primarily designed.
  30
  31 A set of Python bindings is also provided.
  32
  33 Namespace Conventions
  34 =====================
  35
  36 The implementation conforms to Amazon Web Services (AWS) hierarchical
  37 namespace conventions which map UNIX-style path names onto S3 buckets
  38 and objects.
  39
  40 The top level of the attached namespace consists of S3 buckets,
  41 represented as NFS directories. Files and directories subordinate to
  42 buckets are each represented as objects, following S3 prefix and
  43 delimiter conventions, with '/' being the only supported path
  44 delimiter [#]_.
  45
  46 For example, if an NFS client has mounted an RGW namespace at "/nfs",
  47 then a file "/nfs/mybucket/www/index.html" in the NFS namespace
  48 corresponds to an RGW object "www/index.html" in a bucket/container
  49 "mybucket."
  50
  51 Although it is generally invisible to clients, the NFS namespace is
  52 assembled through concatenation of the corresponding paths implied by
  53 the objects in the namespace.  Leaf objects, whether files or
  54 directories, will always be materialized in an RGW object of the
  55 corresponding key name, "<name>" if a file, "<name>/" if a directory.
  56 Non-leaf directories (e.g., "www" above) might only be implied by
  57 their appearance in the names of one or more leaf objects. Directories
  58 created within NFS or directly operated on by an NFS client (e.g., via
  59 an attribute-setting operation such as chown or chmod) always have a
  60 leaf object representation used to store materialized attributes such
  61 as Unix ownership and permissions.
  62
  63 Supported Operations
  64 ====================
  65
  66 The RGW NFS interface supports most operations on files and
  67 directories, with the following restrictions:
  68
  69 - Links, including symlinks, are not supported.
  70 - NFS ACLs are not supported.
  71
  72   + Unix user and group ownership and permissions *are* supported.
  73
  74 - Directories may not be moved/renamed.
  75
  76   + Files may be moved between directories.
  77
  78 - Only full, sequential *write* I/O is supported
  79
  80   + i.e., write operations are constrained to be **uploads**.
  81   + Many typical I/O operations such as editing files in place will necessarily fail as they perform non-sequential stores.
  82   + Some file utilities *apparently* writing sequentially (e.g., some versions of GNU tar) may fail due to infrequent non-sequential stores.
  83   + When mounting via NFS, sequential application I/O can generally be constrained to be written sequentially to the NFS server via a synchronous mount option (e.g. -osync in Linux).
  84   + NFS clients which cannot mount synchronously (e.g., MS Windows) will not be able to upload files.
  85
  86 Security
  87 ========
  88
  89 The RGW NFS interface provides a hybrid security model with the
  90 following characteristics:
  91
  92 - NFS protocol security is provided by the NFS-Ganesha server, as negotiated by the NFS server and clients
  93
  94   + e.g., clients can by trusted (AUTH_SYS), or required to present Kerberos user credentials (RPCSEC_GSS)
  95   + RPCSEC_GSS wire security can be integrity only (krb5i) or integrity and privacy (encryption, krb5p)
  96   + various NFS-specific security and permission rules are available
  97
  98     * e.g., root-squashing
  99
 100 - a set of RGW/S3 security credentials (unknown to NFS) is associated with each RGW NFS mount (i.e., NFS-Ganesha EXPORT)
 101
 102   + all RGW object operations performed via the NFS server will be performed by the RGW user associated with the credentials stored in the export being accessed (currently only RGW and RGW LDAP credentials are supported)
 103
 104     * additional RGW authentication types such as Keystone are not currently supported
 105
 106 Manually configuring an NFS-Ganesha Instance
 107 ============================================
 108
 109 Each NFS RGW instance is an NFS-Ganesha server instance *embedding*
 110 a full Ceph RGW instance.
 111
 112 Therefore, the RGW NFS configuration includes Ceph and Ceph Object
 113 Gateway-specific configuration in a local ceph.conf, as well as
 114 NFS-Ganesha-specific configuration in the NFS-Ganesha config file,
 115 ganesha.conf.
 116
 117 ceph.conf
 118 ---------
 119
 120 Required ceph.conf configuration for RGW NFS includes:
 121
 122 * valid [client.rgw.{instance-name}] section
 123 * valid values for minimal instance configuration, in particular, an installed and correct ``keyring``
 124
 125 Other config variables (e.g., ``rgw data`` and ``rgw backend store``) are
 126 optional.
 127
 128 A small number of config variables (e.g., ``rgw_nfs_namespace_expire_secs``)
 129 are unique to RGW NFS.
 130
 131 In particular, front-end selection is handled specially by the librgw.so runtime.  By default, only the
 132 ``rgw-nfs`` frontend is started. Additional frontends (e.g., ``beast``) are enabled via the
 133 ``rgw nfs frontends`` config option.  Its syntax is identical to the ordinary ``rgw frontends`` option.
 134 Default options for non-default frontends are specified via ``rgw frontend defaults`` as normal.
 135
 136 ganesha.conf
 137 ------------
 138
 139 A strictly minimal ganesha.conf for use with RGW NFS includes one
 140 EXPORT block with embedded FSAL block of type RGW::
 141
 142    EXPORT
 143    {
 144         Export_ID={numeric-id};
 145         Path = "/";
 146         Pseudo = "/";
 147         Access_Type = RW;
 148         SecType = "sys";
 149         NFS_Protocols = 4;
 150         Transport_Protocols = TCP;
 151
 152         # optional, permit unsquashed access by client "root" user
 153         #Squash = No_Root_Squash;
 154
 155         FSAL {
 156                 Name = RGW;
 157                 User_Id = {s3-user-id};
 158                 Access_Key_Id ="{s3-access-key}";
 159                 Secret_Access_Key = "{s3-secret}";
 160         }
 161    }
 162
 163 ``Export_ID`` must have an integer value, e.g., "77"
 164
 165 ``Path`` (for RGW) should be "/"
 166
 167 ``Pseudo`` defines an NFSv4 pseudo root name (NFSv4 only)
 168
 169 ``SecType = sys;`` allows clients to attach without Kerberos
 170 authentication
 171
 172 ``Squash = No_Root_Squash;`` enables the client root user to override
 173 permissions (Unix convention).  When root-squashing is enabled,
 174 operations attempted by the root user are performed as if by the local
 175 "nobody" (and "nogroup") user on the NFS-Ganesha server
 176
 177 The RGW FSAL additionally supports RGW-specific configuration
 178 variables in the RGW config section::
 179
 180  RGW {
 181      cluster = "{cluster name, default 'ceph'}";
 182      name = "client.rgw.{instance-name}";
 183      ceph_conf = "/opt/ceph-rgw/etc/ceph/ceph.conf";
 184      init_args = "-d --debug-rgw=16";
 185  }
 186
 187 ``cluster`` sets a Ceph cluster name (must match the cluster being exported)
 188
 189 ``name`` sets an RGW instance name (must match the cluster being exported)
 190
 191 ``ceph_conf`` gives a path to a non-default ceph.conf file to use
 192
 193
 194 Other useful NFS-Ganesha configuration:
 195 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 196
 197 Any EXPORT block which should support NFSv3 should include version 3
 198 in the NFS_Protocols setting. Additionally, NFSv3 is the last major
 199 version to support the UDP transport. To enable UDP, include it in the
 200 Transport_Protocols setting. For example::
 201
 202  EXPORT {
 203   ...
 204     NFS_Protocols = 3,4;
 205     Transport_Protocols = UDP,TCP;
 206   ...
 207  }
 208
 209 One important family of options pertains to interaction with the Linux
 210 idmapping service, which is used to normalize user and group names
 211 across systems.  Details of idmapper integration are not provided here.
 212
 213 With Linux NFS clients, NFS-Ganesha can be configured
 214 to accept client-supplied numeric user and group identifiers with
 215 NFSv4, which by default stringifies these--this may be useful in small
 216 setups and for experimentation::
 217
 218  NFSV4 {
 219      Allow_Numeric_Owners = true;
 220      Only_Numeric_Owners = true;
 221  }
 222
 223 Troubleshooting
 224 ~~~~~~~~~~~~~~~
 225
 226 NFS-Ganesha configuration problems are usually debugged by running the
 227 server with debugging options, controlled by the LOG config section.
 228
 229 NFS-Ganesha log messages are grouped into various components, logging
 230 can be enabled separately for each component. Valid values for
 231 component logging include::
 232
 233   *FATAL* critical errors only
 234   *WARN* unusual condition
 235   *DEBUG* mildly verbose trace output
 236   *FULL_DEBUG* verbose trace output
 237
 238 Example::
 239
 240   LOG {
 241
 242         Components {
 243                 MEMLEAKS = FATAL;
 244                 FSAL = FATAL;
 245                 NFSPROTO = FATAL;
 246                 NFS_V4 = FATAL;
 247                 EXPORT = FATAL;
 248                 FILEHANDLE = FATAL;
 249                 DISPATCH = FATAL;
 250                 CACHE_INODE = FATAL;
 251                 CACHE_INODE_LRU = FATAL;
 252                 HASHTABLE = FATAL;
 253                 HASHTABLE_CACHE = FATAL;
 254                 DUPREQ = FATAL;
 255                 INIT = DEBUG;
 256                 MAIN = DEBUG;
 257                 IDMAPPER = FATAL;
 258                 NFS_READDIR = FATAL;
 259                 NFS_V4_LOCK = FATAL;
 260                 CONFIG = FATAL;
 261                 CLIENTID = FATAL;
 262                 SESSIONS = FATAL;
 263                 PNFS = FATAL;
 264                 RW_LOCK = FATAL;
 265                 NLM = FATAL;
 266                 RPC = FATAL;
 267                 NFS_CB = FATAL;
 268                 THREAD = FATAL;
 269                 NFS_V4_ACL = FATAL;
 270                 STATE = FATAL;
 271                 FSAL_UP = FATAL;
 272                 DBUS = FATAL;
 273         }
 274         # optional: redirect log output
 275  #      Facility {
 276  #              name = FILE;
 277  #              destination = "/tmp/ganesha-rgw.log";
 278  #              enable = active;
 279         }
 280  }
 281
 282 Running Multiple NFS Gateways
 283 =============================
 284
 285 Each NFS-Ganesha instance acts as a full gateway endpoint, with the
 286 limitation that currently an NFS-Ganesha instance cannot be configured
 287 to export HTTP services. As with ordinary gateway instances, any
 288 number of NFS-Ganesha instances can be started, exporting the same or
 289 different resources from the cluster. This enables the clustering of
 290 NFS-Ganesha instances. However, this does not imply high availability.
 291
 292 When regular gateway instances and NFS-Ganesha instances overlap the
 293 same data resources, they will be accessible from both the standard S3
 294 API and through the NFS-Ganesha instance as exported. You can
 295 co-locate the NFS-Ganesha instance with a Ceph Object Gateway instance
 296 on the same host.
 297
 298 RGW vs RGW NFS
 299 ==============
 300
 301 Exporting an NFS namespace and other RGW namespaces (e.g., S3 or Swift
 302 via the Civetweb HTTP front-end) from the same program instance is
 303 currently not supported.
 304
 305 When adding objects and buckets outside of NFS, those objects will
 306 appear in the NFS namespace in the time set by
 307 ``rgw_nfs_namespace_expire_secs``, which defaults to 300 seconds (5 minutes).
 308 Override the default value for ``rgw_nfs_namespace_expire_secs`` in the
 309 Ceph configuration file to change the refresh rate.
 310
 311 If exporting Swift containers that do not conform to valid S3 bucket
 312 naming requirements, set ``rgw_relaxed_s3_bucket_names`` to true in the
 313 [client.rgw] section of the Ceph configuration file. For example,
 314 if a Swift container name contains underscores, it is not a valid S3
 315 bucket name and will be rejected unless ``rgw_relaxed_s3_bucket_names``
 316 is set to true.
 317
 318 Configuring NFSv4 clients
 319 =========================
 320
 321 To access the namespace, mount the configured NFS-Ganesha export(s)
 322 into desired locations in the local POSIX namespace. As noted, this
 323 implementation has a few unique restrictions:
 324
 325 - NFS 4.1 and higher protocol flavors are preferred
 326
 327   + NFSv4 OPEN and CLOSE operations are used to track upload transactions
 328
 329 - To upload data successfully, clients must preserve write ordering
 330
 331   + on Linux and many Unix NFS clients, use the -osync mount option
 332
 333 Conventions for mounting NFS resources are platform-specific. The
 334 following conventions work on Linux and some Unix platforms:
 335
 336 From the command line::
 337
 338   mount -t nfs -o nfsvers=4.1,noauto,soft,sync,proto=tcp <ganesha-host-name>:/ <mount-point>
 339
 340 In /etc/fstab::
 341
 342 <ganesha-host-name>:/ <mount-point> nfs noauto,soft,nfsvers=4.1,sync,proto=tcp 0 0
 343
 344 Specify the NFS-Ganesha host name and the path to the mount point on
 345 the client.
 346
 347 Configuring NFSv3 Clients
 348 =========================
 349
 350 Linux clients can be configured to mount with NFSv3 by supplying
 351 ``nfsvers=3`` and ``noacl`` as mount options. To use UDP as the
 352 transport, add ``proto=udp`` to the mount options. However, TCP is the
 353 preferred transport::
 354
 355   <ganesha-host-name>:/ <mount-point> nfs noauto,noacl,soft,nfsvers=3,sync,proto=tcp 0 0
 356
 357 Configure the NFS Ganesha EXPORT block Protocols setting with version
 358 3 and the Transports setting with UDP if the mount will use version 3 with UDP.
 359
 360 NFSv3 Semantics
 361 ---------------
 362
 363 Since NFSv3 does not communicate client OPEN and CLOSE operations to
 364 file servers, RGW NFS cannot use these operations to mark the
 365 beginning and ending of file upload transactions. Instead, RGW NFS
 366 starts a new upload when the first write is sent to a file at offset
 367 0, and finalizes the upload when no new writes to the file have been
 368 seen for a period of time, by default, 10 seconds. To change this
 369 timeout, set an alternate value for ``rgw_nfs_write_completion_interval_s``
 370 in the RGW section(s) of the Ceph configuration file.
 371
 372 References
 373 ==========
 374
 375 .. [#] http://docs.aws.amazon.com/AmazonS3/latest/dev/ListingKeysHierarchy.html