ceph/doc/mgr/nfs.rst

   1 .. _mgr-nfs:
   2
   3 =============================
   4 CephFS & RGW Exports over NFS
   5 =============================
   6
   7 CephFS namespaces and RGW buckets can be exported over NFS protocol
   8 using the `NFS-Ganesha NFS server`_.
   9
  10 The ``nfs`` manager module provides a general interface for managing
  11 NFS exports of either CephFS directories or RGW buckets.  Exports can
  12 be managed either via the CLI ``ceph nfs export ...`` commands
  13 or via the dashboard.
  14
  15 The deployment of the nfs-ganesha daemons can also be managed
  16 automatically if either the :ref:`cephadm` or :ref:`mgr-rook`
  17 orchestrators are enabled.  If neither are in use (e.g., Ceph is
  18 deployed via an external orchestrator like Ansible or Puppet), the
  19 nfs-ganesha daemons must be manually deployed; for more information,
  20 see :ref:`nfs-ganesha-config`.
  21
  22 .. note:: Starting with Ceph Pacific, the ``nfs`` mgr module must be enabled.
  23
  24 NFS Cluster management
  25 ======================
  26
  27 .. _nfs-module-cluster-create:
  28
  29 Create NFS Ganesha Cluster
  30 --------------------------
  31
  32 .. code:: bash
  33
  34     $ ceph nfs cluster create <cluster_id> [<placement>] [--ingress] [--virtual_ip <value>] [--ingress-mode {default|keepalive-only|haproxy-standard|haproxy-protocol}] [--port <int>]
  35
  36 This creates a common recovery pool for all NFS Ganesha daemons, new user based on
  37 ``cluster_id``, and a common NFS Ganesha config RADOS object.
  38
  39 .. note:: Since this command also brings up NFS Ganesha daemons using a ceph-mgr
  40    orchestrator module (see :doc:`/mgr/orchestrator`) such as cephadm or rook, at
  41    least one such module must be enabled for it to work.
  42
  43    Currently, NFS Ganesha daemon deployed by cephadm listens on the standard
  44    port. So only one daemon will be deployed on a host.
  45
  46 ``<cluster_id>`` is an arbitrary string by which this NFS Ganesha cluster will be
  47 known (e.g., ``mynfs``).
  48
  49 ``<placement>`` is an optional string signifying which hosts should have NFS Ganesha
  50 daemon containers running on them and, optionally, the total number of NFS
  51 Ganesha daemons on the cluster (should you want to have more than one NFS Ganesha
  52 daemon running per node). For example, the following placement string means
  53 "deploy NFS Ganesha daemons on nodes host1 and host2 (one daemon per host)::
  54
  55     "host1,host2"
  56
  57 and this placement specification says to deploy single NFS Ganesha daemon each
  58 on nodes host1 and host2 (for a total of two NFS Ganesha daemons in the
  59 cluster)::
  60
  61     "2 host1,host2"
  62
  63 NFS can be deployed on a port other than 2049 (the default) with ``--port <port>``.
  64
  65 To deploy NFS with a high-availability front-end (virtual IP and load balancer), add the
  66 ``--ingress`` flag and specify a virtual IP address. This will deploy a combination
  67 of keepalived and haproxy to provide an high-availability NFS frontend for the NFS
  68 service.
  69
  70 .. note:: The ingress implementation is not yet complete.  Enabling
  71           ingress will deploy multiple ganesha instances and balance
  72           load across them, but a host failure will not immediately
  73           cause cephadm to deploy a replacement daemon before the NFS
  74           grace period expires.  This high-availability functionality
  75           is expected to be completed by the Quincy release (March
  76           2022).
  77
  78 For more details, refer :ref:`orchestrator-cli-placement-spec` but keep
  79 in mind that specifying the placement via a YAML file is not supported.
  80
  81 Deployment of NFS daemons and the ingress service is asynchronous: the
  82 command may return before the services have completely started. You may
  83 wish to check that these services do successfully start and stay running.
  84 When using cephadm orchestration, these commands check service status:
  85
  86 .. code:: bash
  87
  88     $ ceph orch ls --service_name=nfs.<cluster_id>
  89     $ ceph orch ls --service_name=ingress.nfs.<cluster_id>
  90
  91
  92 Ingress
  93 -------
  94
  95 The core *nfs* service will deploy one or more nfs-ganesha daemons,
  96 each of which will provide a working NFS endpoint.  The IP for each
  97 NFS endpoint will depend on which host the nfs-ganesha daemons are
  98 deployed.  By default, daemons are placed semi-randomly, but users can
  99 also explicitly control where daemons are placed; see
 100 :ref:`orchestrator-cli-placement-spec`.
 101
 102 When a cluster is created with ``--ingress``, an *ingress* service is
 103 additionally deployed to provide load balancing and high-availability
 104 for the NFS servers.  A virtual IP is used to provide a known, stable
 105 NFS endpoint that all clients can use to mount.  Ceph will take care
 106 of the details of NFS redirecting traffic on the virtual IP to the
 107 appropriate backend NFS servers, and redeploying NFS servers when they
 108 fail.
 109
 110 An optional ``--ingress-mode`` parameter can be provided to choose
 111 how the *ingress* service is configured:
 112
 113 - Setting ``--ingress-mode keepalive-only`` deploys a simplified *ingress*
 114   service that provides a virtual IP with the nfs server directly binding to
 115   that virtual IP and leaves out any sort of load balancing or traffic
 116   redirection. This setup will restrict users to deploying only 1 nfs daemon
 117   as multiple cannot bind to the same port on the virtual IP.
 118 - Setting ``--ingress-mode haproxy-standard`` deploys a full *ingress* service
 119   to provide load balancing and high-availability using HAProxy and keepalived.
 120   Client IP addresses are not visible to the back-end NFS server and IP level
 121   restrictions on NFS exports will not function.
 122 - Setting ``--ingress-mode haproxy-protocol`` deploys a full *ingress* service
 123   to provide load balancing and high-availability using HAProxy and keepalived.
 124   Client IP addresses are visible to the back-end NFS server and IP level
 125   restrictions on NFS exports are usable. This mode requires NFS Ganesha version
 126   5.0 or later.
 127 - Setting ``--ingress-mode default`` is equivalent to not providing any other
 128   ingress mode by name. When no other ingress mode is specified by name
 129   the default ingress mode used is ``haproxy-standard``.
 130
 131 Ingress can be added to an existing NFS service (e.g., one initially created
 132 without the ``--ingress`` flag), and the basic NFS service can
 133 also be modified after the fact to include non-default options, by modifying
 134 the services directly.  For more information, see :ref:`cephadm-ha-nfs`.
 135
 136 Show NFS Cluster IP(s)
 137 ----------------------
 138
 139 To examine an NFS cluster's IP endpoints, including the IPs for the individual NFS
 140 daemons, and the virtual IP (if any) for the ingress service,
 141
 142 .. code:: bash
 143
 144     $ ceph nfs cluster info [<cluster_id>]
 145
 146 .. note:: This will not work with the rook backend. Instead, expose the port with
 147    the kubectl patch command and fetch the port details with kubectl get services
 148    command::
 149
 150     $ kubectl patch service -n rook-ceph -p '{"spec":{"type": "NodePort"}}' rook-ceph-nfs-<cluster-name>-<node-id>
 151     $ kubectl get services -n rook-ceph rook-ceph-nfs-<cluster-name>-<node-id>
 152
 153
 154 Delete NFS Ganesha Cluster
 155 --------------------------
 156
 157 .. code:: bash
 158
 159     $ ceph nfs cluster rm <cluster_id>
 160
 161 This deletes the deployed cluster.
 162
 163
 164 Removal of NFS daemons and the ingress service is asynchronous: the
 165 command may return before the services have been completely deleted. You may
 166 wish to check that these services are no longer reported. When using cephadm
 167 orchestration, these commands check service status:
 168
 169 .. code:: bash
 170
 171     $ ceph orch ls --service_name=nfs.<cluster_id>
 172     $ ceph orch ls --service_name=ingress.nfs.<cluster_id>
 173
 174
 175 Updating an NFS Cluster
 176 -----------------------
 177
 178 In order to modify cluster parameters (like the port or placement), you need to
 179 use the orchestrator interface to update the NFS service spec.  The safest way to do
 180 that is to export the current spec, modify it, and then re-apply it.  For example,
 181 to modify the ``nfs.foo`` service,
 182
 183 .. code:: bash
 184
 185     $ ceph orch ls --service-name nfs.foo --export > nfs.foo.yaml
 186     $ vi nfs.foo.yaml
 187     $ ceph orch apply -i nfs.foo.yaml
 188
 189 For more information about the NFS service spec, see :ref:`deploy-cephadm-nfs-ganesha`.
 190
 191 List NFS Ganesha Clusters
 192 -------------------------
 193
 194 .. code:: bash
 195
 196     $ ceph nfs cluster ls
 197
 198 This lists deployed clusters.
 199
 200 .. _nfs-cluster-set:
 201
 202 Set Customized NFS Ganesha Configuration
 203 ----------------------------------------
 204
 205 .. code:: bash
 206
 207     $ ceph nfs cluster config set <cluster_id> -i <config_file>
 208
 209 With this the nfs cluster will use the specified config and it will have
 210 precedence over default config blocks.
 211
 212 Example use cases include:
 213
 214 #. Changing log level.  The logging level can be adjusted with the following config
 215    fragment::
 216
 217      LOG {
 218          COMPONENTS {
 219              ALL = FULL_DEBUG;
 220          }
 221      }
 222
 223 #. Adding custom export block.
 224
 225    The following sample block creates a single export. This export will not be
 226    managed by `ceph nfs export` interface::
 227
 228     EXPORT {
 229       Export_Id = 100;
 230       Transports = TCP;
 231       Path = /;
 232       Pseudo = /ceph/;
 233       Protocols = 4;
 234       Access_Type = RW;
 235       Attr_Expiration_Time = 0;
 236       Squash = None;
 237       FSAL {
 238         Name = CEPH;
 239         Filesystem = "filesystem name";
 240         User_Id = "user id";
 241         Secret_Access_Key = "secret key";
 242       }
 243     }
 244
 245 .. note:: User specified in FSAL block should have proper caps for NFS-Ganesha
 246    daemons to access ceph cluster. User can be created in following way using
 247    `auth get-or-create`::
 248
 249          # ceph auth get-or-create client.<user_id> mon 'allow r' osd 'allow rw pool=.nfs namespace=<nfs_cluster_name>, allow rw tag cephfs data=<fs_name>' mds 'allow rw path=<export_path>'
 250
 251 View Customized NFS Ganesha Configuration
 252 -----------------------------------------
 253
 254 .. code:: bash
 255
 256     $ ceph nfs cluster config get <cluster_id>
 257
 258 This will output the user defined configuration (if any).
 259
 260 Reset NFS Ganesha Configuration
 261 -------------------------------
 262
 263 .. code:: bash
 264
 265     $ ceph nfs cluster config reset <cluster_id>
 266
 267 This removes the user defined configuration.
 268
 269 .. note:: With a rook deployment, ganesha pods must be explicitly restarted
 270    for the new config blocks to be effective.
 271
 272
 273 Export Management
 274 =================
 275
 276 .. warning:: Currently, the nfs interface is not integrated with dashboard. Both
 277    dashboard and nfs interface have different export requirements and
 278    create exports differently. Management of dashboard created exports is not
 279    supported.
 280
 281 Create CephFS Export
 282 --------------------
 283
 284 .. code:: bash
 285
 286     $ ceph nfs export create cephfs --cluster-id <cluster_id> --pseudo-path <pseudo_path> --fsname <fsname> [--readonly] [--path=/path/in/cephfs] [--client_addr <value>...] [--squash <value>] [--sectype <value>...]
 287
 288 This creates export RADOS objects containing the export block, where
 289
 290 ``<cluster_id>`` is the NFS Ganesha cluster ID.
 291
 292 ``<pseudo_path>`` is the export position within the NFS v4 Pseudo Filesystem where the export will be available on the server. It must be an absolute path and unique.
 293
 294 ``<fsname>`` is the name of the FS volume used by the NFS Ganesha cluster
 295 that will serve this export.
 296
 297 ``<path>`` is the path within cephfs. Valid path should be given and default
 298 path is '/'. It need not be unique. Subvolume path can be fetched using:
 299
 300 .. code::
 301
 302    $ ceph fs subvolume getpath <vol_name> <subvol_name> [--group_name <subvol_group_name>]
 303
 304 ``<client_addr>`` is the list of client address for which these export
 305 permissions will be applicable. By default all clients can access the export
 306 according to specified export permissions. See the `NFS-Ganesha Export Sample`_
 307 for permissible values.
 308
 309 ``<squash>`` defines the kind of user id squashing to be performed. The default
 310 value is `no_root_squash`. See the `NFS-Ganesha Export Sample`_ for
 311 permissible values.
 312
 313 ``<sectype>`` specifies which authentication methods will be used when
 314 connecting to the export. Valid values include "krb5p", "krb5i", "krb5", "sys",
 315 and "none". More than one value can be supplied. The flag may be specified
 316 multiple times (example: ``--sectype=krb5p --sectype=krb5i``) or multiple
 317 values may be separated by a comma (example: ``--sectype krb5p,krb5i``). The
 318 server will negotatiate a supported security type with the client preferring
 319 the supplied methods left-to-right.
 320
 321 .. note:: Specifying values for sectype that require Kerberos will only function on servers
 322           that are configured to support Kerberos. Setting up NFS-Ganesha to support Kerberos
 323           is outside the scope of this document.
 324
 325 .. note:: Export creation is supported only for NFS Ganesha clusters deployed using nfs interface.
 326
 327 Create RGW Export
 328 -----------------
 329
 330 There are two kinds of RGW exports:
 331
 332 - a *user* export will export all buckets owned by an
 333   RGW user, where the top-level directory of the export is a list of buckets.
 334 - a *bucket* export will export a single bucket, where the top-level directory contains
 335   the objects in the bucket.
 336
 337 RGW bucket export
 338 ^^^^^^^^^^^^^^^^^
 339
 340 To export a *bucket*:
 341
 342 .. code::
 343
 344    $ ceph nfs export create rgw --cluster-id <cluster_id> --pseudo-path <pseudo_path> --bucket <bucket_name> [--user-id <user-id>] [--readonly] [--client_addr <value>...] [--squash <value>] [--sectype <value>...]
 345
 346 For example, to export *mybucket* via NFS cluster *mynfs* at the pseudo-path */bucketdata* to any host in the ``192.168.10.0/24`` network
 347
 348 .. code::
 349
 350    $ ceph nfs export create rgw --cluster-id mynfs --pseudo-path /bucketdata --bucket mybucket --client_addr 192.168.10.0/24
 351
 352 .. note:: Export creation is supported only for NFS Ganesha clusters deployed using nfs interface.
 353
 354 ``<cluster_id>`` is the NFS Ganesha cluster ID.
 355
 356 ``<pseudo_path>`` is the export position within the NFS v4 Pseudo Filesystem where the export will be available on the server. It must be an absolute path and unique.
 357
 358 ``<bucket_name>`` is the name of the bucket that will be exported.
 359
 360 ``<user_id>`` is optional, and specifies which RGW user will be used for read and write
 361 operations to the bucket.  If it is not specified, the user who owns the bucket will be
 362 used.
 363
 364 .. note:: Currently, if multi-site RGW is enabled, Ceph can only export RGW buckets in the default realm.
 365
 366 ``<client_addr>`` is the list of client address for which these export
 367 permissions will be applicable. By default all clients can access the export
 368 according to specified export permissions. See the `NFS-Ganesha Export Sample`_
 369 for permissible values.
 370
 371 ``<squash>`` defines the kind of user id squashing to be performed. The default
 372 value is `no_root_squash`. See the `NFS-Ganesha Export Sample`_ for
 373 permissible values.
 374
 375 ``<sectype>`` specifies which authentication methods will be used when
 376 connecting to the export. Valid values include "krb5p", "krb5i", "krb5", "sys",
 377 and "none". More than one value can be supplied. The flag may be specified
 378 multiple times (example: ``--sectype=krb5p --sectype=krb5i``) or multiple
 379 values may be separated by a comma (example: ``--sectype krb5p,krb5i``). The
 380 server will negotatiate a supported security type with the client preferring
 381 the supplied methods left-to-right.
 382
 383 .. note:: Specifying values for sectype that require Kerberos will only function on servers
 384           that are configured to support Kerberos. Setting up NFS-Ganesha to support Kerberos
 385           is outside the scope of this document.
 386
 387 RGW user export
 388 ^^^^^^^^^^^^^^^
 389
 390 To export an RGW *user*:
 391
 392 .. code::
 393
 394    $ ceph nfs export create rgw --cluster-id <cluster_id> --pseudo-path <pseudo_path> --user-id <user-id> [--readonly] [--client_addr <value>...] [--squash <value>]
 395
 396 For example, to export *myuser* via NFS cluster *mynfs* at the pseudo-path */myuser* to any host in the ``192.168.10.0/24`` network
 397
 398 .. code::
 399
 400    $ ceph nfs export create rgw --cluster-id mynfs --pseudo-path /bucketdata --user-id myuser --client_addr 192.168.10.0/24
 401
 402
 403 Delete Export
 404 -------------
 405
 406 .. code:: bash
 407
 408     $ ceph nfs export rm <cluster_id> <pseudo_path>
 409
 410 This deletes an export in an NFS Ganesha cluster, where:
 411
 412 ``<cluster_id>`` is the NFS Ganesha cluster ID.
 413
 414 ``<pseudo_path>`` is the pseudo root path (must be an absolute path).
 415
 416 List Exports
 417 ------------
 418
 419 .. code:: bash
 420
 421     $ ceph nfs export ls <cluster_id> [--detailed]
 422
 423 It lists exports for a cluster, where:
 424
 425 ``<cluster_id>`` is the NFS Ganesha cluster ID.
 426
 427 With the ``--detailed`` option enabled it shows entire export block.
 428
 429 Get Export
 430 ----------
 431
 432 .. code:: bash
 433
 434     $ ceph nfs export info <cluster_id> <pseudo_path>
 435
 436 This displays export block for a cluster based on pseudo root name,
 437 where:
 438
 439 ``<cluster_id>`` is the NFS Ganesha cluster ID.
 440
 441 ``<pseudo_path>`` is the pseudo root path (must be an absolute path).
 442
 443
 444 Create or update export via JSON specification
 445 ----------------------------------------------
 446
 447 An existing export can be dumped in JSON format with:
 448
 449 .. prompt:: bash #
 450
 451     ceph nfs export info *<cluster_id>* *<pseudo_path>*
 452
 453 An export can be created or modified by importing a JSON description in the
 454 same format:
 455
 456 .. prompt:: bash #
 457
 458     ceph nfs export apply *<cluster_id>* -i <json_file>
 459
 460 For example,::
 461
 462    $ ceph nfs export info mynfs /cephfs > update_cephfs_export.json
 463    $ cat update_cephfs_export.json
 464    {
 465      "export_id": 1,
 466      "path": "/",
 467      "cluster_id": "mynfs",
 468      "pseudo": "/cephfs",
 469      "access_type": "RW",
 470      "squash": "no_root_squash",
 471      "security_label": true,
 472      "protocols": [
 473        4
 474      ],
 475      "transports": [
 476        "TCP"
 477      ],
 478      "fsal": {
 479        "name": "CEPH",
 480        "user_id": "nfs.mynfs.1",
 481        "fs_name": "a",
 482        "sec_label_xattr": ""
 483      },
 484      "clients": []
 485    }
 486
 487 The imported JSON can be a single dict describing a single export, or a JSON list
 488 containing multiple export dicts.
 489
 490 The exported JSON can be modified and then reapplied.  Below, *pseudo*
 491 and *access_type* are modified.  When modifying an export, the
 492 provided JSON should fully describe the new state of the export (just
 493 as when creating a new export), with the exception of the
 494 authentication credentials, which will be carried over from the
 495 previous state of the export where possible.
 496
 497 ::
 498
 499    $ ceph nfs export apply mynfs -i update_cephfs_export.json
 500    $ cat update_cephfs_export.json
 501    {
 502      "export_id": 1,
 503      "path": "/",
 504      "cluster_id": "mynfs",
 505      "pseudo": "/cephfs_testing",
 506      "access_type": "RO",
 507      "squash": "no_root_squash",
 508      "security_label": true,
 509      "protocols": [
 510        4
 511      ],
 512      "transports": [
 513        "TCP"
 514      ],
 515      "fsal": {
 516        "name": "CEPH",
 517        "user_id": "nfs.mynfs.1",
 518        "fs_name": "a",
 519        "sec_label_xattr": ""
 520      },
 521      "clients": []
 522    }
 523
 524 An export can also be created or updated by injecting a Ganesha NFS EXPORT config
 525 fragment.  For example,::
 526
 527    $ ceph nfs export apply mynfs -i update_cephfs_export.conf
 528    $ cat update_cephfs_export.conf
 529    EXPORT {
 530        FSAL {
 531            name = "CEPH";
 532            filesystem = "a";
 533        }
 534        export_id = 1;
 535        path = "/";
 536        pseudo = "/a";
 537        access_type = "RW";
 538        squash = "none";
 539        attr_expiration_time = 0;
 540        security_label = true;
 541        protocols = 4;
 542        transports = "TCP";
 543    }
 544
 545
 546 Mounting
 547 ========
 548
 549 After the exports are successfully created and NFS Ganesha daemons are
 550 deployed, exports can be mounted with:
 551
 552 .. code:: bash
 553
 554     $ mount -t nfs <ganesha-host-name>:<pseudo_path> <mount-point>
 555
 556 For example, if the NFS cluster was created with ``--ingress --virtual-ip 192.168.10.10``
 557 and the export's pseudo-path was ``/foo``, the export can be mounted at ``/mnt`` with:
 558
 559 .. code:: bash
 560
 561     $ mount -t nfs 192.168.10.10:/foo /mnt
 562
 563 If the NFS service is running on a non-standard port number:
 564
 565 .. code:: bash
 566
 567     $ mount -t nfs -o port=<ganesha-port> <ganesha-host-name>:<ganesha-pseudo_path> <mount-point>
 568
 569 .. note:: Only NFS v4.0+ is supported.
 570
 571 Troubleshooting
 572 ===============
 573
 574 Checking NFS-Ganesha logs with
 575
 576 1) ``cephadm``: The NFS daemons can be listed with:
 577
 578    .. code:: bash
 579
 580             $ ceph orch ps --daemon-type nfs
 581
 582    You can via the logs for a specific daemon (e.g., ``nfs.mynfs.0.0.myhost.xkfzal``) on
 583    the relevant host with:
 584
 585    .. code:: bash
 586
 587       # cephadm logs --fsid <fsid> --name nfs.mynfs.0.0.myhost.xkfzal
 588
 589 2) ``rook``:
 590
 591    .. code:: bash
 592
 593       $ kubectl logs -n rook-ceph rook-ceph-nfs-<cluster_id>-<node_id> nfs-ganesha
 594
 595 The NFS log level can be adjusted using `nfs cluster config set` command (see :ref:`nfs-cluster-set`).
 596
 597
 598 .. _nfs-ganesha-config:
 599
 600
 601 Manual Ganesha deployment
 602 =========================
 603
 604 It may be possible to deploy and manage the NFS ganesha daemons without
 605 orchestration frameworks such as cephadm or rook.
 606
 607 .. note:: Manual configuration is not tested or fully documented; your
 608           mileage may vary. If you make this work, please help us by
 609           updating this documentation.
 610
 611 Limitations
 612 ------------
 613
 614 If no orchestrator module is enabled for the Ceph Manager the NFS cluster
 615 management commands, such as those starting with ``ceph nfs cluster``, will not
 616 function. However, commands that manage NFS exports, like those prefixed with
 617 ``ceph nfs export`` are expected to work as long as the necessary RADOS objects
 618 have already been created. The exact RADOS objects required are not documented
 619 at this time as support for this feature is incomplete. A curious reader can
 620 find some details about the object by reading the source code for the
 621 ``mgr/nfs`` module (found in the ceph source tree under
 622 ``src/pybind/mgr/nfs``).
 623
 624
 625 Requirements
 626 ------------
 627
 628 The following packages are required to enable CephFS and RGW exports with nfs-ganesha:
 629
 630 -  ``nfs-ganesha``, ``nfs-ganesha-ceph``, ``nfs-ganesha-rados-grace`` and
 631    ``nfs-ganesha-rados-urls`` packages (version 3.3 and above)
 632
 633 Ganesha Configuration Hierarchy
 634 -------------------------------
 635
 636 Cephadm and rook start each nfs-ganesha daemon with a minimal
 637 `bootstrap` configuration file that pulls from a shared `common`
 638 configuration stored in the ``.nfs`` RADOS pool and watches the common
 639 config for changes.  Each export is written to a separate RADOS object
 640 that is referenced by URL from the common config.
 641
 642 .. ditaa::
 643
 644                              rados://$pool/$namespace/export-$i        rados://$pool/$namespace/userconf-nfs.$cluster_id
 645                                       (export config)                          (user config)
 646
 647                         +----------+    +----------+    +----------+      +---------------------------+
 648                         |          |    |          |    |          |      |                           |
 649                         | export-1 |    | export-2 |    | export-3 |      | userconf-nfs.$cluster_id  |
 650                         |          |    |          |    |          |      |                           |
 651                         +----+-----+    +----+-----+    +-----+----+      +-------------+-------------+
 652                              ^               ^                ^                         ^
 653                              |               |                |                         |
 654                              +--------------------------------+-------------------------+
 655                                         %url |
 656                                              |
 657                                     +--------+--------+
 658                                     |                 |  rados://$pool/$namespace/conf-nfs.$svc
 659                                     |  conf+nfs.$svc  |  (common config)
 660                                     |                 |
 661                                     +--------+--------+
 662                                              ^
 663                                              |
 664                                    watch_url |
 665                      +----------------------------------------------+
 666                      |                       |                      |
 667                      |                       |                      |            RADOS
 668              +----------------------------------------------------------------------------------+
 669                      |                       |                      |            CONTAINER
 670            watch_url |             watch_url |            watch_url |
 671                      |                       |                      |
 672             +--------+-------+      +--------+-------+      +-------+--------+
 673             |                |      |                |      |                |  /etc/ganesha/ganesha.conf
 674             |   nfs.$svc.a   |      |   nfs.$svc.b   |      |   nfs.$svc.c   |  (bootstrap config)
 675             |                |      |                |      |                |
 676             +----------------+      +----------------+      +----------------+
 677
 678
 679 .. _NFS-Ganesha NFS Server: https://github.com/nfs-ganesha/nfs-ganesha/wiki
 680 .. _NFS-Ganesha Export Sample: https://github.com/nfs-ganesha/nfs-ganesha/blob/next/src/config_samples/export.txt