9 The Ceph Dashboard is a built-in web-based Ceph management and monitoring
10 application to administer various aspects and objects of the cluster. It is
11 implemented as a :ref:`ceph-manager-daemon` module.
13 The original Ceph Dashboard that was shipped with Ceph Luminous started
14 out as a simple read-only view into various run-time information and performance
15 data of a Ceph cluster. It used a very simple architecture to achieve the
16 original goal. However, there was a growing demand for adding more web-based
17 management capabilities, to make it easier to administer Ceph for users that
18 prefer a WebUI over using the command line.
20 The new :term:`Ceph Dashboard` module is a replacement of the previous one and
21 adds a built-in web based monitoring and administration application to the Ceph
22 Manager. The architecture and functionality of this new module is derived from
23 and inspired by the `openATTIC Ceph management and monitoring tool
24 <https://openattic.org/>`_. The development is actively driven by the team
25 behind openATTIC at `SUSE <https://www.suse.com/>`_, with a lot of support from
26 companies like `Red Hat <https://redhat.com/>`_ and other members of the Ceph
29 The dashboard module's backend code uses the CherryPy framework and a custom
30 REST API implementation. The WebUI implementation is based on
31 Angular/TypeScript, merging both functionality from the original dashboard as
32 well as adding new functionality originally developed for the standalone version
33 of openATTIC. The Ceph Dashboard module is implemented as a web
34 application that visualizes information and statistics about the Ceph cluster
35 using a web server hosted by ``ceph-mgr``.
40 The dashboard provides the following features:
42 * **Multi-User and Role Management**: The dashboard supports multiple user
43 accounts with different permissions (roles). The user accounts and roles
44 can be modified on both the command line and via the WebUI. The dashboard
45 supports various methods to enhance password security, e.g. by enforcing
46 configurable password complexity rules, forcing users to change their password
47 after the first login or after a configurable time period. See
48 :ref:`dashboard-user-role-management` for details.
49 * **Single Sign-On (SSO)**: the dashboard supports authentication
50 via an external identity provider using the SAML 2.0 protocol. See
51 :ref:`dashboard-sso-support` for details.
52 * **SSL/TLS support**: All HTTP communication between the web browser and the
53 dashboard is secured via SSL. A self-signed certificate can be created with
54 a built-in command, but it's also possible to import custom certificates
55 signed and issued by a CA. See :ref:`dashboard-ssl-tls-support` for details.
56 * **Auditing**: the dashboard backend can be configured to log all PUT, POST
57 and DELETE API requests in the Ceph audit log. See :ref:`dashboard-auditing`
58 for instructions on how to enable this feature.
59 * **Internationalization (I18N)**: the dashboard can be used in different
60 languages that can be selected at run-time.
62 Currently, Ceph Dashboard is capable of monitoring and managing the following
63 aspects of your Ceph cluster:
65 * **Overall cluster health**: Display overall cluster status, performance
67 * **Embedded Grafana Dashboards**: Ceph Dashboard is capable of embedding
68 `Grafana`_ dashboards in many locations, to display additional information
69 and performance metrics gathered by the :ref:`mgr-prometheus`. See
70 :ref:`dashboard-grafana` for details on how to configure this functionality.
71 * **Cluster logs**: Display the latest updates to the cluster's event and
72 audit log files. Log entries can be filtered by priority, date or keyword.
73 * **Hosts**: Display a list of all hosts associated to the cluster, which
74 disks are attached, which services are running and which version of Ceph is
76 * **Performance counters**: Display detailed service-specific statistics for
78 * **Monitors**: List all MONs, their quorum status, open sessions.
79 * **Monitoring**: Enable creation, re-creation, editing and expiration of
80 Prometheus' silences, list the alerting configuration of Prometheus and all
81 configured and firing alerts. Show notifications for firing alerts.
82 * **Configuration Editor**: Display all available configuration options,
83 their description, type and default values and edit the current values.
84 * **Pools**: List all Ceph pools and their details (e.g. applications,
85 pg-autoscaling, placement groups, replication size, EC profile, CRUSH
86 rulesets, quotas etc.)
87 * **OSDs**: List all OSDs, their status and usage statistics as well as
88 detailed information like attributes (OSD map), metadata, performance
89 counters and usage histograms for read/write operations. Mark OSDs
90 up/down/out, purge and reweight OSDs, perform scrub operations, modify
91 various scrub-related configuration options, select different profiles to
92 adjust the level of backfilling activity. List all disks associated with an
93 OSD. Set and change the device class of an OSD, display and sort OSDs by
94 device class. Deploy new OSDs on new disks/hosts.
95 * **Device management**: List all hosts known by the orchestrator. List all
96 disks and their properties attached to a node. Display disk health information
97 (health prediction and SMART data). Blink enclosure LEDs.
98 * **iSCSI**: List all hosts that run the TCMU runner service, display all
99 images and their performance characteristics (read/write ops, traffic).
100 Create, modify and delete iSCSI targets (via ``ceph-iscsi``). Display the
101 iSCSI gateway status on the landing page and info about active initiators.
102 See :ref:`dashboard-iscsi-management` for instructions on how to configure
104 * **RBD**: List all RBD images and their properties (size, objects, features).
105 Create, copy, modify and delete RBD images (incl. snapshots) and manage RBD
106 namespaces. Define various I/O or bandwidth limitation settings on a global,
107 per-pool or per-image level. Create, delete and rollback snapshots of selected
108 images, protect/unprotect these snapshots against modification. Copy or clone
109 snapshots, flatten cloned images.
110 * **RBD mirroring**: Enable and configure RBD mirroring to a remote Ceph server.
111 Lists all active sync daemons and their status, pools and RBD images including
112 their synchronization state.
113 * **CephFS**: List all active file system clients and associated pools,
114 including their usage statistics. Evict active CephFS clients. Manage CephFS
115 quotas and snapshots. Browse a CephFS directory structure.
116 * **Object Gateway**: List all active object gateways and their performance
117 counters. Display and manage (add/edit/delete) object gateway users and their
118 details (e.g. quotas) as well as the users' buckets and their details (e.g.
119 placement targets, owner, quotas, versioning, multi-factor authentication).
120 See :ref:`dashboard-enabling-object-gateway` for configuration instructions.
121 * **NFS**: Manage NFS exports of CephFS file systems and RGW S3 buckets via NFS
122 Ganesha. See :ref:`dashboard-nfs-ganesha-management` for details on how to
123 enable this functionality.
124 * **Ceph Manager Modules**: Enable and disable all Ceph Manager modules, change
125 the module-specific configuration settings.
131 Ceph Dashboard is primarily tested and developed using the following web
134 +-----------------------------------------------+----------+
135 | Browser | Versions |
136 +===============================================+==========+
137 | `Chrome <https://www.google.com/chrome/>`_ | 68+ |
138 +-----------------------------------------------+----------+
139 | `Firefox <https://www.mozilla.org/firefox/>`_ | 61+ |
140 +-----------------------------------------------+----------+
142 While Ceph Dashboard might work in older browsers, we cannot guarantee it and
143 recommend you to update your browser to the latest version.
148 If you have installed ``ceph-mgr-dashboard`` from distribution packages, the
149 package management system should have taken care of installing all the required
152 If you're installing Ceph from source and want to start the dashboard from your
153 development environment, please see the files ``README.rst`` and ``HACKING.rst``
154 in directory ``src/pybind/mgr/dashboard`` of the source code.
156 Within a running Ceph cluster, the Ceph Dashboard is enabled with::
158 $ ceph mgr module enable dashboard
163 .. _dashboard-ssl-tls-support:
168 All HTTP connections to the dashboard are secured with SSL/TLS by default.
170 To get the dashboard up and running quickly, you can generate and install a
171 self-signed certificate using the following built-in command::
173 $ ceph dashboard create-self-signed-cert
175 Note that most web browsers will complain about such self-signed certificates
176 and require explicit confirmation before establishing a secure connection to the
179 To properly secure a deployment and to remove the certificate warning, a
180 certificate that is issued by a certificate authority (CA) should be used.
182 For example, a key pair can be generated with a command similar to::
184 $ openssl req -new -nodes -x509 \
185 -subj "/O=IT/CN=ceph-mgr-dashboard" -days 3650 \
186 -keyout dashboard.key -out dashboard.crt -extensions v3_ca
188 The ``dashboard.crt`` file should then be signed by a CA. Once that is done, you
189 can enable it for all Ceph manager instances by running the following commands::
191 $ ceph dashboard set-ssl-certificate -i dashboard.crt
192 $ ceph dashboard set-ssl-certificate-key -i dashboard.key
194 If different certificates are desired for each manager instance for some reason,
195 the name of the instance can be included as follows (where ``$name`` is the name
196 of the ``ceph-mgr`` instance, usually the hostname)::
198 $ ceph dashboard set-ssl-certificate $name -i dashboard.crt
199 $ ceph dashboard set-ssl-certificate-key $name -i dashboard.key
201 SSL can also be disabled by setting this configuration value::
203 $ ceph config set mgr mgr/dashboard/ssl false
205 This might be useful if the dashboard will be running behind a proxy which does
206 not support SSL for its upstream servers or other situations where SSL is not
207 wanted or required. See :ref:`dashboard-proxy-configuration` for more details.
211 Use caution when disabling SSL as usernames and passwords will be sent to the
212 dashboard unencrypted.
217 You need to restart the Ceph manager processes manually after changing the SSL
218 certificate and key. This can be accomplished by either running ``ceph mgr
219 fail mgr`` or by disabling and re-enabling the dashboard module (which also
220 triggers the manager to respawn itself)::
222 $ ceph mgr module disable dashboard
223 $ ceph mgr module enable dashboard
228 Like most web applications, dashboard binds to a TCP/IP address and TCP port.
230 By default, the ``ceph-mgr`` daemon hosting the dashboard (i.e., the currently
231 active manager) will bind to TCP port 8443 or 8080 when SSL is disabled.
233 If no specific address has been configured, the web app will bind to ``::``,
234 which corresponds to all available IPv4 and IPv6 addresses.
236 These defaults can be changed via the configuration key facility on a
237 cluster-wide level (so they apply to all manager instances) as follows::
239 $ ceph config set mgr mgr/dashboard/server_addr $IP
240 $ ceph config set mgr mgr/dashboard/server_port $PORT
241 $ ceph config set mgr mgr/dashboard/ssl_server_port $PORT
243 Since each ``ceph-mgr`` hosts its own instance of dashboard, it may also be
244 necessary to configure them separately. The IP address and port for a specific
245 manager instance can be changed with the following commands::
247 $ ceph config set mgr mgr/dashboard/$name/server_addr $IP
248 $ ceph config set mgr mgr/dashboard/$name/server_port $PORT
249 $ ceph config set mgr mgr/dashboard/$name/ssl_server_port $PORT
251 Replace ``$name`` with the ID of the ceph-mgr instance hosting the dashboard web
256 The command ``ceph mgr services`` will show you all endpoints that are
257 currently configured. Look for the ``dashboard`` key to obtain the URL for
258 accessing the dashboard.
260 Username and Password
261 ^^^^^^^^^^^^^^^^^^^^^
263 In order to be able to log in, you need to create a user account and associate
264 it with at least one role. We provide a set of predefined *system roles* that
265 you can use. For more details please refer to the `User and Role Management`_
268 To create a user with the administrator role you can use the following
271 $ ceph dashboard ac-user-create <username> <password> administrator
276 It disables a user account if a user repeatedly enters the wrong credentials
277 for multiple times. It is enabled by default to prevent brute-force or dictionary
278 attacks. The user can get or set the default number of lock-out attempts using
279 these commands respectively::
281 $ ceph dashboard get-account-lockout-attempts
282 $ ceph dashboard set-account-lockout-attempts <value:int>
286 This feature can be disabled by setting the default number of lock-out attempts to 0.
287 However, by disabling this feature, the account is more vulnerable to brute-force or
288 dictionary based attacks. This can be disabled by::
290 $ ceph dashboard set-account-lockout-attempts 0
295 If a user account is disabled as a result of multiple invalid login attempts, then
296 it needs to be manually enabled by the administrator. This can be done by the following
299 $ ceph dashboard ac-user-enable <username>
301 Accessing the Dashboard
302 ^^^^^^^^^^^^^^^^^^^^^^^
304 You can now access the dashboard using your (JavaScript-enabled) web browser, by
305 pointing it to any of the host names or IP addresses and the selected TCP port
306 where a manager instance is running: e.g., ``http(s)://<$IP>:<$PORT>/``.
308 You should then be greeted by the dashboard login page, requesting your
309 previously defined username and password.
311 .. _dashboard-enabling-object-gateway:
313 Enabling the Object Gateway Management Frontend
314 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
316 To use the Object Gateway management functionality of the dashboard, you will
317 need to provide the login credentials of a user with the ``system`` flag
320 If you do not have a user which shall be used for providing those credentials,
321 you will also need to create one::
323 $ radosgw-admin user create --uid=<user_id> --display-name=<display_name> \
326 Take note of the keys ``access_key`` and ``secret_key`` in the output of this
329 The credentials of an existing user can also be obtained by using
332 $ radosgw-admin user info --uid=<user_id>
334 Finally, provide the credentials to the dashboard::
336 $ ceph dashboard set-rgw-api-access-key <access_key>
337 $ ceph dashboard set-rgw-api-secret-key <secret_key>
339 In a typical default configuration with a single RGW endpoint, this is all you
340 have to do to get the Object Gateway management functionality working. The
341 dashboard will try to automatically determine the host and port of the Object
342 Gateway by obtaining this information from the Ceph Manager's service map.
344 If multiple zones are used, it will automatically determine the host within the
345 master zone group and master zone. This should be sufficient for most setups,
346 but in some circumstances you might want to set the host and port manually::
348 $ ceph dashboard set-rgw-api-host <host>
349 $ ceph dashboard set-rgw-api-port <port>
351 In addition to the settings mentioned so far, the following settings do also
352 exist and you may find yourself in the situation that you have to use them::
354 $ ceph dashboard set-rgw-api-scheme <scheme> # http or https
355 $ ceph dashboard set-rgw-api-admin-resource <admin_resource>
356 $ ceph dashboard set-rgw-api-user-id <user_id>
358 If you are using a self-signed certificate in your Object Gateway setup, then
359 you should disable certificate verification in the dashboard to avoid refused
360 connections, e.g. caused by certificates signed by unknown CA or not matching
363 $ ceph dashboard set-rgw-api-ssl-verify False
365 If the Object Gateway takes too long to process requests and the dashboard runs
366 into timeouts, then you can set the timeout value to your needs::
368 $ ceph dashboard set-rest-requests-timeout <seconds>
370 The default value is 45 seconds.
372 .. _dashboard-iscsi-management:
374 Enabling iSCSI Management
375 ^^^^^^^^^^^^^^^^^^^^^^^^^
377 The Ceph Dashboard can manage iSCSI targets using the REST API provided by the
378 `rbd-target-api` service of the :ref:`ceph-iscsi`. Please make sure that it's
379 installed and enabled on the iSCSI gateways.
383 The iSCSI management functionality of Ceph Dashboard depends on the latest
384 version 3 of the `ceph-iscsi <https://github.com/ceph/ceph-iscsi>`_ project.
385 Make sure that your operating system provides the correct version, otherwise
386 the dashboard won't enable the management features.
388 If ceph-iscsi REST API is configured in HTTPS mode and its using a self-signed
389 certificate, then you need to configure the dashboard to avoid SSL certificate
390 verification when accessing ceph-iscsi API.
392 To disable API SSL verification run the following command::
394 $ ceph dashboard set-iscsi-api-ssl-verification false
396 The available iSCSI gateways must be defined using the following commands::
398 $ ceph dashboard iscsi-gateway-list
399 $ ceph dashboard iscsi-gateway-add <scheme>://<username>:<password>@<host>[:port]
400 $ ceph dashboard iscsi-gateway-rm <gateway_name>
403 .. _dashboard-grafana:
405 Enabling the Embedding of Grafana Dashboards
406 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
408 `Grafana`_ requires data from `Prometheus <https://prometheus.io/>`_. Although
409 Grafana can use other data sources, the Grafana dashboards we provide contain
410 queries that are specific to Prometheus. Our Grafana dashboards therefore
411 require Prometheus as the data source. The Ceph :ref:`mgr-prometheus` also only
412 exports its data in the Prometheus' common format. The Grafana dashboards rely
413 on metric names from the Prometheus module and `Node exporter
414 <https://prometheus.io/docs/guides/node-exporter/>`_. The Node exporter is a
415 separate application that provides machine metrics.
419 Prometheus' security model presumes that untrusted users have access to the
420 Prometheus HTTP endpoint and logs. Untrusted users have access to all the
421 (meta)data Prometheus collects that is contained in the database, plus a
422 variety of operational and debugging information.
424 However, Prometheus' HTTP API is limited to read-only operations.
425 Configurations can *not* be changed using the API and secrets are not
426 exposed. Moreover, Prometheus has some built-in measures to mitigate the
427 impact of denial of service attacks.
429 Please see `Prometheus' Security model
430 <https://prometheus.io/docs/operating/security/>` for more detailed
433 Grafana and Prometheus are likely going to be bundled and installed by some
434 orchestration tools along Ceph in the near future, but currently, you will have
435 to install and configure both manually. After you have installed Prometheus and
436 Grafana on your preferred hosts, proceed with the following steps.
438 1. Enable the Ceph Exporter which comes as Ceph Manager module by running::
440 $ ceph mgr module enable prometheus
442 More details can be found in the documentation of the :ref:`mgr-prometheus`.
444 2. Add the corresponding scrape configuration to Prometheus. This may look
451 - job_name: 'prometheus'
453 - targets: ['localhost:9090']
456 - targets: ['localhost:9283']
457 - job_name: 'node-exporter'
459 - targets: ['localhost:9100']
461 3. Add Prometheus as data source to Grafana
463 4. Install the `vonage-status-panel and grafana-piechart-panel` plugins using::
465 grafana-cli plugins install vonage-status-panel
466 grafana-cli plugins install grafana-piechart-panel
468 5. Add the Dashboards to Grafana:
470 Dashboards can be added to Grafana by importing dashboard jsons.
471 Following command can be used for downloading json files::
473 wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/<Dashboard-name>.json
475 You can find all the dashboard jsons `here <https://github.com/ceph/ceph/tree/
476 master/monitoring/grafana/dashboards>`_ .
478 For Example, for ceph-cluster overview you can use::
480 wget https://raw.githubusercontent.com/ceph/ceph/master/monitoring/grafana/dashboards/ceph-cluster.json
482 6. Configure Grafana in `/etc/grafana/grafana.ini` to adapt anonymous mode::
489 In newer versions of Grafana (starting with 6.2.0-beta1) a new setting named
490 ``allow_embedding`` has been introduced. This setting needs to be explicitly
491 set to ``true`` for the Grafana integration in Ceph Dashboard to work, as its
492 default is ``false``.
497 allow_embedding = true
499 Enabling RBD-Image monitoring
500 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
502 Due to performance reasons, monitoring of RBD images is disabled by default. For
503 more information please see :ref:`prometheus-rbd-io-statistics`. If disabled,
504 the overview and details dashboards will stay empty in Grafana and the metrics
505 will not be visible in Prometheus.
507 After you have set up Grafana and Prometheus, you will need to configure the
508 connection information that the Ceph Dashboard will use to access Grafana.
510 You need to tell the dashboard on which URL the Grafana instance is
513 $ ceph dashboard set-grafana-api-url <grafana-server-url> # default: ''
515 The format of url is : `<protocol>:<IP-address>:<port>`
519 Ceph Dashboard embeds the Grafana dashboards via ``iframe`` HTML elements.
520 If Grafana is configured without SSL/TLS support, most browsers will block the
521 embedding of insecure content into a secured web page, if the SSL support in
522 the dashboard has been enabled (which is the default configuration). If you
523 can't see the embedded Grafana dashboards after enabling them as outlined
524 above, check your browser's documentation on how to unblock mixed content.
525 Alternatively, consider enabling SSL/TLS support in Grafana.
527 If you are using a self-signed certificate in your Grafana setup, then you should
528 disable certificate verification in the dashboard to avoid refused connections,
529 e.g. caused by certificates signed by unknown CA or not matching the host name::
531 $ ceph dashboard set-grafana-api-ssl-verify False
533 You can directly access Grafana Instance as well to monitor your cluster.
535 Alternative URL for Browsers
536 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
538 The Ceph Dashboard backend requires the Grafana URL to be able to verify the
539 existence of Grafana Dashboards before the frontend even loads them. Due to the
540 nature of how Grafana is implemented in Ceph Dashboard, this means that two
541 working connections are required in order to be able to see Grafana graphs in
544 - The backend (Ceph Mgr module) needs to verify the existence of the requested
545 graph. If this request succeeds, it lets the frontend know that it can safely
547 - The frontend then requests the Grafana graphs directly from the user's
548 browser using an iframe. The Grafana instance is accessed directly without any
549 detour through Ceph Dashboard.
551 Now, it might be the case that your environment makes it difficult for the
552 user's browser to directly access the URL configured in Ceph Dashboard. To solve
553 this issue, a separate URL can be configured which will solely be used to tell
554 the frontend (the user's browser) which URL it should use to access Grafana.
555 This setting won't ever be changed automatically, unlike the GRAFANA_API_URL
556 which is set by :ref:`cephadm` (only if cephadm is used to deploy monitoring
559 To change the URL that is returned to the frontend issue the following command::
561 $ ceph dashboard set-grafana-frontend-api-url <grafana-server-url>
563 If no value is set for that option, it will simply fall back to the value of the
564 GRAFANA_API_URL option. If set, it will instruct the browser to use this URL to
567 .. _dashboard-sso-support:
569 Enabling Single Sign-On (SSO)
570 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
572 The Ceph Dashboard supports external authentication of users via the
573 `SAML 2.0 <https://en.wikipedia.org/wiki/SAML_2.0>`_ protocol. You need to create
574 the user accounts and associate them with the desired roles first, as authorization
575 is still performed by the Dashboard. However, the authentication process can be
576 performed by an existing Identity Provider (IdP).
580 Ceph Dashboard SSO support relies on onelogin's
581 `python-saml <https://pypi.org/project/python-saml/>`_ library.
582 Please ensure that this library is installed on your system, either by using
583 your distribution's package management or via Python's `pip` installer.
585 To configure SSO on Ceph Dashboard, you should use the following command::
587 $ ceph dashboard sso setup saml2 <ceph_dashboard_base_url> <idp_metadata> {<idp_username_attribute>} {<idp_entity_id>} {<sp_x_509_cert>} {<sp_private_key>}
591 * **<ceph_dashboard_base_url>**: Base URL where Ceph Dashboard is accessible (e.g., `https://cephdashboard.local`)
592 * **<idp_metadata>**: URL to remote (`http://`, `https://`) or local (`file://`) path or content of the IdP metadata XML (e.g., `https://myidp/metadata`, `file:///home/myuser/metadata.xml`).
593 * **<idp_username_attribute>** *(optional)*: Attribute that should be used to get the username from the authentication response. Defaults to `uid`.
594 * **<idp_entity_id>** *(optional)*: Use this when more than one entity id exists on the IdP metadata.
595 * **<sp_x_509_cert> / <sp_private_key>** *(optional)*: File path of the certificate that should be used by Ceph Dashboard (Service Provider) for signing and encryption.
599 The issuer value of SAML requests will follow this pattern: **<ceph_dashboard_base_url>**/auth/saml2/metadata
601 To display the current SAML 2.0 configuration, use the following command::
603 $ ceph dashboard sso show saml2
607 For more information about `onelogin_settings`, please check the `onelogin documentation <https://github.com/onelogin/python-saml>`_.
611 $ ceph dashboard sso disable
613 To check if SSO is enabled::
615 $ ceph dashboard sso status
619 $ ceph dashboard sso enable saml2
621 .. _dashboard-alerting:
623 Enabling Prometheus Alerting
624 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
626 Using Prometheus for monitoring, you have to define `alerting rules
627 <https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules>`_.
628 To manage them you need to use the `Alertmanager
629 <https://prometheus.io/docs/alerting/alertmanager>`_.
630 If you are not using the Alertmanager yet, please `install it
631 <https://github.com/prometheus/alertmanager#install>`_ as it's mandatory in
632 order to receive and manage alerts from Prometheus.
634 The Alertmanager capabilities can be consumed by the dashboard in three different
637 #. Use the notification receiver of the dashboard.
639 #. Use the Prometheus Alertmanager API.
641 #. Use both sources simultaneously.
643 All three methods are going to notify you about alerts. You won't be notified
644 twice if you use both sources, but you need to consume at least the Alertmanager API
645 in order to manage silences.
647 1. Use the notification receiver of the dashboard
649 This allows you to get notifications as `configured
650 <https://prometheus.io/docs/alerting/configuration/>`_ from the Alertmanager.
651 You will get notified inside the dashboard once a notification is send out,
652 but you are not able to manage alerts.
654 Add the dashboard receiver and the new route to your Alertmanager
655 configuration. This should look like::
658 receiver: 'ceph-dashboard'
661 - name: 'ceph-dashboard'
663 - url: '<url-to-dashboard>/api/prometheus_receiver'
666 Please make sure that the Alertmanager considers your SSL certificate in terms
667 of the dashboard as valid. For more information about the correct
668 configuration checkout the `<http_config> documentation
669 <https://prometheus.io/docs/alerting/configuration/#%3Chttp_config%3E>`_.
671 2. Use the API of Prometheus and the Alertmanager
673 This allows you to manage alerts and silences. This will enable the "Active
674 Alerts", "All Alerts" as well as the "Silences" tabs in the "Monitoring"
675 section of the "Cluster" menu entry.
677 Alerts can be sorted by name, job, severity, state and start time.
678 Unfortunately it's not possible to know when an alert was sent out through a
679 notification by the Alertmanager based on your configuration, that's why the
680 dashboard will notify the user on any visible change to an alert and will
681 notify the changed alert.
683 Silences can be sorted by id, creator, status, start, updated and end time.
684 Silences can be created in various ways, it's also possible to expire them.
686 #. Create from scratch
688 #. Based on a selected alert
690 #. Recreate from expired silence
692 #. Update a silence (which will recreate and expire it (default Alertmanager behaviour))
694 To use it, specify the host and port of the Alertmanager server::
696 $ ceph dashboard set-alertmanager-api-host <alertmanager-host:port> # default: ''
700 $ ceph dashboard set-alertmanager-api-host 'http://localhost:9093'
702 To be able to see all configured alerts, you will need to configure the URL to
703 the Prometheus API. Using this API, the UI will also help you in verifying
704 that a new silence will match a corresponding alert.
708 $ ceph dashboard set-prometheus-api-host <prometheus-host:port> # default: ''
712 $ ceph dashboard set-prometheus-api-host 'http://localhost:9090'
714 After setting up the hosts, you have to refresh the dashboard in your browser window.
718 The different behaviors of both methods are configured in a way that they
719 should not disturb each other through annoying duplicated notifications
722 .. _dashboard-user-role-management:
724 User and Role Management
725 ------------------------
730 By default the password policy feature is enabled including the following
733 - Is the password longer than N characters?
734 - Are the old and new password the same?
736 The password policy feature can be switched on or off completely::
738 $ ceph dashboard set-pwd-policy-enabled <true|false>
740 The following individual checks can be switched on or off::
742 $ ceph dashboard set-pwd-policy-check-length-enabled <true|false>
743 $ ceph dashboard set-pwd-policy-check-oldpwd-enabled <true|false>
744 $ ceph dashboard set-pwd-policy-check-username-enabled <true|false>
745 $ ceph dashboard set-pwd-policy-check-exclusion-list-enabled <true|false>
746 $ ceph dashboard set-pwd-policy-check-complexity-enabled <true|false>
747 $ ceph dashboard set-pwd-policy-check-sequential-chars-enabled <true|false>
748 $ ceph dashboard set-pwd-policy-check-repetitive-chars-enabled <true|false>
750 Additionally the following options are available to configure the password
753 - The minimum password length (defaults to 8)::
755 $ ceph dashboard set-pwd-policy-min-length <N>
757 - The minimum password complexity (defaults to 10)::
759 $ ceph dashboard set-pwd-policy-min-complexity <N>
761 The password complexity is calculated by classifying each character in
762 the password. The complexity count starts by 0. A character is rated by
763 the following rules in the given order.
765 - Increase by 1 if the character is a digit.
766 - Increase by 1 if the character is a lower case ASCII character.
767 - Increase by 2 if the character is an upper case ASCII character.
768 - Increase by 3 if the character is a special character like ``!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~``.
769 - Increase by 5 if the character has not been classified by one of the previous rules.
771 - A list of comma separated words that are not allowed to be used in a
774 $ ceph dashboard set-pwd-policy-exclusion-list <word>[,...]
780 Ceph Dashboard supports managing multiple user accounts. Each user account
781 consists of a username, a password (stored in encrypted form using ``bcrypt``),
782 an optional name, and an optional email address.
784 If a new user is created via Web UI, it is possible to set an option that this
785 user must assign a new password when they log in for the first time.
787 User accounts are stored in MON's configuration database, and are globally
788 shared across all ceph-mgr instances.
790 We provide a set of CLI commands to manage user accounts:
794 $ ceph dashboard ac-user-show [<username>]
798 $ ceph dashboard ac-user-create [--enabled] [--force-password] [--pwd_update_required] <username> [<password>] [<rolename>] [<name>] [<email>] [<pwd_expiration_date>]
800 To bypass the password policy checks use the `force-password` option.
801 Use the option `pwd_update_required` so that a newly created user has
802 to change their password after the first login.
806 $ ceph dashboard ac-user-delete <username>
808 - *Change Password*::
810 $ ceph dashboard ac-user-set-password [--force-password] <username> <password>
812 - *Change Password Hash*::
814 $ ceph dashboard ac-user-set-password-hash <username> <hash>
816 The hash must be a bcrypt hash and salt, e.g. ``$2b$12$Pt3Vq/rDt2y9glTPSV.VFegiLkQeIpddtkhoFetNApYmIJOY8gau2``.
817 This can be used to import users from an external database.
819 - *Modify User (name, and email)*::
821 $ ceph dashboard ac-user-set-info <username> <name> <email>
825 $ ceph dashboard ac-user-disable <username>
829 $ ceph dashboard ac-user-enable <username>
831 User Roles and Permissions
832 ^^^^^^^^^^^^^^^^^^^^^^^^^^
834 User accounts are also associated with a set of roles that define which
835 dashboard functionality can be accessed by the user.
837 The Dashboard functionality/modules are grouped within a *security scope*.
838 Security scopes are predefined and static. The current available security
841 - **hosts**: includes all features related to the ``Hosts`` menu
843 - **config-opt**: includes all features related to management of Ceph
844 configuration options.
845 - **pool**: includes all features related to pool management.
846 - **osd**: includes all features related to OSD management.
847 - **monitor**: includes all features related to Monitor management.
848 - **rbd-image**: includes all features related to RBD image
850 - **rbd-mirroring**: includes all features related to RBD-Mirroring
852 - **iscsi**: includes all features related to iSCSI management.
853 - **rgw**: includes all features related to Rados Gateway management.
854 - **cephfs**: includes all features related to CephFS management.
855 - **manager**: include all features related to Ceph Manager
857 - **log**: include all features related to Ceph logs management.
858 - **grafana**: include all features related to Grafana proxy.
859 - **prometheus**: include all features related to Prometheus alert management.
860 - **dashboard-settings**: allows to change dashboard settings.
862 A *role* specifies a set of mappings between a *security scope* and a set of
863 *permissions*. There are four types of permissions:
870 See below for an example of a role specification based on a Python dictionary::
874 'role': 'my_new_role',
875 'description': 'My new role',
876 'scopes_permissions': {
877 'pool': ['read', 'create'],
878 'rbd-image': ['read', 'create', 'update', 'delete']
882 The above role dictates that a user has *read* and *create* permissions for
883 features related to pool management, and has full permissions for
884 features related to RBD image management.
886 The Dashboard already provides a set of predefined roles that we call
887 *system roles*, and can be used right away in a fresh Ceph Dashboard
890 The list of system roles are:
892 - **administrator**: provides full permissions for all security scopes.
893 - **read-only**: provides *read* permission for all security scopes except
894 the dashboard settings.
895 - **block-manager**: provides full permissions for *rbd-image*,
896 *rbd-mirroring*, and *iscsi* scopes.
897 - **rgw-manager**: provides full permissions for the *rgw* scope
898 - **cluster-manager**: provides full permissions for the *hosts*, *osd*,
899 *monitor*, *manager*, and *config-opt* scopes.
900 - **pool-manager**: provides full permissions for the *pool* scope.
901 - **cephfs-manager**: provides full permissions for the *cephfs* scope.
903 The list of currently available roles can be retrieved by the following
906 $ ceph dashboard ac-role-show [<rolename>]
908 It is also possible to create new roles using CLI commands. The available
909 commands to manage roles are the following:
913 $ ceph dashboard ac-role-create <rolename> [<description>]
917 $ ceph dashboard ac-role-delete <rolename>
919 - *Add Scope Permissions to Role*::
921 $ ceph dashboard ac-role-add-scope-perms <rolename> <scopename> <permission> [<permission>...]
923 - *Delete Scope Permission from Role*::
925 $ ceph dashboard ac-role-del-scope-perms <rolename> <scopename>
927 To associate roles to users, the following CLI commands are available:
931 $ ceph dashboard ac-user-set-roles <username> <rolename> [<rolename>...]
933 - *Add Roles To User*::
935 $ ceph dashboard ac-user-add-roles <username> <rolename> [<rolename>...]
937 - *Delete Roles from User*::
939 $ ceph dashboard ac-user-del-roles <username> <rolename> [<rolename>...]
942 Example of User and Custom Role Creation
943 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
945 In this section we show a full example of the commands that need to be used
946 in order to create a user account, that should be able to manage RBD images,
947 view and create Ceph pools, and have read-only access to any other scopes.
949 1. *Create the user*::
951 $ ceph dashboard ac-user-create bob mypassword
953 2. *Create role and specify scope permissions*::
955 $ ceph dashboard ac-role-create rbd/pool-manager
956 $ ceph dashboard ac-role-add-scope-perms rbd/pool-manager rbd-image read create update delete
957 $ ceph dashboard ac-role-add-scope-perms rbd/pool-manager pool read create
959 3. *Associate roles to user*::
961 $ ceph dashboard ac-user-set-roles bob rbd/pool-manager read-only
963 .. _dashboard-proxy-configuration:
968 In a Ceph cluster with multiple ceph-mgr instances, only the dashboard running
969 on the currently active ceph-mgr daemon will serve incoming requests. Accessing
970 the dashboard's TCP port on any of the other ceph-mgr instances that are
971 currently on standby will perform a HTTP redirect (303) to the currently active
972 manager's dashboard URL. This way, you can point your browser to any of the
973 ceph-mgr instances in order to access the dashboard.
975 If you want to establish a fixed URL to reach the dashboard or if you don't want
976 to allow direct connections to the manager nodes, you could set up a proxy that
977 automatically forwards incoming requests to the currently active ceph-mgr
980 Configuring a URL Prefix
981 ^^^^^^^^^^^^^^^^^^^^^^^^
983 If you are accessing the dashboard via a reverse proxy configuration,
984 you may wish to service it under a URL prefix. To get the dashboard
985 to use hyperlinks that include your prefix, you can set the
986 ``url_prefix`` setting:
990 ceph config set mgr mgr/dashboard/url_prefix $PREFIX
992 so you can access the dashboard at ``http://$IP:$PORT/$PREFIX/``.
994 Disable the redirection
995 ^^^^^^^^^^^^^^^^^^^^^^^
997 If the dashboard is behind a load-balancing proxy like `HAProxy <https://www.haproxy.org/>`_
998 you might want to disable the redirection behaviour to prevent situations that
999 internal (unresolvable) URL's are published to the frontend client. Use the
1000 following command to get the dashboard to respond with a HTTP error (500 by default)
1001 instead of redirecting to the active dashboard::
1003 $ ceph config set mgr mgr/dashboard/standby_behaviour "error"
1005 To reset the setting to the default redirection behaviour, use the following command::
1007 $ ceph config set mgr mgr/dashboard/standby_behaviour "redirect"
1009 Configure the error status code
1010 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1012 When the redirection behaviour is disabled, then you want to customize the HTTP status
1013 code of standby dashboards. To do so you need to run the command::
1015 $ ceph config set mgr mgr/dashboard/standby_error_status_code 503
1017 HAProxy example configuration
1018 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1020 Below you will find an example configuration for SSL/TLS pass through using
1021 `HAProxy <https://www.haproxy.org/>`_.
1023 Please note that the configuration works under the following conditions.
1024 If the dashboard fails over, the front-end client might receive a HTTP redirect
1025 (303) response and will be redirected to an unresolvable host. This happens when
1026 the failover occurs during two HAProxy health checks. In this situation the
1027 previously active dashboard node will now respond with a 303 which points to
1028 the new active node. To prevent that situation you should consider to disable
1029 the redirection behaviour on standby nodes.
1035 option log-health-checks
1040 frontend dashboard_front
1044 redirect scheme https code 301 if !{ ssl_fc }
1046 frontend dashboard_front_ssl
1050 default_backend dashboard_back_ssl
1052 backend dashboard_back_ssl
1054 option httpchk GET /
1055 http-check expect status 200
1056 server x <HOST>:<PORT> check-ssl check verify none
1057 server y <HOST>:<PORT> check-ssl check verify none
1058 server z <HOST>:<PORT> check-ssl check verify none
1060 .. _dashboard-auditing:
1062 Auditing API Requests
1063 ---------------------
1065 The REST API is capable of logging PUT, POST and DELETE requests to the Ceph
1066 audit log. This feature is disabled by default, but can be enabled with the
1069 $ ceph dashboard set-audit-api-enabled <true|false>
1071 If enabled, the following parameters are logged per each request:
1073 * from - The origin of the request, e.g. https://[::1]:44410
1074 * path - The REST API path, e.g. /api/auth
1075 * method - e.g. PUT, POST or DELETE
1076 * user - The name of the user, otherwise 'None'
1078 The logging of the request payload (the arguments and their values) is enabled
1079 by default. Execute the following command to disable this behaviour::
1081 $ ceph dashboard set-audit-api-log-payload <true|false>
1083 A log entry may look like this::
1085 2018-10-22 15:27:01.302514 mgr.x [INF] [DASHBOARD] from='https://[::ffff:127.0.0.1]:37022' path='/api/rgw/user/klaus' method='PUT' user='admin' params='{"max_buckets": "1000", "display_name": "Klaus Mustermann", "uid": "klaus", "suspended": "0", "email": "klaus.mustermann@ceph.com"}'
1087 .. _dashboard-nfs-ganesha-management:
1089 NFS-Ganesha Management
1090 ----------------------
1092 Ceph Dashboard can manage `NFS Ganesha <http://nfs-ganesha.github.io/>`_ exports that use
1093 CephFS or RadosGW as their backstore.
1095 To enable this feature in Ceph Dashboard there are some assumptions that need
1096 to be met regarding the way NFS-Ganesha services are configured.
1098 The dashboard manages NFS-Ganesha config files stored in RADOS objects on the Ceph Cluster.
1099 NFS-Ganesha must store part of their configuration in the Ceph cluster.
1101 These configuration files must follow some conventions.
1102 Each export block must be stored in its own RADOS object named
1103 ``export-<id>``, where ``<id>`` must match the ``Export_ID`` attribute of the
1104 export configuration. Then, for each NFS-Ganesha service daemon there should
1105 exist a RADOS object named ``conf-<daemon_id>``, where ``<daemon_id>`` is an
1106 arbitrary string that should uniquely identify the daemon instance (e.g., the
1107 hostname where the daemon is running).
1108 Each ``conf-<daemon_id>`` object contains the RADOS URLs to the exports that
1109 the NFS-Ganesha daemon should serve. These URLs are of the form::
1111 %url rados://<pool_name>[/<namespace>]/export-<id>
1113 Both the ``conf-<daemon_id>`` and ``export-<id>`` objects must be stored in the
1114 same RADOS pool/namespace.
1117 Configuring NFS-Ganesha in the Dashboard
1118 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1120 To enable the management of NFS-Ganesha exports in Ceph Dashboard, we only
1121 need to tell the Dashboard, in which RADOS pool and namespace the
1122 configuration objects are stored. Then, Ceph Dashboard can access the objects
1123 by following the naming convention described above.
1125 The Dashboard command to configure the NFS-Ganesha configuration objects
1128 $ ceph dashboard set-ganesha-clusters-rados-pool-namespace <pool_name>[/<namespace>]
1130 After running the above command, Ceph Dashboard is able to find the NFS-Ganesha
1131 configuration objects and we can start manage the exports through the Web UI.
1134 Support for Multiple NFS-Ganesha Clusters
1135 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1137 Ceph Dashboard also supports the management of NFS-Ganesha exports belonging
1138 to different NFS-Ganesha clusters. An NFS-Ganesha cluster is a group of
1139 NFS-Ganesha service daemons sharing the same exports. Different NFS-Ganesha
1140 clusters are independent and don't share the exports configuration between each
1143 Each NFS-Ganesha cluster should store its configuration objects in a
1144 different RADOS pool/namespace to isolate the configuration from each other.
1146 To specify the locations of the configuration of each NFS-Ganesha cluster we
1147 can use the same command as above but with a different value pattern::
1149 $ ceph dashboard set-ganesha-clusters-rados-pool-namespace <cluster_id>:<pool_name>[/<namespace>](,<cluster_id>:<pool_name>[/<namespace>])*
1151 The ``<cluster_id>`` is an arbitrary string that should uniquely identify the
1152 NFS-Ganesha cluster.
1154 When configuring the Ceph Dashboard with multiple NFS-Ganesha clusters, the
1155 Web UI will automatically allow to choose to which cluster an export belongs.
1158 Support for NFS-Ganesha Clusters Deployed by the Orchestrator
1159 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1161 Ceph Dashboard can be used to manage NFS-Ganesha clusters deployed by the
1162 Orchestrator. It can detect the clusters automatically. For more details
1163 on deploying NFS-Ganesha clusters with the Orchestrator, please see :ref:`orchestrator-cli-stateless-services`.
1164 Or particularly, see :ref:`deploy-cephadm-nfs-ganesha` for how to deploy
1165 NFS-Ganesha clusters with the Cephadm backend.
1171 Dashboard Plug-ins extend the functionality of the dashboard in a modular
1172 and loosely coupled fashion.
1174 .. _Grafana: https://grafana.com/
1176 .. include:: dashboard_plugins/feature_toggles.inc.rst
1177 .. include:: dashboard_plugins/debug.inc.rst
1180 Troubleshooting the Dashboard
1181 -----------------------------
1183 Locating the Dashboard
1184 ^^^^^^^^^^^^^^^^^^^^^^
1186 If you are unsure of the location of the Ceph Dashboard, run the following command::
1188 $ ceph mgr services | jq .dashboard
1191 The command returns the URL where the Ceph Dashboard is located: ``https://<host>:<port>/``
1195 Many Ceph command line tools return results in JSON format. You may have to install
1196 the `jq <https://stedolan.github.io/jq>`_ command-line JSON processor utility on
1197 your operating system beforehand.
1200 Accessing the Dashboard
1201 ^^^^^^^^^^^^^^^^^^^^^^^
1203 If you are unable to access the Ceph Dashboard, run through the following
1206 #. Verify the Ceph Dashboard module is enabled::
1208 $ ceph mgr module ls | jq .enabled_modules
1210 Ensure the Ceph Dashboard module is listed in the return value of the
1211 command. Example snipped output from the command above::
1219 #. If it is not listed, activate the module with the following command::
1221 $ ceph mgr module enable dashboard
1223 #. Check the Ceph Dashboard and/or mgr log file for any errors. The exact
1224 location of the log files depends on the Ceph configuration.
1226 * Check if mgr log messages are written to a file by::
1228 $ ceph config get mgr log_to_file
1231 * Get the location of the log file (it's ``/var/log/ceph/<cluster-name>-<daemon-name>.log``
1234 $ ceph config get mgr log_file
1235 /var/log/ceph/$cluster-$name.log
1237 #. Ensure the SSL/TSL support is configured properly:
1239 * Check if the SSL/TSL support is enabled::
1241 $ ceph config get mgr mgr/dashboard/ssl
1243 * If the command returns ``true``, verify a certificate exists by::
1245 $ ceph config-key get mgr/dashboard/crt
1249 $ ceph config-key get mgr/dashboard/key
1251 * If it doesn't, run the following command to generate a self-signed
1252 certificate or follow the instructions outlined in
1253 :ref:`dashboard-ssl-tls-support`::
1255 $ ceph dashboard create-self-signed-cert
1258 Trouble Logging into the Dashboard
1259 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1261 If you are unable to log into the Ceph Dashboard and you receive the following
1262 error, run through the procedural checks below:
1264 .. image:: ../images/dashboard/invalid-credentials.png
1267 #. Check that your user credentials are correct. If you are seeing the
1268 notification message above when trying to log into the Ceph Dashboard, it
1269 is likely you are using the wrong credentials. Double check your username
1270 and password, and ensure the caps lock key is not enabled by accident.
1272 #. If your user credentials are correct, but you are experiencing the same
1273 error, check that the user account exists::
1275 $ ceph dashboard ac-user-show <username>
1277 This command returns your user data. If the user does not exist, it will
1280 $ Error ENOENT: User <username> does not exist
1282 #. Check if the user is enabled::
1284 $ ceph dashboard ac-user-show <username> | jq .enabled
1287 Check if ``enabled`` is set to ``true`` for your user. If not the user is
1290 $ ceph dashboard ac-user-enable <username>
1292 Please see :ref:`dashboard-user-role-management` for more information.
1295 A Dashboard Feature is Not Working
1296 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1298 When an error occurs on the backend, you will usually receive an error
1299 notification on the frontend. Run through the following scenarios to debug.
1301 #. Check the Ceph Dashboard/mgr logfile(s) for any errors. These can be
1302 identified by searching for keywords, such as *500 Internal Server Error*,
1303 followed by ``traceback``. The end of a traceback contains more details about
1304 what exact error occurred.
1305 #. Check your web browser's Javascript Console for any errors.
1311 Dashboard Debug Flag
1312 ''''''''''''''''''''
1314 With this flag enabled, traceback of errors are included in backend responses.
1316 To enable this flag via the Ceph Dashboard, navigate from *Cluster* to *Manager
1317 modules*. Select *Dashboard module* and click the edit button. Click the
1318 *debug* checkbox and update.
1320 To enable it via the CLI, run the following command::
1322 $ ceph dashboard debug enable
1325 Setting Logging Level of Dashboard Module
1326 '''''''''''''''''''''''''''''''''''''''''
1328 Setting the logging level to debug makes the log more verbose and helpful for
1331 #. Increase the logging level of manager daemons::
1333 $ ceph tell mgr config set debug_mgr 20
1335 #. Adjust the logging level of the Ceph Dashboard module via the Dashboard or
1338 * Navigate from *Cluster* to *Manager modules*. Select *Dashboard module*
1339 and click the edit button. Modify the ``log_level`` configuration.
1340 * To adjust it via the CLI, run the following command::
1342 $ bin/ceph config set mgr mgr/dashboard/log_level debug