Host Management
===============
-To list hosts associated with the cluster:
+Listing Hosts
+=============
+
+Run a command of this form to list hosts associated with the cluster:
.. prompt:: bash #
- ceph orch host ls [--format yaml] [--host-pattern <name>] [--label <label>] [--host-status <status>]
+ ceph orch host ls [--format yaml] [--host-pattern <name>] [--label <label>] [--host-status <status>]
+
+In commands of this form, the arguments "host-pattern", "label" and
+"host-status" are optional and are used for filtering.
-where the optional arguments "host-pattern", "label" and "host-status" are used for filtering.
-"host-pattern" is a regex that will match against hostnames and will only return matching hosts
-"label" will only return hosts with the given label
-"host-status" will only return hosts with the given status (currently "offline" or "maintenance")
-Any combination of these filtering flags is valid. You may filter against name, label and/or status simultaneously
+- "host-pattern" is a regex that matches against hostnames and returns only
+ matching hosts.
+- "label" returns only hosts with the specified label.
+- "host-status" returns only hosts with the specified status (currently
+ "offline" or "maintenance").
+- Any combination of these filtering flags is valid. It is possible to filter
+ against name, label and status simultaneously, or to filter against any
+ proper subset of name, label and status.
.. _cephadm-adding-hosts:
.. prompt:: bash #
- ssh-copy-id -f -i /etc/ceph/ceph.pub root@*<new-host>*
+ ssh-copy-id -f -i /etc/ceph/ceph.pub root@*<new-host>*
For example:
.. prompt:: bash #
- ceph orch host add *<newhost>* [*<ip>*] [*<label1> ...*]
+ ceph orch host add *<newhost>* [*<ip>*] [*<label1> ...*]
For example:
.. prompt:: bash #
- ceph orch host add host4 10.10.0.104 --labels _admin
+ ceph orch host add host4 10.10.0.104 --labels _admin
.. _cephadm-removing-hosts:
Removing Hosts
==============
-A host can safely be removed from a the cluster once all daemons are removed from it.
+A host can safely be removed from the cluster after all daemons are removed
+from it.
-To drain all daemons from a host do the following:
+To drain all daemons from a host, run a command of the following form:
.. prompt:: bash #
- ceph orch host drain *<host>*
+ ceph orch host drain *<host>*
-The '_no_schedule' label will be applied to the host. See :ref:`cephadm-special-host-labels`
+The ``_no_schedule`` label will be applied to the host. See
+:ref:`cephadm-special-host-labels`.
-All osds on the host will be scheduled to be removed. You can check osd removal progress with the following:
+All OSDs on the host will be scheduled to be removed. You can check the progress of the OSD removal operation with the following command:
.. prompt:: bash #
- ceph orch osd rm status
+ ceph orch osd rm status
-see :ref:`cephadm-osd-removal` for more details about osd removal
+See :ref:`cephadm-osd-removal` for more details about OSD removal.
-You can check if there are no daemons left on the host with the following:
+Use the following command to determine whether any daemons are still on the
+host:
.. prompt:: bash #
- ceph orch ps <host>
+ ceph orch ps <host>
-Once all daemons are removed you can remove the host with the following:
+After all daemons have been removed from the host, remove the host from the
+cluster by running the following command:
.. prompt:: bash #
- ceph orch host rm <host>
+ ceph orch host rm <host>
Offline host removal
--------------------
-If a host is offline and can not be recovered it can still be removed from the cluster with the following:
+Even if a host is offline and can not be recovered, it can be removed from the
+cluster by running a command of the following form:
.. prompt:: bash #
- ceph orch host rm <host> --offline --force
+ ceph orch host rm <host> --offline --force
-This can potentially cause data loss as osds will be forcefully purged from the cluster by calling ``osd purge-actual`` for each osd.
-Service specs that still contain this host should be manually updated.
+.. warning:: This can potentially cause data loss. This command forcefully
+ purges OSDs from the cluster by calling ``osd purge-actual`` for each OSD.
+ Any service specs that still contain this host should be manually updated.
.. _orchestrator-host-labels:
can have multiple labels. They can be used to specify placement
of daemons. See :ref:`orch-placement-by-labels`
-Labels can be added when adding a host with the ``--labels`` flag::
+Labels can be added when adding a host with the ``--labels`` flag:
+
+.. prompt:: bash #
- ceph orch host add my_hostname --labels=my_label1
- ceph orch host add my_hostname --labels=my_label1,my_label2
+ ceph orch host add my_hostname --labels=my_label1
+ ceph orch host add my_hostname --labels=my_label1,my_label2
-To add a label a existing host, run::
+To add a label a existing host, run:
- ceph orch host label add my_hostname my_label
+.. prompt:: bash #
-To remove a label, run::
+ ceph orch host label add my_hostname my_label
- ceph orch host label rm my_hostname my_label
+To remove a label, run:
+
+.. prompt:: bash #
+
+ ceph orch host label rm my_hostname my_label
.. _cephadm-special-host-labels:
Maintenance Mode
================
-Place a host in and out of maintenance mode (stops all Ceph daemons on host)::
+Place a host in and out of maintenance mode (stops all Ceph daemons on host):
- ceph orch host maintenance enter <hostname> [--force]
- ceph orch host maintenance exit <hostname>
+.. prompt:: bash #
+
+ ceph orch host maintenance enter <hostname> [--force]
+ ceph orch host maintenance exit <hostname>
Where the force flag when entering maintenance allows the user to bypass warnings (but not alerts)
Some servers and external enclosures may not register device removal or insertion with the
kernel. In these scenarios, you'll need to perform a host rescan. A rescan is typically
-non-disruptive, and can be performed with the following CLI command.::
+non-disruptive, and can be performed with the following CLI command:
+
+.. prompt:: bash #
- ceph orch host rescan <hostname> [--with-summary]
+ ceph orch host rescan <hostname> [--with-summary]
The ``with-summary`` flag provides a breakdown of the number of HBAs found and scanned, together
-with any that failed.::
+with any that failed:
+
+.. prompt:: bash [ceph:root@rh9-ceph1/]#
+
+ ceph orch host rescan rh9-ceph1 --with-summary
+
+::
- [ceph: root@rh9-ceph1 /]# ceph orch host rescan rh9-ceph1 --with-summary
- Ok. 2 adapters detected: 2 rescanned, 0 skipped, 0 failed (0.32s)
+ Ok. 2 adapters detected: 2 rescanned, 0 skipped, 0 failed (0.32s)
Creating many hosts at once
===========================
hostname: node-02
addr: 192.168.0.12
-This can be combined with service specifications (below) to create a cluster spec
-file to deploy a whole cluster in one command. see ``cephadm bootstrap --apply-spec``
-also to do this during bootstrap. Cluster SSH Keys must be copied to hosts prior to adding them.
+This can be combined with :ref:`service specifications<orchestrator-cli-service-spec>`
+to create a cluster spec file to deploy a whole cluster in one command. see
+``cephadm bootstrap --apply-spec`` also to do this during bootstrap. Cluster
+SSH Keys must be copied to hosts prior to adding them.
Setting the initial CRUSH location of host
==========================================
OS Tuning Profiles
==================
-Cephadm can manage operating system tuning profiles that apply a set of sysctl settings
-to a given set of hosts. First create a YAML spec file in the following format
+Cephadm can be used to manage operating-system-tuning profiles that apply sets
+of sysctl settings to sets of hosts.
+
+Create a YAML spec file in the following format:
.. code-block:: yaml
fs.file-max: 1000000
vm.swappiness: '13'
-Then apply the tuning profile with::
+Apply the tuning profile with the following command:
- ceph orch tuned-profile apply -i <tuned-profile-file-name>
+.. prompt:: bash #
-This profile will then be written to ``/etc/sysctl.d/`` on each host matching the
-given placement and `sysctl --system` will be run on the host.
+ ceph orch tuned-profile apply -i <tuned-profile-file-name>
+
+This profile is written to ``/etc/sysctl.d/`` on each host that matches the
+hosts specified in the placement block of the yaml, and ``sysctl --system`` is
+run on the host.
.. note::
- The exact filename the profile will be written to is within ``/etc/sysctl.d/`` is
- ``<profile-name>-cephadm-tuned-profile.conf`` where <profile-name>
- is the `profile_name` setting specified in the provided YAML spec. Since sysctl
- settings are applied in lexicographical order by the filename the setting is
- specified in, you may want to set the `profile_name` in your spec so
- that it is applied before or after other conf files that may exist.
+ The exact filename that the profile is written to within ``/etc/sysctl.d/``
+ is ``<profile-name>-cephadm-tuned-profile.conf``, where ``<profile-name>`` is
+ the ``profile_name`` setting that you specify in the YAML spec. Because
+ sysctl settings are applied in lexicographical order (sorted by the filename
+ in which the setting is specified), you may want to set the ``profile_name``
+ in your spec so that it is applied before or after other conf files.
.. note::
These settings are applied only at the host level, and are not specific
- to any certain daemon or container
+ to any particular daemon or container.
.. note::
- Applying tuned profiles is idempotent when the ``--no-overwrite`` option is passed.
- In this case existing profiles with the same name are not overwritten.
+ Applying tuned profiles is idempotent when the ``--no-overwrite`` option is
+ passed. Moreover, if the ``--no-overwrite`` option is passed, existing
+ profiles with the same name are not overwritten.
Viewing Profiles
----------------
-To view all current profiles cephadm is managing::
+Run the following command to view all the profiles that cephadm currently manages:
+
+.. prompt:: bash #
- ceph orch tuned-profile ls
+ ceph orch tuned-profile ls
.. note::
- If you'd like to make modifications and re-apply a profile passing `--format yaml` to the
- ``tuned-profile ls`` command will present the profiles in a format where they can be copied
- and re-applied.
+ To make modifications and re-apply a profile, pass ``--format yaml`` to the
+ ``tuned-profile ls`` command. The ``tuned-profile ls --format yaml`` command
+ presents the profiles in a format that is easy to copy and re-apply.
Removing Profiles
-----------------
-If you no longer want one of the previously applied profiles, it can be removed with::
+To remove a previously applied profile, run this command:
+
+.. prompt:: bash #
-
ceph orch tuned-profile rm <profile-name>
+ ceph orch tuned-profile rm <profile-name>
-When a profile is removed, cephadm will clean up the file previously written to /etc/sysctl.d
+When a profile is removed, cephadm cleans up the file previously written to ``/etc/sysctl.d``.
Modifying Profiles
------------------
-While you can modify a profile by simply re-applying a YAML spec with the same profile name,
-you may also want to adjust a setting within a given profile, so there are commands
-for this purpose.
+Profiles can be modified by re-applying a YAML spec with the same name as the
+profile that you want to modify, but settings within existing profiles can be
+adjusted with the following commands.
-To add or modify a setting for an existing profile::
+To add or modify a setting in an existing profile:
- ceph orch tuned-profile add-setting <setting-name> <value>
+.. prompt:: bash #
+
+ ceph orch tuned-profile add-setting <profile-name> <setting-name> <value>
+
+To remove a setting from an existing profile:
-To remove a setting from an existing profile::
+.. prompt:: bash #
- ceph orch tuned-profile rm-setting <setting-name>
+ ceph orch tuned-profile rm-setting <profile-name> <setting-name>
.. note::
- Modifying the placement will require re-applying a profile with the same name. Keep
- in mind that profiles are tracked by their name, so whenever a profile with the same
- name as an existing profile is applied, it will overwrite the old profile.
+ Modifying the placement requires re-applying a profile with the same name.
+ Remember that profiles are tracked by their names, so when a profile with the
+ same name as an existing profile is applied, it overwrites the old profile
+ unless the ``--no-overwrite`` flag is passed.
SSH Configuration
=================
key is generated automatically and no additional configuration
is necessary.
-A *new* SSH key can be generated with::
+A *new* SSH key can be generated with:
+
+.. prompt:: bash #
+
+ ceph cephadm generate-key
+
+The public portion of the SSH key can be retrieved with:
+
+.. prompt:: bash #
+
+ ceph cephadm get-pub-key
- ceph cephadm generate-key
+The currently stored SSH key can be deleted with:
-The public portion of the SSH key can be retrieved with::
+.. prompt:: bash #
- ceph cephadm get-pub-key
+ ceph cephadm clear-key
-The currently stored SSH key can be deleted with::
+You can make use of an existing key by directly importing it with:
- ceph cephadm clear-key
+.. prompt:: bash #
-You can make use of an existing key by directly importing it with::
+ ceph config-key set mgr/cephadm/ssh_identity_key -i <key>
+ ceph config-key set mgr/cephadm/ssh_identity_pub -i <pub>
- ceph config-key set mgr/cephadm/ssh_identity_key -i <key>
- ceph config-key set mgr/cephadm/ssh_identity_pub -i <pub>
+You will then need to restart the mgr daemon to reload the configuration with:
-You will then need to restart the mgr daemon to reload the configuration with::
+.. prompt:: bash #
- ceph mgr fail
+ ceph mgr fail
.. _cephadm-ssh-user:
and execute commands without prompting for a password. If you do not want
to use the "root" user (default option in cephadm), you must provide
cephadm the name of the user that is going to be used to perform all the
-cephadm operations. Use the command::
+cephadm operations. Use the command:
- ceph cephadm set-user <user>
+.. prompt:: bash #
+
+ ceph cephadm set-user <user>
-Prior to running this the cluster ssh key needs to be added to this users
+Prior to running this the cluster SSH key needs to be added to this users
authorized_keys file and non-root users must have passwordless sudo access.
There are two ways to customize this configuration for your environment:
#. Import a customized configuration file that will be stored
- by the monitor with::
+ by the monitor with:
- ceph cephadm set-ssh-config -i <ssh_config_file>
+ .. prompt:: bash #
- To remove a customized SSH config and revert back to the default behavior::
+ ceph cephadm set-ssh-config -i <ssh_config_file>
- ceph cephadm clear-ssh-config
+ To remove a customized SSH config and revert back to the default behavior:
-#. You can configure a file location for the SSH configuration file with::
+ .. prompt:: bash #
+
+ ceph cephadm clear-ssh-config
+
+#. You can configure a file location for the SSH configuration file with:
+
+ .. prompt:: bash #
- ceph config set mgr mgr/cephadm/ssh_config_file <path>
+
ceph config set mgr mgr/cephadm/ssh_config_file <path>
We do *not recommend* this approach. The path name must be
visible to *any* mgr daemon, and cephadm runs all daemons as
..
TODO: This chapter needs to provide way for users to configure
- Grafana in the dashboard, as this is right no very hard to do.
+ Grafana in the dashboard, as this is right now very hard to do.
projects / ceph.git / blobdiff