.. index:: monitor, high availability
-If a cluster encounters monitor-related problems, this does not necessarily
-mean that the cluster is in danger of going down. Even if multiple monitors are
-lost, the cluster can still be up and running, as long as there are enough
-surviving monitors to form a quorum.
-
-However serious your cluster's monitor-related problems might be, we recommend
-that you take the following troubleshooting steps.
-
+Even if a cluster experiences monitor-related problems, the cluster is not
+necessarily in danger of going down. If a cluster has lost multiple monitors,
+it can still remain up and running as long as there are enough surviving
+monitors to form a quorum.
+
+If your cluster is having monitor-related problems, we recommend that you
+consult the following troubleshooting information.
Initial Troubleshooting
=======================
-**Are the monitors running?**
-
- First, make sure that the monitor (*mon*) daemon processes (``ceph-mon``) are
- running. Sometimes Ceph admins either forget to start the mons or forget to
- restart the mons after an upgrade. Checking for this simple oversight can
- save hours of painstaking troubleshooting. It is also important to make sure
- that the manager daemons (``ceph-mgr``) are running. Remember that typical
- cluster configurations provide one ``ceph-mgr`` for each ``ceph-mon``.
-
- .. note:: Rook will not run more than two managers.
-
-**Can you reach the monitor nodes?**
-
- In certain rare cases, there may be ``iptables`` rules that block access to
- monitor nodes or TCP ports. These rules might be left over from earlier
- stress testing or rule development. To check for the presence of such rules,
- SSH into the server and then try to connect to the monitor's ports
- (``tcp/3300`` and ``tcp/6789``) using ``telnet``, ``nc``, or a similar tool.
-
-**Does the ``ceph status`` command run and receive a reply from the cluster?**
-
- If the ``ceph status`` command does receive a reply from the cluster, then the
- cluster is up and running. The monitors will answer to a ``status`` request
- only if there is a formed quorum. Confirm that one or more ``mgr`` daemons
- are reported as running. Under ideal conditions, all ``mgr`` daemons will be
- reported as running.
-
-
- If the ``ceph status`` command does not receive a reply from the cluster, then
- there are probably not enough monitors ``up`` to form a quorum. The ``ceph
- -s`` command with no further options specified connects to an arbitrarily
- selected monitor. In certain cases, however, it might be helpful to connect
- to a specific monitor (or to several specific monitors in sequence) by adding
- the ``-m`` flag to the command: for example, ``ceph status -m mymon1``.
-
+The first steps in the process of troubleshooting Ceph Monitors involve making
+sure that the Monitors are running and that they are able to communicate with
+the network and on the network. Follow the steps in this section to rule out
+the simplest causes of Monitor malfunction.
-**None of this worked. What now?**
+#. **Make sure that the Monitors are running.**
- If the above solutions have not resolved your problems, you might find it
- helpful to examine each individual monitor in turn. Whether or not a quorum
- has been formed, it is possible to contact each monitor individually and
- request its status by using the ``ceph tell mon.ID mon_status`` command (here
- ``ID`` is the monitor's identifier).
-
- Run the ``ceph tell mon.ID mon_status`` command for each monitor in the
- cluster. For more on this command's output, see :ref:`Understanding
- mon_status
- <rados_troubleshoting_troubleshooting_mon_understanding_mon_status>`.
-
- There is also an alternative method: SSH into each monitor node and query the
- daemon's admin socket. See :ref:`Using the Monitor's Admin
- Socket<rados_troubleshoting_troubleshooting_mon_using_admin_socket>`.
+ Make sure that the Monitor (*mon*) daemon processes (``ceph-mon``) are
+ running. It might be the case that the mons have not be restarted after an
+ upgrade. Checking for this simple oversight can save hours of painstaking
+ troubleshooting.
+
+ It is also important to make sure that the manager daemons (``ceph-mgr``)
+ are running. Remember that typical cluster configurations provide one
+ Manager (``ceph-mgr``) for each Monitor (``ceph-mon``).
+
+ .. note:: In releases prior to v1.12.5, Rook will not run more than two
+ managers.
+
+#. **Make sure that you can reach the Monitor nodes.**
+
+ In certain rare cases, ``iptables`` rules might be blocking access to
+ Monitor nodes or TCP ports. These rules might be left over from earlier
+ stress testing or rule development. To check for the presence of such
+ rules, SSH into each Monitor node and use ``telnet`` or ``nc`` or a similar
+ tool to attempt to connect to each of the other Monitor nodes on ports
+ ``tcp/3300`` and ``tcp/6789``.
+
+#. **Make sure that the "ceph status" command runs and receives a reply from the cluster.**
+
+ If the ``ceph status`` command receives a reply from the cluster, then the
+ cluster is up and running. Monitors answer to a ``status`` request only if
+ there is a formed quorum. Confirm that one or more ``mgr`` daemons are
+ reported as running. In a cluster with no deficiencies, ``ceph status``
+ will report that all ``mgr`` daemons are running.
+
+ If the ``ceph status`` command does not receive a reply from the cluster,
+ then there are probably not enough Monitors ``up`` to form a quorum. If the
+ ``ceph -s`` command is run with no further options specified, it connects
+ to an arbitrarily selected Monitor. In certain cases, however, it might be
+ helpful to connect to a specific Monitor (or to several specific Monitors
+ in sequence) by adding the ``-m`` flag to the command: for example, ``ceph
+ status -m mymon1``.
+
+#. **None of this worked. What now?**
+
+ If the above solutions have not resolved your problems, you might find it
+ helpful to examine each individual Monitor in turn. Even if no quorum has
+ been formed, it is possible to contact each Monitor individually and
+ request its status by using the ``ceph tell mon.ID mon_status`` command
+ (here ``ID`` is the Monitor's identifier).
+
+ Run the ``ceph tell mon.ID mon_status`` command for each Monitor in the
+ cluster. For more on this command's output, see :ref:`Understanding
+ mon_status
+ <rados_troubleshoting_troubleshooting_mon_understanding_mon_status>`.
+
+ There is also an alternative method for contacting each individual Monitor:
+ SSH into each Monitor node and query the daemon's admin socket. See
+ :ref:`Using the Monitor's Admin
+ Socket<rados_troubleshoting_troubleshooting_mon_using_admin_socket>`.
.. _rados_troubleshoting_troubleshooting_mon_using_admin_socket:
``IP:PORT`` combination, the **lower** the rank. In this case, because
``127.0.0.1:6789`` is lower than the other two ``IP:PORT`` combinations,
``mon.a`` has the highest rank: namely, rank 0.
-
+
Most Common Monitor Issues
===========================
-Have Quorum but at least one Monitor is down
----------------------------------------------
+The Cluster Has Quorum but at Least One Monitor is Down
+-------------------------------------------------------
-When this happens, depending on the version of Ceph you are running,
-you should be seeing something similar to::
+When the cluster has quorum but at least one monitor is down, ``ceph health
+detail`` returns a message similar to the following::
$ ceph health detail
[snip]
mon.a (rank 0) addr 127.0.0.1:6789/0 is down (out of quorum)
-How to troubleshoot this?
+**How do I troubleshoot a Ceph cluster that has quorum but also has at least one monitor down?**
- First, make sure ``mon.a`` is running.
+ #. Make sure that ``mon.a`` is running.
- Second, make sure you are able to connect to ``mon.a``'s node from the
- other mon nodes. Check the TCP ports as well. Check ``iptables`` and
- ``nf_conntrack`` on all nodes and ensure that you are not
- dropping/rejecting connections.
+ #. Make sure that you can connect to ``mon.a``'s node from the
+ other Monitor nodes. Check the TCP ports as well. Check ``iptables`` and
+ ``nf_conntrack`` on all nodes and make sure that you are not
+ dropping/rejecting connections.
- If this initial troubleshooting doesn't solve your problems, then it's
- time to go deeper.
+ If this initial troubleshooting doesn't solve your problem, then further
+ investigation is necessary.
First, check the problematic monitor's ``mon_status`` via the admin
socket as explained in `Using the monitor's admin socket`_ and
`Understanding mon_status`_.
- If the monitor is out of the quorum, its state should be one of ``probing``,
- ``electing`` or ``synchronizing``. If it happens to be either ``leader`` or
- ``peon``, then the monitor believes to be in quorum, while the remaining
- cluster is sure it is not; or maybe it got into the quorum while we were
- troubleshooting the monitor, so check you ``ceph status`` again just to make
- sure. Proceed if the monitor is not yet in the quorum.
-
-What if the state is ``probing``?
-
- This means the monitor is still looking for the other monitors. Every time
- you start a monitor, the monitor will stay in this state for some time while
- trying to connect the rest of the monitors specified in the ``monmap``. The
- time a monitor will spend in this state can vary. For instance, when on a
- single-monitor cluster (never do this in production), the monitor will pass
- through the probing state almost instantaneously. In a multi-monitor
- cluster, the monitors will stay in this state until they find enough monitors
- to form a quorum |---| this means that if you have 2 out of 3 monitors down, the
- one remaining monitor will stay in this state indefinitely until you bring
- one of the other monitors up.
-
- If you have a quorum the starting daemon should be able to find the
- other monitors quickly, as long as they can be reached. If your
- monitor is stuck probing and you have gone through with all the communication
- troubleshooting, then there is a fair chance that the monitor is trying
- to reach the other monitors on a wrong address. ``mon_status`` outputs the
- ``monmap`` known to the monitor: check if the other monitor's locations
- match reality. If they don't, jump to
- `Recovering a Monitor's Broken monmap`_; if they do, then it may be related
- to severe clock skews amongst the monitor nodes and you should refer to
- `Clock Skews`_ first, but if that doesn't solve your problem then it is
- the time to prepare some logs and reach out to the community (please refer
- to `Preparing your logs`_ on how to best prepare your logs).
-
-
-What if state is ``electing``?
-
- This means the monitor is in the middle of an election. With recent Ceph
- releases these typically complete quickly, but at times the monitors can
- get stuck in what is known as an *election storm*. This can indicate
- clock skew among the monitor nodes; jump to
- `Clock Skews`_ for more information. If all your clocks are properly
- synchronized, you should search the mailing lists and tracker.
- This is not a state that is likely to persist and aside from
- (*really*) old bugs there is not an obvious reason besides clock skews on
- why this would happen. Worst case, if there are enough surviving mons,
- down the problematic one while you investigate.
-
-What if state is ``synchronizing``?
-
- This means the monitor is catching up with the rest of the cluster in
- order to join the quorum. Time to synchronize is a function of the size
- of your monitor store and thus of cluster size and state, so if you have a
- large or degraded cluster this may take a while.
-
- If you notice that the monitor jumps from ``synchronizing`` to
- ``electing`` and then back to ``synchronizing``, then you do have a
- problem: the cluster state may be advancing (i.e., generating new maps)
- too fast for the synchronization process to keep up. This was a more common
- thing in early days (Cuttlefish), but since then the synchronization process
- has been refactored and enhanced to avoid this dynamic. If you experience
- this in later versions please let us know via a bug tracker. And bring some logs
- (see `Preparing your logs`_).
-
-What if state is ``leader`` or ``peon``?
-
- This should not happen: famous last words. If it does, however, it likely
- has a lot to do with clock skew -- see `Clock Skews`_. If you are not
- suffering from clock skew, then please prepare your logs (see
- `Preparing your logs`_) and reach out to the community.
+ If the Monitor is out of the quorum, then its state will be one of the
+ following: ``probing``, ``electing`` or ``synchronizing``. If the state of
+ the Monitor is ``leader`` or ``peon``, then the Monitor believes itself to be
+ in quorum but the rest of the cluster believes that it is not in quorum. It
+ is possible that a Monitor that is in one of the ``probing``, ``electing``,
+ or ``synchronizing`` states has entered the quorum during the process of
+ troubleshooting. Check ``ceph status`` again to determine whether the Monitor
+ has entered quorum during your troubleshooting. If the Monitor remains out of
+ the quorum, then proceed with the investigations described in this section of
+ the documentation.
+
+
+**What does it mean when a Monitor's state is ``probing``?**
+
+ If ``ceph health detail`` shows that a Monitor's state is
+ ``probing``, then the Monitor is still looking for the other Monitors. Every
+ Monitor remains in this state for some time when it is started. When a
+ Monitor has connected to the other Monitors specified in the ``monmap``, it
+ ceases to be in the ``probing`` state. The amount of time that a Monitor is
+ in the ``probing`` state depends upon the parameters of the cluster of which
+ it is a part. For example, when a Monitor is a part of a single-monitor
+ cluster (never do this in production), the monitor passes through the probing
+ state almost instantaneously. In a multi-monitor cluster, the Monitors stay
+ in the ``probing`` state until they find enough monitors to form a quorum
+ |---| this means that if two out of three Monitors in the cluster are
+ ``down``, the one remaining Monitor stays in the ``probing`` state
+ indefinitely until you bring one of the other monitors up.
+
+ If quorum has been established, then the Monitor daemon should be able to
+ find the other Monitors quickly, as long as they can be reached. If a Monitor
+ is stuck in the ``probing`` state and you have exhausted the procedures above
+ that describe the troubleshooting of communications between the Monitors,
+ then it is possible that the problem Monitor is trying to reach the other
+ Monitors at a wrong address. ``mon_status`` outputs the ``monmap`` that is
+ known to the monitor: determine whether the other Monitors' locations as
+ specified in the ``monmap`` match the locations of the Monitors in the
+ network. If they do not, see `Recovering a Monitor's Broken monmap`_.
+ If the locations of the Monitors as specified in the ``monmap`` match the
+ locations of the Monitors in the network, then the persistent
+ ``probing`` state could be related to severe clock skews amongst the monitor
+ nodes. See `Clock Skews`_. If the information in `Clock Skews`_ does not
+ bring the Monitor out of the ``probing`` state, then prepare your system logs
+ and ask the Ceph community for help. See `Preparing your logs`_ for
+ information about the proper preparation of logs.
+
+
+**What does it mean when a Monitor's state is ``electing``?**
+
+ If ``ceph health detail`` shows that a Monitor's state is ``electing``, the
+ monitor is in the middle of an election. Elections typically complete
+ quickly, but sometimes the monitors can get stuck in what is known as an
+ *election storm*. See :ref:`Monitor Elections <dev_mon_elections>` for more
+ on monitor elections.
+
+ The presence of election storm might indicate clock skew among the monitor
+ nodes. See `Clock Skews`_ for more information.
+
+ If your clocks are properly synchronized, search the mailing lists and bug
+ tracker for issues similar to your issue. The ``electing`` state is not
+ likely to persist. In versions of Ceph after the release of Cuttlefish, there
+ is no obvious reason other than clock skew that explains why an ``electing``
+ state would persist.
+
+ It is possible to investigate the cause of a persistent ``electing`` state if
+ you put the problematic Monitor into a ``down`` state while you investigate.
+ This is possible only if there are enough surviving Monitors to form quorum.
+
+**What does it mean when a Monitor's state is ``synchronizing``?**
+
+ If ``ceph health detail`` shows that the Monitor is ``synchronizing``, the
+ monitor is catching up with the rest of the cluster so that it can join the
+ quorum. The amount of time that it takes for the Monitor to synchronize with
+ the rest of the quorum is a function of the size of the cluster's monitor
+ store, the cluster's size, and the state of the cluster. Larger and degraded
+ clusters generally keep Monitors in the ``synchronizing`` state longer than
+ do smaller, new clusters.
+
+ A Monitor that changes its state from ``synchronizing`` to ``electing`` and
+ then back to ``synchronizing`` indicates a problem: the cluster state may be
+ advancing (that is, generating new maps) too fast for the synchronization
+ process to keep up with the pace of the creation of the new maps. This issue
+ presented more frequently prior to the Cuttlefish release than it does in
+ more recent releases, because the synchronization process has since been
+ refactored and enhanced to avoid this dynamic. If you experience this in
+ later versions, report the issue in the `Ceph bug tracker
+ <https://tracker.ceph.com>`_. Prepare and provide logs to substantiate any
+ bug you raise. See `Preparing your logs`_ for information about the proper
+ preparation of logs.
+
+**What does it mean when a Monitor's state is ``leader`` or ``peon``?**
+
+ If ``ceph health detail`` shows that the Monitor is in the ``leader`` state
+ or in the ``peon`` state, it is likely that clock skew is present. Follow the
+ instructions in `Clock Skews`_. If you have followed those instructions and
+ ``ceph health detail`` still shows that the Monitor is in the ``leader``
+ state or the ``peon`` state, report the issue in the `Ceph bug tracker
+ <https://tracker.ceph.com>`_. If you raise an issue, provide logs to
+ substantiate it. See `Preparing your logs`_ for information about the
+ proper preparation of logs.
Recovering a Monitor's Broken ``monmap``
Inject a monmap into the monitor
- Usually the safest path. You should grab the monmap from the remaining
- monitors and inject it into the monitor with the corrupted/lost monmap.
-
These are the basic steps:
- 1. Is there a formed quorum? If so, grab the monmap from the quorum::
+ Retrieve the ``monmap`` from the surviving monitors and inject it into the
+ monitor whose ``monmap`` is corrupted or lost.
+
+ Implement this solution by carrying out the following procedure:
+
+ 1. Is there a quorum of monitors? If so, retrieve the ``monmap`` from the
+ quorum::
$ ceph mon getmap -o /tmp/monmap
- 2. No quorum? Grab the monmap directly from another monitor (this
- assumes the monitor you are grabbing the monmap from has id ID-FOO
- and has been stopped)::
+ 2. If there is no quorum, then retrieve the ``monmap`` directly from another
+ monitor that has been stopped (in this example, the other monitor has
+ the ID ``ID-FOO``)::
$ ceph-mon -i ID-FOO --extract-monmap /tmp/monmap
5. Start the monitor
- Please keep in mind that the ability to inject monmaps is a powerful
- feature that can cause havoc with your monitors if misused as it will
- overwrite the latest, existing monmap kept by the monitor.
-
+ .. warning:: Injecting ``monmaps`` can cause serious problems because doing
+ so will overwrite the latest existing ``monmap`` stored on the monitor. Be
+ careful!
Clock Skews
-------------
+-----------
-Monitor operation can be severely affected by clock skew among the quorum's
-mons, as the PAXOS consensus algorithm requires tight time alignment.
-Skew can result in weird behavior with no obvious
-cause. To avoid such issues, you must run a clock synchronization tool
-on your monitor nodes: ``Chrony`` or the legacy ``ntpd``. Be sure to
-configure the mon nodes with the `iburst` option and multiple peers:
+The Paxos consensus algorithm requires close time synchroniziation, which means
+that clock skew among the monitors in the quorum can have a serious effect on
+monitor operation. The resulting behavior can be puzzling. To avoid this issue,
+run a clock synchronization tool on your monitor nodes: for example, use
+``Chrony`` or the legacy ``ntpd`` utility. Configure each monitor nodes so that
+the `iburst` option is in effect and so that each monitor has multiple peers,
+including the following:
* Each other
* Internal ``NTP`` servers
* Multiple external, public pool servers
-For good measure, *all* nodes in your cluster should also sync against
-internal and external servers, and perhaps even your mons. ``NTP`` servers
-should run on bare metal; VM virtualized clocks are not suitable for steady
-timekeeping. Visit `https://www.ntp.org <https://www.ntp.org>`_ for more info. Your
-organization may already have quality internal ``NTP`` servers you can use.
-Sources for ``NTP`` server appliances include:
+.. note:: The ``iburst`` option sends a burst of eight packets instead of the
+ usual single packet, and is used during the process of getting two peers
+ into initial synchronization.
+
+Furthermore, it is advisable to synchronize *all* nodes in your cluster against
+internal and external servers, and perhaps even against your monitors. Run
+``NTP`` servers on bare metal: VM-virtualized clocks are not suitable for
+steady timekeeping. See `https://www.ntp.org <https://www.ntp.org>`_ for more
+information about the Network Time Protocol (NTP). Your organization might
+already have quality internal ``NTP`` servers available. Sources for ``NTP``
+server appliances include the following:
* Microsemi (formerly Symmetricom) `https://microsemi.com <https://www.microsemi.com/product-directory/3425-timing-synchronization>`_
* EndRun `https://endruntechnologies.com <https://endruntechnologies.com/products/ntp-time-servers>`_
* Netburner `https://www.netburner.com <https://www.netburner.com/products/network-time-server/pk70-ex-ntp-network-time-server>`_
+Clock Skew Questions and Answers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-What's the maximum tolerated clock skew?
-
- By default the monitors will allow clocks to drift up to 0.05 seconds (50 ms).
+**What's the maximum tolerated clock skew?**
+ By default, monitors allow clocks to drift up to a maximum of 0.05 seconds
+ (50 milliseconds).
-Can I increase the maximum tolerated clock skew?
+**Can I increase the maximum tolerated clock skew?**
- The maximum tolerated clock skew is configurable via the
- ``mon-clock-drift-allowed`` option, and
- although you *CAN* you almost certainly *SHOULDN'T*. The clock skew mechanism
- is in place because clock-skewed monitors are likely to misbehave. We, as
- developers and QA aficionados, are comfortable with the current default
- value, as it will alert the user before the monitors get out hand. Changing
- this value may cause unforeseen effects on the
- stability of the monitors and overall cluster health.
+ Yes, but we strongly recommend against doing so. The maximum tolerated clock
+ skew is configurable via the ``mon-clock-drift-allowed`` option, but it is
+ almost certainly a bad idea to make changes to this option. The clock skew
+ maximum is in place because clock-skewed monitors cannot be relied upon. The
+ current default value has proven its worth at alerting the user before the
+ monitors encounter serious problems. Changing this value might cause
+ unforeseen effects on the stability of the monitors and overall cluster
+ health.
-How do I know there's a clock skew?
+**How do I know whether there is a clock skew?**
- The monitors will warn you via the cluster status ``HEALTH_WARN``. ``ceph
- health detail`` or ``ceph status`` should show something like::
+ The monitors will warn you via the cluster status ``HEALTH_WARN``. When clock
+ skew is present, the ``ceph health detail`` and ``ceph status`` commands
+ return an output resembling the following::
mon.c addr 10.10.0.1:6789/0 clock skew 0.08235s > max 0.05s (latency 0.0045s)
- That means that ``mon.c`` has been flagged as suffering from a clock skew.
-
- On releases beginning with Luminous you can issue the ``ceph
- time-sync-status`` command to check status. Note that the lead mon is
- typically the one with the numerically lowest IP address. It will always
- show ``0``: the reported offsets of other mons are relative to the lead mon,
- not to any external reference source.
+ In this example, the monitor ``mon.c`` has been flagged as suffering from
+ clock skew.
+ In Luminous and later releases, it is possible to check for a clock skew by
+ running the ``ceph time-sync-status`` command. Note that the lead monitor
+ typically has the numerically lowest IP address. It will always show ``0``:
+ the reported offsets of other monitors are relative to the lead monitor, not
+ to any external reference source.
-What should I do if there's a clock skew?
+**What should I do if there is a clock skew?**
- Synchronize your clocks. Running an NTP client may help. If you are already
- using one and you hit this sort of issues, check if you are using some NTP
- server remote to your network and consider hosting your own NTP server on
- your network. This last option tends to reduce the amount of issues with
- monitor clock skews.
+ Synchronize your clocks. Using an NTP client might help. However, if you
+ are already using an NTP client and you still encounter clock skew problems,
+ determine whether the NTP server that you are using is remote to your network
+ or instead hosted on your network. Hosting your own NTP servers tends to
+ mitigate clock skew problems.
Client Can't Connect or Mount
-------------------------------
+-----------------------------
-Check your IP tables. Some OS install utilities add a ``REJECT`` rule to
-``iptables``. The rule rejects all clients trying to connect to the host except
-for ``ssh``. If your monitor host's IP tables have such a ``REJECT`` rule in
-place, clients connecting from a separate node will fail to mount with a timeout
-error. You need to address ``iptables`` rules that reject clients trying to
-connect to Ceph daemons. For example, you would need to address rules that look
-like this appropriately::
+Check your IP tables. Some operating-system install utilities add a ``REJECT``
+rule to ``iptables``. ``iptables`` rules will reject all clients other than
+``ssh`` that try to connect to the host. If your monitor host's IP tables have
+a ``REJECT`` rule in place, clients that are connecting from a separate node
+will fail and will raise a timeout error. Any ``iptables`` rules that reject
+clients trying to connect to Ceph daemons must be addressed. For example::
- REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
+ REJECT all -- anywhere anywhere reject-with icmp-host-prohibited
-You may also need to add rules to IP tables on your Ceph hosts to ensure
-that clients can access the ports associated with your Ceph monitors (i.e., port
-6789 by default) and Ceph OSDs (i.e., 6800 through 7300 by default). For
+It might also be necessary to add rules to iptables on your Ceph hosts to
+ensure that clients are able to access the TCP ports associated with your Ceph
+monitors (default: port 6789) and Ceph OSDs (default: 6800 through 7300). For
example::
- iptables -A INPUT -m multiport -p tcp -s {ip-address}/{netmask} --dports 6789,6800:7300 -j ACCEPT
+ iptables -A INPUT -m multiport -p tcp -s {ip-address}/{netmask} --dports 6789,6800:7300 -j ACCEPT
+
Monitor Store Failures
======================
Symptoms of store corruption
----------------------------
-Ceph monitor stores the :term:`Cluster Map` in a key/value store such as LevelDB. If
-a monitor fails due to the key/value store corruption, following error messages
-might be found in the monitor log::
+Ceph monitors store the :term:`Cluster Map` in a key-value store. If key-value
+store corruption causes a monitor to fail, then the monitor log might contain
+one of the following error messages::
Corruption: error in middle of record
Recovery using healthy monitor(s)
---------------------------------
-If there are any survivors, we can always :ref:`replace <adding-and-removing-monitors>` the corrupted one with a
-new one. After booting up, the new joiner will sync up with a healthy
-peer, and once it is fully sync'ed, it will be able to serve the clients.
+If there are surviving monitors, we can always :ref:`replace
+<adding-and-removing-monitors>` the corrupted monitor with a new one. After the
+new monitor boots, it will synchronize with a healthy peer. After the new
+monitor is fully synchronized, it will be able to serve clients.
.. _mon-store-recovery-using-osds:
Recovery using OSDs
-------------------
-But what if all monitors fail at the same time? Since users are encouraged to
-deploy at least three (and preferably five) monitors in a Ceph cluster, the chance of simultaneous
-failure is rare. But unplanned power-downs in a data center with improperly
-configured disk/fs settings could fail the underlying file system, and hence
-kill all the monitors. In this case, we can recover the monitor store with the
-information stored in OSDs.
+Even if all monitors fail at the same time, it is possible to recover the
+monitor store by using information stored in OSDs. You are encouraged to deploy
+at least three (and preferably five) monitors in a Ceph cluster. In such a
+deployment, complete monitor failure is unlikely. However, unplanned power loss
+in a data center whose disk settings or filesystem settings are improperly
+configured could cause the underlying filesystem to fail and this could kill
+all of the monitors. In such a case, data in the OSDs can be used to recover
+the monitors. The following is such a script and can be used to recover the
+monitors:
+
.. code-block:: bash
mv $ms/store.db /var/lib/ceph/mon/mon.foo/store.db
chown -R ceph:ceph /var/lib/ceph/mon/mon.foo/store.db
-The steps above
+This script performs the following steps:
+
+#. Collects the map from each OSD host.
+#. Rebuilds the store.
+#. Fills the entities in the keyring file with appropriate capabilities.
+#. Replaces the corrupted store on ``mon.foo`` with the recovered copy.
-#. collect the map from all OSD hosts,
-#. then rebuild the store,
-#. fill the entities in keyring file with appropriate caps
-#. replace the corrupted store on ``mon.foo`` with the recovered copy.
Known limitations
~~~~~~~~~~~~~~~~~
-Following information are not recoverable using the steps above:
+The above recovery tool is unable to recover the following information:
-- **some added keyrings**: all the OSD keyrings added using ``ceph auth add`` command
- are recovered from the OSD's copy. And the ``client.admin`` keyring is imported
- using ``ceph-monstore-tool``. But the MDS keyrings and other keyrings are missing
- in the recovered monitor store. You might need to re-add them manually.
+- **Certain added keyrings**: All of the OSD keyrings added using the ``ceph
+ auth add`` command are recovered from the OSD's copy, and the
+ ``client.admin`` keyring is imported using ``ceph-monstore-tool``. However,
+ the MDS keyrings and all other keyrings will be missing in the recovered
+ monitor store. You might need to manually re-add them.
-- **creating pools**: If any RADOS pools were in the process of being creating, that state is lost. The recovery tool assumes that all pools have been created. If there are PGs that are stuck in the 'unknown' after the recovery for a partially created pool, you can force creation of the *empty* PG with the ``ceph osd force-create-pg`` command. Note that this will create an *empty* PG, so only do this if you know the pool is empty.
-
-- **MDS Maps**: the MDS maps are lost.
+- **Creating pools**: If any RADOS pools were in the process of being created,
+ that state is lost. The recovery tool operates on the assumption that all
+ pools have already been created. If there are PGs that are stuck in the
+ 'unknown' state after the recovery for a partially created pool, you can
+ force creation of the *empty* PG by running the ``ceph osd force-create-pg``
+ command. Note that this will create an *empty* PG, so take this action only
+ if you know the pool is empty.
+- **MDS Maps**: The MDS maps are lost.
Everything Failed! Now What?
-=============================
+============================
Reaching out for help
-----------------------
+---------------------
-You can find us on IRC at #ceph and #ceph-devel at OFTC (server irc.oftc.net)
-and on ``dev@ceph.io`` and ``ceph-users@lists.ceph.com``. Make
-sure you have grabbed your logs and have them ready if someone asks: the faster
-the interaction and lower the latency in response, the better chances everyone's
-time is optimized.
+You can find help on IRC in #ceph and #ceph-devel on OFTC (server
+irc.oftc.net), or at ``dev@ceph.io`` and ``ceph-users@lists.ceph.com``. Make
+sure that you have prepared your logs and that you have them ready upon
+request.
+
+See https://ceph.io/en/community/connect/ for current (as of October 2023)
+information on getting in contact with the upstream Ceph community.
Preparing your logs
----------------------
+-------------------
-Monitor logs are, by default, kept in ``/var/log/ceph/ceph-mon.FOO.log*``. We
-may want them. However, your logs may not have the necessary information. If
-you don't find your monitor logs at their default location, you can check
-where they should be by running::
+The default location for monitor logs is ``/var/log/ceph/ceph-mon.FOO.log*``.
+However, if they are not there, you can find their current location by running
+the following command:
- ceph-conf --name mon.FOO --show-config-value log_file
+.. prompt:: bash
-The amount of information in the logs are subject to the debug levels being
-enforced by your configuration files. If you have not enforced a specific
-debug level then Ceph is using the default levels and your logs may not
-contain important information to track down you issue.
-A first step in getting relevant information into your logs will be to raise
-debug levels. In this case we will be interested in the information from the
-monitor.
-Similarly to what happens on other components, different parts of the monitor
-will output their debug information on different subsystems.
+ ceph-conf --name mon.FOO --show-config-value log_file
-You will have to raise the debug levels of those subsystems more closely
-related to your issue. This may not be an easy task for someone unfamiliar
-with troubleshooting Ceph. For most situations, setting the following options
-on your monitors will be enough to pinpoint a potential source of the issue::
+The amount of information in the logs is determined by the debug levels in the
+cluster's configuration files. If Ceph is using the default debug levels, then
+your logs might be missing important information that would help the upstream
+Ceph community address your issue.
+
+To make sure your monitor logs contain relevant information, you can raise
+debug levels. Here we are interested in information from the monitors. As with
+other components, the monitors have different parts that output their debug
+information on different subsystems.
+
+If you are an experienced Ceph troubleshooter, we recommend raising the debug
+levels of the most relevant subsystems. Of course, this approach might not be
+easy for beginners. In most cases, however, enough information to address the
+issue will be secured if the following debug levels are entered::
debug_mon = 10
debug_ms = 1
-If we find that these debug levels are not enough, there's a chance we may
-ask you to raise them or even define other debug subsystems to obtain infos
-from -- but at least we started off with some useful information, instead
-of a massively empty log without much to go on with.
+Sometimes these debug levels do not yield enough information. In such cases,
+members of the upstream Ceph community might ask you to make additional changes
+to these or to other debug levels. In any case, it is better for us to receive
+at least some useful information than to receive an empty log.
+
Do I need to restart a monitor to adjust debug levels?
------------------------------------------------------
-No. You may do it in one of two ways:
+No, restarting a monitor is not necessary. Debug levels may be adjusted by
+using two different methods, depending on whether or not there is a quorum:
-You have quorum
+There is a quorum
- Either inject the debug option into the monitor you want to debug::
+ Either inject the debug option into the specific monitor that needs to
+ be debugged::
ceph tell mon.FOO config set debug_mon 10/10
- or into all monitors at once::
+ Or inject it into all monitors at once::
ceph tell mon.* config set debug_mon 10/10
-No quorum
- Use the monitor's admin socket and directly adjust the configuration
- options::
+There is no quorum
+
+ Use the admin socket of the specific monitor that needs to be debugged
+ and directly adjust the monitor's configuration options::
ceph daemon mon.FOO config set debug_mon 10/10
-Going back to default values is as easy as rerunning the above commands
-using the debug level ``1/10`` instead. You can check your current
-values using the admin socket and the following commands::
+To return the debug levels to their default values, run the above commands
+using the debug level ``1/10`` rather than ``10/10``. To check a monitor's
+current values, use the admin socket and run either of the following commands:
- ceph daemon mon.FOO config show
+ .. prompt:: bash
-or::
+ ceph daemon mon.FOO config show
+
+or:
+
+ .. prompt:: bash
+
+ ceph daemon mon.FOO config get 'OPTION_NAME'
- ceph daemon mon.FOO config get 'OPTION_NAME'
-Reproduced the problem with appropriate debug levels. Now what?
-----------------------------------------------------------------
+I Reproduced the problem with appropriate debug levels. Now what?
+-----------------------------------------------------------------
-Ideally you would send us only the relevant portions of your logs.
-We realise that figuring out the corresponding portion may not be the
-easiest of tasks. Therefore, we won't hold it to you if you provide the
-full log, but common sense should be employed. If your log has hundreds
-of thousands of lines, it may get tricky to go through the whole thing,
-specially if we are not aware at which point, whatever your issue is,
-happened. For instance, when reproducing, keep in mind to write down
-current time and date and to extract the relevant portions of your logs
-based on that.
+We prefer that you send us only the portions of your logs that are relevant to
+your monitor problems. Of course, it might not be easy for you to determine
+which portions are relevant so we are willing to accept complete and
+unabridged logs. However, we request that you avoid sending logs containing
+hundreds of thousands of lines with no additional clarifying information. One
+common-sense way of making our task easier is to write down the current time
+and date when you are reproducing the problem and then extract portions of your
+logs based on that information.
-Finally, you should reach out to us on the mailing lists, on IRC or file
-a new issue on the `tracker`_.
+Finally, reach out to us on the mailing lists or IRC or Slack, or by filing a
+new issue on the `tracker`_.
.. _tracker: http://tracker.ceph.com/projects/ceph/issues/new
projects / ceph.git / blobdiff