ceph/doc/cephfs/health-messages.rst

   1
   2 .. _cephfs-health-messages:
   3
   4 ======================
   5 CephFS health messages
   6 ======================
   7
   8 Cluster health checks
   9 =====================
  10
  11 The Ceph monitor daemons will generate health messages in response
  12 to certain states of the file system map structure (and the enclosed MDS maps).
  13
  14 Message: mds rank(s) *ranks* have failed
  15 Description: One or more MDS ranks are not currently assigned to
  16 an MDS daemon; the cluster will not recover until a suitable replacement
  17 daemon starts.
  18
  19 Message: mds rank(s) *ranks* are damaged
  20 Description: One or more MDS ranks has encountered severe damage to
  21 its stored metadata, and cannot start again until it is repaired.
  22
  23 Message: mds cluster is degraded
  24 Description: One or more MDS ranks are not currently up and running, clients
  25 may pause metadata IO until this situation is resolved.  This includes
  26 ranks being failed or damaged, and additionally includes ranks
  27 which are running on an MDS but have not yet made it to the *active*
  28 state (e.g. ranks currently in *replay* state).
  29
  30 Message: mds *names* are laggy
  31 Description: The named MDS daemons have failed to send beacon messages
  32 to the monitor for at least ``mds_beacon_grace`` (default 15s), while
  33 they are supposed to send beacon messages every ``mds_beacon_interval``
  34 (default 4s).  The daemons may have crashed.  The Ceph monitor will
  35 automatically replace laggy daemons with standbys if any are available.
  36
  37 Message: insufficient standby daemons available
  38 Description: One or more file systems are configured to have a certain number
  39 of standby daemons available (including daemons in standby-replay) but the
  40 cluster does not have enough standby daemons. The standby daemons not in replay
  41 count towards any file system (i.e. they may overlap). This warning can
  42 configured by setting ``ceph fs set <fs> standby_count_wanted <count>``.  Use
  43 zero for ``count`` to disable.
  44
  45
  46 Daemon-reported health checks
  47 =============================
  48
  49 MDS daemons can identify a variety of unwanted conditions, and
  50 indicate these to the operator in the output of ``ceph status``.
  51 These conditions have human readable messages, and additionally
  52 a unique code starting with ``MDS_``.
  53
  54 .. highlight:: console
  55
  56 ``ceph health detail`` shows the details of the conditions. Following
  57 is a typical health report from a cluster experiencing MDS related
  58 performance issues::
  59
  60   $ ceph health detail
  61   HEALTH_WARN 1 MDSs report slow metadata IOs; 1 MDSs report slow requests
  62   MDS_SLOW_METADATA_IO 1 MDSs report slow metadata IOs
  63      mds.fs-01(mds.0): 3 slow metadata IOs are blocked > 30 secs, oldest blocked for 51123 secs
  64   MDS_SLOW_REQUEST 1 MDSs report slow requests
  65      mds.fs-01(mds.0): 5 slow requests are blocked > 30 secs
  66
  67 Where, for instance, ``MDS_SLOW_REQUEST`` is the unique code representing the
  68 condition where requests are taking long time to complete. And the following
  69 description shows its severity and the MDS daemons which are serving these
  70 slow requests.
  71
  72 This page lists the health checks raised by MDS daemons. For the checks from
  73 other daemons, please see :ref:`health-checks`.
  74
  75 ``MDS_TRIM``
  76 ------------
  77
  78   Message
  79     "Behind on trimming..."
  80   Description
  81     CephFS maintains a metadata journal that is divided into
  82     *log segments*.  The length of journal (in number of segments) is controlled
  83     by the setting ``mds_log_max_segments``, and when the number of segments
  84     exceeds that setting the MDS starts writing back metadata so that it
  85     can remove (trim) the oldest segments.  If this writeback is happening
  86     too slowly, or a software bug is preventing trimming, then this health
  87     message may appear.  The threshold for this message to appear is controlled by
  88     the config option ``mds_log_warn_factor``, the default is 2.0.
  89
  90 ``MDS_HEALTH_CLIENT_LATE_RELEASE``, ``MDS_HEALTH_CLIENT_LATE_RELEASE_MANY``
  91 ---------------------------------------------------------------------------
  92
  93   Message
  94     "Client *name* failing to respond to capability release"
  95   Description
  96     CephFS clients are issued *capabilities* by the MDS, which
  97     are like locks.  Sometimes, for example when another client needs access,
  98     the MDS will request clients release their capabilities.  If the client
  99     is unresponsive or buggy, it might fail to do so promptly or fail to do
 100     so at all.  This message appears if a client has taken longer than
 101     ``session_timeout`` (default 60s) to comply.
 102
 103 ``MDS_CLIENT_RECALL``, ``MDS_HEALTH_CLIENT_RECALL_MANY``
 104 --------------------------------------------------------
 105
 106   Message
 107     "Client *name* failing to respond to cache pressure"
 108   Description
 109     Clients maintain a metadata cache.  Items (such as inodes) in the
 110     client cache are also pinned in the MDS cache, so when the MDS needs to shrink
 111     its cache (to stay within ``mds_cache_memory_limit``), it sends messages to
 112     clients to shrink their caches too.  If the client is unresponsive or buggy,
 113     this can prevent the MDS from properly staying within its cache limits and it
 114     may eventually run out of memory and crash.  This message appears if a client
 115     has failed to release more than
 116     ``mds_recall_warning_threshold`` capabilities (decaying with a half-life of
 117     ``mds_recall_max_decay_rate``) within the last
 118     ``mds_recall_warning_decay_rate`` second.
 119
 120 ``MDS_CLIENT_OLDEST_TID``, ``MDS_CLIENT_OLDEST_TID_MANY``
 121 ---------------------------------------------------------
 122
 123   Message
 124     "Client *name* failing to advance its oldest client/flush tid"
 125   Description
 126     The CephFS client-MDS protocol uses a field called the
 127     *oldest tid* to inform the MDS of which client requests are fully
 128     complete and may therefore be forgotten about by the MDS.  If a buggy
 129     client is failing to advance this field, then the MDS may be prevented
 130     from properly cleaning up resources used by client requests.  This message
 131     appears if a client appears to have more than ``max_completed_requests``
 132     (default 100000) requests that are complete on the MDS side but haven't
 133     yet been accounted for in the client's *oldest tid* value.
 134
 135 ``MDS_DAMAGE``
 136 --------------
 137
 138   Message
 139     "Metadata damage detected"
 140   Description
 141     Corrupt or missing metadata was encountered when reading
 142     from the metadata pool.  This message indicates that the damage was
 143     sufficiently isolated for the MDS to continue operating, although
 144     client accesses to the damaged subtree will return IO errors.  Use
 145     the ``damage ls`` admin socket command to get more detail on the damage.
 146     This message appears as soon as any damage is encountered.
 147
 148 ``MDS_HEALTH_READ_ONLY``
 149 ------------------------
 150
 151   Message
 152     "MDS in read-only mode"
 153   Description
 154     The MDS has gone into readonly mode and will return EROFS
 155     error codes to client operations that attempt to modify any metadata.  The
 156     MDS will go into readonly mode if it encounters a write error while
 157     writing to the metadata pool, or if forced to by an administrator using
 158     the *force_readonly* admin socket command.
 159
 160 ``MDS_SLOW_REQUEST``
 161 --------------------
 162
 163   Message
 164     "*N* slow requests are blocked"
 165
 166   Description
 167     One or more client requests have not been completed promptly,
 168     indicating that the MDS is either running very slowly, or that the RADOS
 169     cluster is not acknowledging journal writes promptly, or that there is a bug.
 170     Use the ``ops`` admin socket command to list outstanding metadata operations.
 171     This message appears if any client requests have taken longer than
 172     ``mds_op_complaint_time`` (default 30s).
 173
 174 ``MDS_CACHE_OVERSIZED``
 175 -----------------------
 176
 177   Message
 178     "Too many inodes in cache"
 179   Description
 180     The MDS is not succeeding in trimming its cache to comply with the
 181     limit set by the administrator.  If the MDS cache becomes too large, the daemon
 182     may exhaust available memory and crash.  By default, this message appears if
 183     the actual cache size (in memory) is at least 50% greater than
 184     ``mds_cache_memory_limit`` (default 4GB). Modify ``mds_health_cache_threshold``
 185     to set the warning ratio.
 186
 187 ``FS_WITH_FAILED_MDS``
 188 ----------------------
 189
 190   Message
 191     "Some MDS ranks do not have standby replacements"
 192
 193   Description
 194     Normally, a failed MDS rank will be replaced by a standby MDS. This situation
 195     is transient and is not considered critical. However, if there are no standby
 196     MDSs available to replace an active MDS rank, this health warning is generated.
 197
 198 ``MDS_INSUFFICIENT_STANDBY``
 199 ----------------------------
 200
 201   Message
 202     "Insufficient number of available standby(-replay) MDS daemons than configured"
 203
 204   Description
 205     The minimum number of standby(-replay) MDS daemons can be configured by setting
 206     ``standby_count_wanted`` configuration variable. This health warning is generated
 207     when the configured value mismatches the number of standby(-replay) MDS daemons
 208     available.
 209
 210 ``FS_DEGRADED``
 211 ----------------------------
 212
 213   Message
 214     "Some MDS ranks have been marked failed or damaged"
 215
 216   Description
 217     When one or more MDS rank ends up in failed or damaged state due to
 218     an unrecoverable error. The file system may be partially or fully
 219     unavailable when one (or more) ranks are offline.
 220
 221 ``MDS_UP_LESS_THAN_MAX``
 222 ----------------------------
 223
 224   Message
 225     "Number of active ranks are less than configured number of maximum MDSs"
 226
 227   Description
 228     The maximum number of MDS ranks can be configured by setting ``max_mds``
 229     configuration variable. This health warning is generated when the number
 230     of MDS ranks falls below this configured value.
 231
 232 ``MDS_ALL_DOWN``
 233 ----------------------------
 234
 235   Message
 236     "None of the MDS ranks are available (file system offline)"
 237
 238   Description
 239     All MDS ranks are unavailable resulting in the file system to be completely
 240     offline.