1 =====================================
2 Configuring Monitor/OSD Interaction
3 =====================================
7 After you have completed your initial Ceph configuration, you may deploy and run
8 Ceph. When you execute a command such as ``ceph health`` or ``ceph -s``, the
9 :term:`Ceph Monitor` reports on the current state of the :term:`Ceph Storage
10 Cluster`. The Ceph Monitor knows about the Ceph Storage Cluster by requiring
11 reports from each :term:`Ceph OSD Daemon`, and by receiving reports from Ceph
12 OSD Daemons about the status of their neighboring Ceph OSD Daemons. If the Ceph
13 Monitor doesn't receive reports, or if it receives reports of changes in the
14 Ceph Storage Cluster, the Ceph Monitor updates the status of the :term:`Ceph
17 Ceph provides reasonable default settings for Ceph Monitor/Ceph OSD Daemon
18 interaction. However, you may override the defaults. The following sections
19 describe how Ceph Monitors and Ceph OSD Daemons interact for the purposes of
20 monitoring the Ceph Storage Cluster.
22 .. index:: heartbeat interval
27 Each Ceph OSD Daemon checks the heartbeat of other Ceph OSD Daemons every 6
28 seconds. You can change the heartbeat interval by adding an ``osd heartbeat
29 interval`` setting under the ``[osd]`` section of your Ceph configuration file,
30 or by setting the value at runtime. If a neighboring Ceph OSD Daemon doesn't
31 show a heartbeat within a 20 second grace period, the Ceph OSD Daemon may
32 consider the neighboring Ceph OSD Daemon ``down`` and report it back to a Ceph
33 Monitor, which will update the Ceph Cluster Map. You may change this grace
34 period by adding an ``osd heartbeat grace`` setting under the ``[mon]``
35 and ``[osd]`` or ``[global]`` section of your Ceph configuration file,
36 or by setting the value at runtime.
39 .. ditaa:: +---------+ +---------+
41 +---------+ +---------+
49 |------------------->|
51 |<-------------------|
60 |------------------->|
71 .. index:: OSD down report
76 By default, a Ceph OSD Daemon must report to the Ceph Monitors that another Ceph
77 OSD Daemon is ``down`` three times before the Ceph Monitors acknowledge that the
78 reported Ceph OSD Daemon is ``down``. By default, only one
79 Ceph OSD Daemon is required to report another Ceph OSD Daemon ``down``. You can
80 change the number of Ceph OSD Daemones required to report a Ceph OSD Daemon
81 ``down`` to a Ceph Monitor by adding an ``mon osd min down reporters`` setting
82 (``osd min down reporters`` prior to v0.62) under the ``[mon]`` section of your
83 Ceph configuration file, or by setting the value at runtime.
86 .. ditaa:: +---------+ +---------+
88 +---------+ +---------+
104 .. index:: peering failure
106 OSDs Report Peering Failure
107 ===========================
109 If a Ceph OSD Daemon cannot peer with any of the Ceph OSD Daemons defined in its
110 Ceph configuration file (or the cluster map), it will ping a Ceph Monitor for
111 the most recent copy of the cluster map every 30 seconds. You can change the
112 Ceph Monitor heartbeat interval by adding an ``osd mon heartbeat interval``
113 setting under the ``[osd]`` section of your Ceph configuration file, or by
114 setting the value at runtime.
116 .. ditaa:: +---------+ +---------+ +-------+ +---------+
117 | OSD 1 | | OSD 2 | | OSD 3 | | Monitor |
118 +---------+ +---------+ +-------+ +---------+
122 |-------------->| | |
123 |<--------------| | |
128 |----------------------------->| |
132 |<---+ Interval Exceeded |
134 | Failed to Peer with OSD 3 |
135 |-------------------------------------------->|
136 |<--------------------------------------------|
137 | Receive New Cluster Map |
140 .. index:: OSD status
142 OSDs Report Their Status
143 ========================
145 If an Ceph OSD Daemon doesn't report to a Ceph Monitor, the Ceph Monitor will
146 consider the Ceph OSD Daemon ``down`` after the ``mon osd report timeout``
147 elapses. A Ceph OSD Daemon sends a report to a Ceph Monitor when a reportable
148 event such as a failure, a change in placement group stats, a change in
149 ``up_thru`` or when it boots within 5 seconds. You can change the Ceph OSD
150 Daemon minimum report interval by adding an ``osd mon report interval min``
151 setting under the ``[osd]`` section of your Ceph configuration file, or by
152 setting the value at runtime. A Ceph OSD Daemon sends a report to a Ceph
153 Monitor every 120 seconds irrespective of whether any notable changes occur.
154 You can change the Ceph Monitor report interval by adding an ``osd mon report
155 interval max`` setting under the ``[osd]`` section of your Ceph configuration
156 file, or by setting the value at runtime.
159 .. ditaa:: +---------+ +---------+
160 | OSD 1 | | Monitor |
161 +---------+ +---------+
173 |------------------->|
181 |------------------->|
197 Configuration Settings
198 ======================
200 When modifying heartbeat settings, you should include them in the ``[global]``
201 section of your configuration file.
203 .. index:: monitor heartbeat
208 ``mon osd min up ratio``
210 :Description: The minimum ratio of ``up`` Ceph OSD Daemons before Ceph will
211 mark Ceph OSD Daemons ``down``.
217 ``mon osd min in ratio``
219 :Description: The minimum ratio of ``in`` Ceph OSD Daemons before Ceph will
220 mark Ceph OSD Daemons ``out``.
226 ``mon osd laggy halflife``
228 :Description: The number of seconds laggy estimates will decay.
233 ``mon osd laggy weight``
235 :Description: The weight for new samples in laggy estimation decay.
240 ``mon osd adjust heartbeat grace``
242 :Description: If set to ``true``, Ceph will scale based on laggy estimations.
247 ``mon osd adjust down out interval``
249 :Description: If set to ``true``, Ceph will scaled based on laggy estimations.
254 ``mon osd auto mark in``
256 :Description: Ceph will mark any booting Ceph OSD Daemons as ``in``
257 the Ceph Storage Cluster.
263 ``mon osd auto mark auto out in``
265 :Description: Ceph will mark booting Ceph OSD Daemons auto marked ``out``
266 of the Ceph Storage Cluster as ``in`` the cluster.
272 ``mon osd auto mark new in``
274 :Description: Ceph will mark booting new Ceph OSD Daemons as ``in`` the
275 Ceph Storage Cluster.
281 ``mon osd down out interval``
283 :Description: The number of seconds Ceph waits before marking a Ceph OSD Daemon
284 ``down`` and ``out`` if it doesn't respond.
286 :Type: 32-bit Integer
290 ``mon osd down out subtree limit``
292 :Description: The smallest :term:`CRUSH` unit type that Ceph will **not**
293 automatically mark out. For instance, if set to ``host`` and if
294 all OSDs of a host are down, Ceph will not automatically mark out
301 ``mon osd report timeout``
303 :Description: The grace period in seconds before declaring
304 unresponsive Ceph OSD Daemons ``down``.
306 :Type: 32-bit Integer
309 ``mon osd min down reporters``
311 :Description: The minimum number of Ceph OSD Daemons required to report a
312 ``down`` Ceph OSD Daemon.
314 :Type: 32-bit Integer
318 .. index:: OSD hearbeat
323 ``osd heartbeat address``
325 :Description: An Ceph OSD Daemon's network address for heartbeats.
327 :Default: The host address.
330 ``osd heartbeat interval``
332 :Description: How often an Ceph OSD Daemon pings its peers (in seconds).
333 :Type: 32-bit Integer
337 ``osd heartbeat grace``
339 :Description: The elapsed time when a Ceph OSD Daemon hasn't shown a heartbeat
340 that the Ceph Storage Cluster considers it ``down``.
341 This setting has to be set in both the [mon] and [osd] or [global]
342 section so that it is read by both the MON and OSD daemons.
344 :Type: 32-bit Integer
348 ``osd mon heartbeat interval``
350 :Description: How often the Ceph OSD Daemon pings a Ceph Monitor if it has no
351 Ceph OSD Daemon peers.
353 :Type: 32-bit Integer
357 ``osd mon report interval max``
359 :Description: The maximum time in seconds that a Ceph OSD Daemon can wait before
360 it must report to a Ceph Monitor.
362 :Type: 32-bit Integer
366 ``osd mon report interval min``
368 :Description: The minimum number of seconds a Ceph OSD Daemon may wait
369 from startup or another reportable event before reporting
372 :Type: 32-bit Integer
374 :Valid Range: Should be less than ``osd mon report interval max``
377 ``osd mon ack timeout``
379 :Description: The number of seconds to wait for a Ceph Monitor to acknowledge a
380 request for statistics.
382 :Type: 32-bit Integer