1 ==========================
2 Monitor Config Reference
3 ==========================
5 Understanding how to configure a :term:`Ceph Monitor` is an important part of
6 building a reliable :term:`Ceph Storage Cluster`. **All Ceph Storage Clusters
7 have at least one monitor**. The monitor complement usually remains fairly
8 consistent, but you can add, remove or replace a monitor in a cluster. See
9 `Adding/Removing a Monitor`_ for details.
12 .. index:: Ceph Monitor; Paxos
17 Ceph Monitors maintain a "master copy" of the :term:`Cluster Map`, which means a
18 :term:`Ceph Client` can determine the location of all Ceph Monitors, Ceph OSD
19 Daemons, and Ceph Metadata Servers just by connecting to one Ceph Monitor and
20 retrieving a current cluster map. Before Ceph Clients can read from or write to
21 Ceph OSD Daemons or Ceph Metadata Servers, they must connect to a Ceph Monitor
22 first. With a current copy of the cluster map and the CRUSH algorithm, a Ceph
23 Client can compute the location for any object. The ability to compute object
24 locations allows a Ceph Client to talk directly to Ceph OSD Daemons, which is a
25 very important aspect of Ceph's high scalability and performance. See
26 `Scalability and High Availability`_ for additional details.
28 The primary role of the Ceph Monitor is to maintain a master copy of the cluster
29 map. Ceph Monitors also provide authentication and logging services. Ceph
30 Monitors write all changes in the monitor services to a single Paxos instance,
31 and Paxos writes the changes to a key/value store for strong consistency. Ceph
32 Monitors can query the most recent version of the cluster map during sync
33 operations. Ceph Monitors leverage the key/value store's snapshots and iterators
34 (using leveldb) to perform store-wide synchronization.
37 /-------------\ /-------------\
38 | Monitor | Write Changes | Paxos |
39 | cCCC +-------------->+ cCCC |
41 +-------------+ \------+------/
43 +-------------+ | Write Changes
46 | Monitor Map | /------+------\
47 +-------------+ | Key / Value |
49 +-------------+ | cCCC |
50 | PG Map | \------+------/
52 | MDS Map | | Read Changes
54 | cCCC |*---------------------+
58 .. deprecated:: version 0.58
60 In Ceph versions 0.58 and earlier, Ceph Monitors use a Paxos instance for
61 each service and store the map as a file.
63 .. index:: Ceph Monitor; cluster map
68 The cluster map is a composite of maps, including the monitor map, the OSD map,
69 the placement group map and the metadata server map. The cluster map tracks a
70 number of important things: which processes are ``in`` the Ceph Storage Cluster;
71 which processes that are ``in`` the Ceph Storage Cluster are ``up`` and running
72 or ``down``; whether, the placement groups are ``active`` or ``inactive``, and
73 ``clean`` or in some other state; and, other details that reflect the current
74 state of the cluster such as the total amount of storage space, and the amount
77 When there is a significant change in the state of the cluster--e.g., a Ceph OSD
78 Daemon goes down, a placement group falls into a degraded state, etc.--the
79 cluster map gets updated to reflect the current state of the cluster.
80 Additionally, the Ceph Monitor also maintains a history of the prior states of
81 the cluster. The monitor map, OSD map, placement group map and metadata server
82 map each maintain a history of their map versions. We call each version an
85 When operating your Ceph Storage Cluster, keeping track of these states is an
86 important part of your system administration duties. See `Monitoring a Cluster`_
87 and `Monitoring OSDs and PGs`_ for additional details.
89 .. index:: high availability; quorum
94 Our Configuring ceph section provides a trivial `Ceph configuration file`_ that
95 provides for one monitor in the test cluster. A cluster will run fine with a
96 single monitor; however, **a single monitor is a single-point-of-failure**. To
97 ensure high availability in a production Ceph Storage Cluster, you should run
98 Ceph with multiple monitors so that the failure of a single monitor **WILL NOT**
99 bring down your entire cluster.
101 When a Ceph Storage Cluster runs multiple Ceph Monitors for high availability,
102 Ceph Monitors use `Paxos`_ to establish consensus about the master cluster map.
103 A consensus requires a majority of monitors running to establish a quorum for
104 consensus about the cluster map (e.g., 1; 2 out of 3; 3 out of 5; 4 out of 6;
107 .. confval:: mon_force_quorum_join
109 .. index:: Ceph Monitor; consistency
114 When you add monitor settings to your Ceph configuration file, you need to be
115 aware of some of the architectural aspects of Ceph Monitors. **Ceph imposes
116 strict consistency requirements** for a Ceph monitor when discovering another
117 Ceph Monitor within the cluster. Whereas, Ceph Clients and other Ceph daemons
118 use the Ceph configuration file to discover monitors, monitors discover each
119 other using the monitor map (monmap), not the Ceph configuration file.
121 A Ceph Monitor always refers to the local copy of the monmap when discovering
122 other Ceph Monitors in the Ceph Storage Cluster. Using the monmap instead of the
123 Ceph configuration file avoids errors that could break the cluster (e.g., typos
124 in ``ceph.conf`` when specifying a monitor address or port). Since monitors use
125 monmaps for discovery and they share monmaps with clients and other Ceph
126 daemons, **the monmap provides monitors with a strict guarantee that their
127 consensus is valid.**
129 Strict consistency also applies to updates to the monmap. As with any other
130 updates on the Ceph Monitor, changes to the monmap always run through a
131 distributed consensus algorithm called `Paxos`_. The Ceph Monitors must agree on
132 each update to the monmap, such as adding or removing a Ceph Monitor, to ensure
133 that each monitor in the quorum has the same version of the monmap. Updates to
134 the monmap are incremental so that Ceph Monitors have the latest agreed upon
135 version, and a set of previous versions. Maintaining a history enables a Ceph
136 Monitor that has an older version of the monmap to catch up with the current
137 state of the Ceph Storage Cluster.
139 If Ceph Monitors were to discover each other through the Ceph configuration file
140 instead of through the monmap, additional risks would be introduced because
141 Ceph configuration files are not updated and distributed automatically. Ceph
142 Monitors might inadvertently use an older Ceph configuration file, fail to
143 recognize a Ceph Monitor, fall out of a quorum, or develop a situation where
144 `Paxos`_ is not able to determine the current state of the system accurately.
147 .. index:: Ceph Monitor; bootstrapping monitors
149 Bootstrapping Monitors
150 ----------------------
152 In most configuration and deployment cases, tools that deploy Ceph help
153 bootstrap the Ceph Monitors by generating a monitor map for you (e.g.,
154 ``cephadm``, etc). A Ceph Monitor requires a few explicit
157 - **Filesystem ID**: The ``fsid`` is the unique identifier for your
158 object store. Since you can run multiple clusters on the same
159 hardware, you must specify the unique ID of the object store when
160 bootstrapping a monitor. Deployment tools usually do this for you
161 (e.g., ``cephadm`` can call a tool like ``uuidgen``), but you
162 may specify the ``fsid`` manually too.
164 - **Monitor ID**: A monitor ID is a unique ID assigned to each monitor within
165 the cluster. It is an alphanumeric value, and by convention the identifier
166 usually follows an alphabetical increment (e.g., ``a``, ``b``, etc.). This
167 can be set in a Ceph configuration file (e.g., ``[mon.a]``, ``[mon.b]``, etc.),
168 by a deployment tool, or using the ``ceph`` commandline.
170 - **Keys**: The monitor must have secret keys. A deployment tool such as
171 ``cephadm`` usually does this for you, but you may
172 perform this step manually too. See `Monitor Keyrings`_ for details.
174 For additional details on bootstrapping, see `Bootstrapping a Monitor`_.
176 .. index:: Ceph Monitor; configuring monitors
181 To apply configuration settings to the entire cluster, enter the configuration
182 settings under ``[global]``. To apply configuration settings to all monitors in
183 your cluster, enter the configuration settings under ``[mon]``. To apply
184 configuration settings to specific monitors, specify the monitor instance
185 (e.g., ``[mon.a]``). By convention, monitor instance names use alpha notation.
200 Minimum Configuration
201 ---------------------
203 The bare minimum monitor settings for a Ceph monitor via the Ceph configuration
204 file include a hostname and a network address for each monitor. You can configure
205 these under ``[mon]`` or under the entry for a specific monitor.
210 mon host = 10.0.0.2,10.0.0.3,10.0.0.4
216 mon addr = 10.0.0.10:6789
218 See the `Network Configuration Reference`_ for details.
220 .. note:: This minimum configuration for monitors assumes that a deployment
221 tool generates the ``fsid`` and the ``mon.`` key for you.
223 Once you deploy a Ceph cluster, you **SHOULD NOT** change the IP addresses of
224 monitors. However, if you decide to change the monitor's IP address, you
225 must follow a specific procedure. See :ref:`Changing a Monitor's IP address` for
228 Monitors can also be found by clients by using DNS SRV records. See `Monitor lookup through DNS`_ for details.
233 Each Ceph Storage Cluster has a unique identifier (``fsid``). If specified, it
234 usually appears under the ``[global]`` section of the configuration file.
235 Deployment tools usually generate the ``fsid`` and store it in the monitor map,
236 so the value may not appear in a configuration file. The ``fsid`` makes it
237 possible to run daemons for multiple clusters on the same hardware.
241 .. index:: Ceph Monitor; initial members
246 We recommend running a production Ceph Storage Cluster with at least three Ceph
247 Monitors to ensure high availability. When you run multiple monitors, you may
248 specify the initial monitors that must be members of the cluster in order to
249 establish a quorum. This may reduce the time it takes for your cluster to come
255 mon_initial_members = a,b,c
258 .. confval:: mon_initial_members
260 .. index:: Ceph Monitor; data path
265 Ceph provides a default path where Ceph Monitors store data. For optimal
266 performance in a production Ceph Storage Cluster, we recommend running Ceph
267 Monitors on separate hosts and drives from Ceph OSD Daemons. As leveldb uses
268 ``mmap()`` for writing the data, Ceph Monitors flush their data from memory to disk
269 very often, which can interfere with Ceph OSD Daemon workloads if the data
270 store is co-located with the OSD Daemons.
272 In Ceph versions 0.58 and earlier, Ceph Monitors store their data in plain files. This
273 approach allows users to inspect monitor data with common tools like ``ls``
274 and ``cat``. However, this approach didn't provide strong consistency.
276 In Ceph versions 0.59 and later, Ceph Monitors store their data as key/value
277 pairs. Ceph Monitors require `ACID`_ transactions. Using a data store prevents
278 recovering Ceph Monitors from running corrupted versions through Paxos, and it
279 enables multiple modification operations in one single atomic batch, among other
282 Generally, we do not recommend changing the default data location. If you modify
283 the default location, we recommend that you make it uniform across Ceph Monitors
284 by setting it in the ``[mon]`` section of the configuration file.
287 .. confval:: mon_data
288 .. confval:: mon_data_size_warn
289 .. confval:: mon_data_avail_warn
290 .. confval:: mon_data_avail_crit
291 .. confval:: mon_warn_on_cache_pools_without_hit_sets
292 .. confval:: mon_warn_on_crush_straw_calc_version_zero
293 .. confval:: mon_warn_on_legacy_crush_tunables
294 .. confval:: mon_crush_min_required_version
295 .. confval:: mon_warn_on_osd_down_out_interval_zero
296 .. confval:: mon_warn_on_slow_ping_ratio
297 .. confval:: mon_warn_on_slow_ping_time
298 .. confval:: mon_warn_on_pool_no_redundancy
299 .. confval:: mon_cache_target_full_warn_ratio
300 .. confval:: mon_health_to_clog
301 .. confval:: mon_health_to_clog_tick_interval
302 .. confval:: mon_health_to_clog_interval
304 .. index:: Ceph Storage Cluster; capacity planning, Ceph Monitor; capacity planning
306 .. _storage-capacity:
311 When a Ceph Storage Cluster gets close to its maximum capacity
312 (see``mon_osd_full ratio``), Ceph prevents you from writing to or reading from OSDs
313 as a safety measure to prevent data loss. Therefore, letting a
314 production Ceph Storage Cluster approach its full ratio is not a good practice,
315 because it sacrifices high availability. The default full ratio is ``.95``, or
316 95% of capacity. This a very aggressive setting for a test cluster with a small
319 .. tip:: When monitoring your cluster, be alert to warnings related to the
320 ``nearfull`` ratio. This means that a failure of some OSDs could result
321 in a temporary service disruption if one or more OSDs fails. Consider adding
322 more OSDs to increase storage capacity.
324 A common scenario for test clusters involves a system administrator removing an
325 OSD from the Ceph Storage Cluster, watching the cluster rebalance, then removing
326 another OSD, and another, until at least one OSD eventually reaches the full
327 ratio and the cluster locks up. We recommend a bit of capacity
328 planning even with a test cluster. Planning enables you to gauge how much spare
329 capacity you will need in order to maintain high availability. Ideally, you want
330 to plan for a series of Ceph OSD Daemon failures where the cluster can recover
331 to an ``active+clean`` state without replacing those OSDs
332 immediately. Cluster operation continues in the ``active+degraded`` state, but this
333 is not ideal for normal operation and should be addressed promptly.
335 The following diagram depicts a simplistic Ceph Storage Cluster containing 33
336 Ceph Nodes with one OSD per host, each OSD reading from
337 and writing to a 3TB drive. So this exemplary Ceph Storage Cluster has a maximum
338 actual capacity of 99TB. With a ``mon osd full ratio`` of ``0.95``, if the Ceph
339 Storage Cluster falls to 5TB of remaining capacity, the cluster will not allow
340 Ceph Clients to read and write data. So the Ceph Storage Cluster's operating
341 capacity is 95TB, not 99TB.
344 +--------+ +--------+ +--------+ +--------+ +--------+ +--------+
345 | Rack 1 | | Rack 2 | | Rack 3 | | Rack 4 | | Rack 5 | | Rack 6 |
346 | cCCC | | cF00 | | cCCC | | cCCC | | cCCC | | cCCC |
347 +--------+ +--------+ +--------+ +--------+ +--------+ +--------+
348 | OSD 1 | | OSD 7 | | OSD 13 | | OSD 19 | | OSD 25 | | OSD 31 |
349 +--------+ +--------+ +--------+ +--------+ +--------+ +--------+
350 | OSD 2 | | OSD 8 | | OSD 14 | | OSD 20 | | OSD 26 | | OSD 32 |
351 +--------+ +--------+ +--------+ +--------+ +--------+ +--------+
352 | OSD 3 | | OSD 9 | | OSD 15 | | OSD 21 | | OSD 27 | | OSD 33 |
353 +--------+ +--------+ +--------+ +--------+ +--------+ +--------+
354 | OSD 4 | | OSD 10 | | OSD 16 | | OSD 22 | | OSD 28 | | Spare |
355 +--------+ +--------+ +--------+ +--------+ +--------+ +--------+
356 | OSD 5 | | OSD 11 | | OSD 17 | | OSD 23 | | OSD 29 | | Spare |
357 +--------+ +--------+ +--------+ +--------+ +--------+ +--------+
358 | OSD 6 | | OSD 12 | | OSD 18 | | OSD 24 | | OSD 30 | | Spare |
359 +--------+ +--------+ +--------+ +--------+ +--------+ +--------+
361 It is normal in such a cluster for one or two OSDs to fail. A less frequent but
362 reasonable scenario involves a rack's router or power supply failing, which
363 brings down multiple OSDs simultaneously (e.g., OSDs 7-12). In such a scenario,
364 you should still strive for a cluster that can remain operational and achieve an
365 ``active + clean`` state--even if that means adding a few hosts with additional
366 OSDs in short order. If your capacity utilization is too high, you may not lose
367 data, but you could still sacrifice data availability while resolving an outage
368 within a failure domain if capacity utilization of the cluster exceeds the full
369 ratio. For this reason, we recommend at least some rough capacity planning.
371 Identify two numbers for your cluster:
373 #. The number of OSDs.
374 #. The total capacity of the cluster
376 If you divide the total capacity of your cluster by the number of OSDs in your
377 cluster, you will find the mean average capacity of an OSD within your cluster.
378 Consider multiplying that number by the number of OSDs you expect will fail
379 simultaneously during normal operations (a relatively small number). Finally
380 multiply the capacity of the cluster by the full ratio to arrive at a maximum
381 operating capacity; then, subtract the number of amount of data from the OSDs
382 you expect to fail to arrive at a reasonable full ratio. Repeat the foregoing
383 process with a higher number of OSD failures (e.g., a rack of OSDs) to arrive at
384 a reasonable number for a near full ratio.
386 The following settings only apply on cluster creation and are then stored in
387 the OSDMap. To clarify, in normal operation the values that are used by OSDs
388 are those found in the OSDMap, not those in the configuration file or central
394 mon_osd_full_ratio = .80
395 mon_osd_backfillfull_ratio = .75
396 mon_osd_nearfull_ratio = .70
399 ``mon_osd_full_ratio``
401 :Description: The threshold percentage of device space utilized before an OSD is
408 ``mon_osd_backfillfull_ratio``
410 :Description: The threshold percentage of device space utilized before an OSD is
411 considered too ``full`` to backfill.
417 ``mon_osd_nearfull_ratio``
419 :Description: The threshold percentage of device space used before an OSD is
420 considered ``nearfull``.
426 .. tip:: If some OSDs are nearfull, but others have plenty of capacity, you
427 may have an inaccurate CRUSH weight set for the nearfull OSDs.
429 .. tip:: These settings only apply during cluster creation. Afterwards they need
430 to be changed in the OSDMap using ``ceph osd set-nearfull-ratio`` and
431 ``ceph osd set-full-ratio``
438 Ceph monitors know about the cluster by requiring reports from each OSD, and by
439 receiving reports from OSDs about the status of their neighboring OSDs. Ceph
440 provides reasonable default settings for monitor/OSD interaction; however, you
441 may modify them as needed. See `Monitor/OSD Interaction`_ for details.
444 .. index:: Ceph Monitor; leader, Ceph Monitor; provider, Ceph Monitor; requester, Ceph Monitor; synchronization
446 Monitor Store Synchronization
447 -----------------------------
449 When you run a production cluster with multiple monitors (recommended), each
450 monitor checks to see if a neighboring monitor has a more recent version of the
451 cluster map (e.g., a map in a neighboring monitor with one or more epoch numbers
452 higher than the most current epoch in the map of the instant monitor).
453 Periodically, one monitor in the cluster may fall behind the other monitors to
454 the point where it must leave the quorum, synchronize to retrieve the most
455 current information about the cluster, and then rejoin the quorum. For the
456 purposes of synchronization, monitors may assume one of three roles:
458 #. **Leader**: The `Leader` is the first monitor to achieve the most recent
459 Paxos version of the cluster map.
461 #. **Provider**: The `Provider` is a monitor that has the most recent version
462 of the cluster map, but wasn't the first to achieve the most recent version.
464 #. **Requester:** A `Requester` is a monitor that has fallen behind the leader
465 and must synchronize in order to retrieve the most recent information about
466 the cluster before it can rejoin the quorum.
468 These roles enable a leader to delegate synchronization duties to a provider,
469 which prevents synchronization requests from overloading the leader--improving
470 performance. In the following diagram, the requester has learned that it has
471 fallen behind the other monitors. The requester asks the leader to synchronize,
472 and the leader tells the requester to synchronize with a provider.
476 +-----------+ +---------+ +----------+
477 | Requester | | Leader | | Provider |
478 +-----------+ +---------+ +----------+
481 | Ask to Synchronize | |
482 |------------------->| |
484 |<-------------------| |
485 | Tell Requester to | |
486 | Sync with Provider | |
489 |--------------------+-------------------->|
491 |<-------------------+---------------------|
492 | Send Chunk to Requester |
493 | (repeat as necessary) |
494 | Requester Acks Chuck to Provider |
495 |--------------------+-------------------->|
499 |------------------->|
501 |<-------------------|
506 Synchronization always occurs when a new monitor joins the cluster. During
507 runtime operations, monitors may receive updates to the cluster map at different
508 times. This means the leader and provider roles may migrate from one monitor to
509 another. If this happens while synchronizing (e.g., a provider falls behind the
510 leader), the provider can terminate synchronization with a requester.
512 Once synchronization is complete, Ceph performs trimming across the cluster.
513 Trimming requires that the placement groups are ``active+clean``.
516 .. confval:: mon_sync_timeout
517 .. confval:: mon_sync_max_payload_size
518 .. confval:: paxos_max_join_drift
519 .. confval:: paxos_stash_full_interval
520 .. confval:: paxos_propose_interval
521 .. confval:: paxos_min
522 .. confval:: paxos_min_wait
523 .. confval:: paxos_trim_min
524 .. confval:: paxos_trim_max
525 .. confval:: paxos_service_trim_min
526 .. confval:: paxos_service_trim_max
527 .. confval:: paxos_service_trim_max_multiplier
528 .. confval:: mon_mds_force_trim_to
529 .. confval:: mon_osd_force_trim_to
530 .. confval:: mon_osd_cache_size
531 .. confval:: mon_election_timeout
532 .. confval:: mon_lease
533 .. confval:: mon_lease_renew_interval_factor
534 .. confval:: mon_lease_ack_timeout_factor
535 .. confval:: mon_accept_timeout_factor
536 .. confval:: mon_min_osdmap_epochs
537 .. confval:: mon_max_log_epochs
540 .. index:: Ceph Monitor; clock
545 Ceph daemons pass critical messages to each other, which must be processed
546 before daemons reach a timeout threshold. If the clocks in Ceph monitors
547 are not synchronized, it can lead to a number of anomalies. For example:
549 - Daemons ignoring received messages (e.g., timestamps outdated)
550 - Timeouts triggered too soon/late when a message wasn't received in time.
552 See `Monitor Store Synchronization`_ for details.
555 .. tip:: You must configure NTP or PTP daemons on your Ceph monitor hosts to
556 ensure that the monitor cluster operates with synchronized clocks.
557 It can be advantageous to have monitor hosts sync with each other
558 as well as with multiple quality upstream time sources.
560 Clock drift may still be noticeable with NTP even though the discrepancy is not
561 yet harmful. Ceph's clock drift / clock skew warnings may get triggered even
562 though NTP maintains a reasonable level of synchronization. Increasing your
563 clock drift may be tolerable under such circumstances; however, a number of
564 factors such as workload, network latency, configuring overrides to default
565 timeouts and the `Monitor Store Synchronization`_ settings may influence
566 the level of acceptable clock drift without compromising Paxos guarantees.
568 Ceph provides the following tunable options to allow you to find
571 .. confval:: mon_tick_interval
572 .. confval:: mon_clock_drift_allowed
573 .. confval:: mon_clock_drift_warn_backoff
574 .. confval:: mon_timecheck_interval
575 .. confval:: mon_timecheck_skew_interval
580 .. confval:: mon_client_hunt_interval
581 .. confval:: mon_client_ping_interval
582 .. confval:: mon_client_max_log_entries_per_message
583 .. confval:: mon_client_bytes
590 Since version v0.94 there is support for pool flags which allow or disallow changes to be made to pools.
591 Monitors can also disallow removal of pools if appropriately configured. The inconvenience of this guardrail
592 is far outweighed by the number of accidental pool (and thus data) deletions it prevents.
594 .. confval:: mon_allow_pool_delete
595 .. confval:: osd_pool_default_ec_fast_read
596 .. confval:: osd_pool_default_flag_hashpspool
597 .. confval:: osd_pool_default_flag_nodelete
598 .. confval:: osd_pool_default_flag_nopgchange
599 .. confval:: osd_pool_default_flag_nosizechange
601 For more information about the pool flags see :ref:`Pool values <setpoolvalues>`.
606 .. confval:: mon_max_osd
607 .. confval:: mon_globalid_prealloc
608 .. confval:: mon_subscribe_interval
609 .. confval:: mon_stat_smooth_intervals
610 .. confval:: mon_probe_timeout
611 .. confval:: mon_daemon_bytes
612 .. confval:: mon_max_log_entries_per_event
613 .. confval:: mon_osd_prime_pg_temp
614 .. confval:: mon_osd_prime_pg_temp_max_time
615 .. confval:: mon_osd_prime_pg_temp_max_estimate
616 .. confval:: mon_mds_skip_sanity
617 .. confval:: mon_max_mdsmap_epochs
618 .. confval:: mon_config_key_max_entry_size
619 .. confval:: mon_scrub_interval
620 .. confval:: mon_scrub_max_keys
621 .. confval:: mon_compact_on_start
622 .. confval:: mon_compact_on_bootstrap
623 .. confval:: mon_compact_on_trim
624 .. confval:: mon_cpu_threads
625 .. confval:: mon_osd_mapping_pgs_per_chunk
626 .. confval:: mon_session_timeout
627 .. confval:: mon_osd_cache_size_min
628 .. confval:: mon_memory_target
629 .. confval:: mon_memory_autotune
631 .. _Paxos: https://en.wikipedia.org/wiki/Paxos_(computer_science)
632 .. _Monitor Keyrings: ../../../dev/mon-bootstrap#secret-keys
633 .. _Ceph configuration file: ../ceph-conf/#monitors
634 .. _Network Configuration Reference: ../network-config-ref
635 .. _Monitor lookup through DNS: ../mon-lookup-dns
636 .. _ACID: https://en.wikipedia.org/wiki/ACID
637 .. _Adding/Removing a Monitor: ../../operations/add-or-rm-mons
638 .. _Monitoring a Cluster: ../../operations/monitoring
639 .. _Monitoring OSDs and PGs: ../../operations/monitoring-osd-pg
640 .. _Bootstrapping a Monitor: ../../../dev/mon-bootstrap
641 .. _Monitor/OSD Interaction: ../mon-osd-interaction
642 .. _Scalability and High Availability: ../../../architecture#scalability-and-high-availability