7 When Ceph services start, the initialization process activates a series
8 of daemons that run in the background. A :term:`Ceph Storage Cluster` runs
9 at a minimum three types of daemons:
11 - :term:`Ceph Monitor` (``ceph-mon``)
12 - :term:`Ceph Manager` (``ceph-mgr``)
13 - :term:`Ceph OSD Daemon` (``ceph-osd``)
15 Ceph Storage Clusters that support the :term:`Ceph File System` also run at
16 least one :term:`Ceph Metadata Server` (``ceph-mds``). Clusters that
17 support :term:`Ceph Object Storage` run Ceph RADOS Gateway daemons
18 (``radosgw``) as well.
20 Each daemon has a number of configuration options, each of which has a
21 default value. You may adjust the behavior of the system by changing these
22 configuration options. Be careful to understand the consequences before
23 overriding default values, as it is possible to significantly degrade the
24 performance and stability of your cluster. Also note that default values
25 sometimes change between releases, so it is best to review the version of
26 this documentation that aligns with your Ceph release.
31 All Ceph configuration options have a unique name consisting of words
32 formed with lower-case characters and connected with underscore
35 When option names are specified on the command line, either underscore
36 (``_``) or dash (``-``) characters can be used interchangeable (e.g.,
37 ``--mon-host`` is equivalent to ``--mon_host``).
39 When option names appear in configuration files, spaces can also be
40 used in place of underscore or dash. We suggest, though, that for
41 clarity and convenience you consistently use underscores, as we do
42 throughout this documentation.
47 Each Ceph daemon, process, and library will pull its configuration
48 from several sources, listed below. Sources later in the list will
49 override those earlier in the list when both are present.
51 - the compiled-in default value
52 - the monitor cluster's centralized configuration database
53 - a configuration file stored on the local host
54 - environment variables
55 - command line arguments
56 - runtime overrides set by an administrator
58 One of the first things a Ceph process does on startup is parse the
59 configuration options provided via the command line, environment, and
60 local configuration file. The process will then contact the monitor
61 cluster to retrieve configuration stored centrally for the entire
62 cluster. Once a complete view of the configuration is available, the
63 daemon or process startup will proceed.
65 .. _bootstrap-options:
70 Some configuration options affect the process's ability to contact the
71 monitors, to authenticate, and to retrieve the cluster-stored configuration.
72 For this reason, these options might need to be stored locally on the node, and
73 set by means of a local configuration file. These options include the
77 .. confval:: mon_host_override
79 - :confval:`mon_dns_srv_name`
80 - :confval:`mon_data`, :confval:`osd_data`, :confval:`mds_data`, :confval:`mgr_data`, and
81 similar options that define which local directory the daemon
83 - :confval:`keyring`, :confval:`keyfile`, and/or :confval:`key`, which can be used to
84 specify the authentication credential to use to authenticate with
85 the monitor. Note that in most cases the default keyring location
86 is in the data directory specified above.
88 In most cases, the default values of these options are suitable. There is one
89 exception to this: the :confval:`mon_host` option that identifies the addresses
90 of the cluster's monitors. When DNS is used to identify monitors, a local Ceph
91 configuration file can be avoided entirely.
93 Skipping monitor config
94 -----------------------
96 Pass the option ``--no-mon-config`` to any process to skip the step that
97 retrieves configuration information from the cluster monitors. This is useful
98 in cases where configuration is managed entirely via configuration files, or
99 when the monitor cluster is down and some maintenance activity needs to be
106 Configuration sections
107 ======================
109 Any given process or daemon has a single value for each configuration
110 option. However, values for an option may vary across different
111 daemon types even daemons of the same type. Ceph options that are
112 stored in the monitor configuration database or in local configuration
113 files are grouped into sections to indicate which daemons or clients
116 These sections include:
120 Settings under ``global`` affect all daemons and clients
121 in a Ceph Storage Cluster.
123 :example: ``log_file = /var/log/ceph/$cluster-$type.$id.log``
127 Settings under ``mon`` affect all ``ceph-mon`` daemons in
128 the Ceph Storage Cluster, and override the same setting in
131 :example: ``mon_cluster_log_to_syslog = true``
135 Settings in the ``mgr`` section affect all ``ceph-mgr`` daemons in
136 the Ceph Storage Cluster, and override the same setting in
139 :example: ``mgr_stats_period = 10``
143 Settings under ``osd`` affect all ``ceph-osd`` daemons in
144 the Ceph Storage Cluster, and override the same setting in
147 :example: ``osd_op_queue = wpq``
151 Settings in the ``mds`` section affect all ``ceph-mds`` daemons in
152 the Ceph Storage Cluster, and override the same setting in
155 :example: ``mds_cache_memory_limit = 10G``
159 Settings under ``client`` affect all Ceph Clients
160 (e.g., mounted Ceph File Systems, mounted Ceph Block Devices,
161 etc.) as well as Rados Gateway (RGW) daemons.
163 :example: ``objecter_inflight_ops = 512``
166 Sections may also specify an individual daemon or client name. For example,
167 ``mon.foo``, ``osd.123``, and ``client.smith`` are all valid section names.
170 Any given daemon will draw its settings from the global section, the
171 daemon or client type section, and the section sharing its name.
172 Settings in the most-specific section take precedence, so for example
173 if the same option is specified in both :confsec:`global`, :confsec:`mon`, and
174 ``mon.foo`` on the same source (i.e., in the same configurationfile),
175 the ``mon.foo`` value will be used.
177 If multiple values of the same configuration option are specified in the same
178 section, the last value wins.
180 Note that values from the local configuration file always take
181 precedence over values from the monitor configuration database,
182 regardless of which section they appear in.
185 .. _ceph-metavariables:
190 Metavariables simplify Ceph Storage Cluster configuration
191 dramatically. When a metavariable is set in a configuration value,
192 Ceph expands the metavariable into a concrete value at the time the
193 configuration value is used. Ceph metavariables are similar to variable expansion in the Bash shell.
195 Ceph supports the following metavariables:
197 .. describe:: $cluster
199 Expands to the Ceph Storage Cluster name. Useful when running
200 multiple Ceph Storage Clusters on the same hardware.
202 :example: ``/etc/ceph/$cluster.keyring``
207 Expands to a daemon or process type (e.g., ``mds``, ``osd``, or ``mon``)
209 :example: ``/var/lib/ceph/$type``
213 Expands to the daemon or client identifier. For
214 ``osd.0``, this would be ``0``; for ``mds.a``, it would
217 :example: ``/var/lib/ceph/$type/$cluster-$id``
221 Expands to the host name where the process is running.
225 Expands to ``$type.$id``.
227 :example: ``/var/run/ceph/$cluster-$name.asok``
231 Expands to daemon pid.
233 :example: ``/var/run/ceph/$cluster-$name-$pid.asok``
237 The Configuration File
238 ======================
240 On startup, Ceph processes search for a configuration file in the
243 #. ``$CEPH_CONF`` (*i.e.,* the path following the ``$CEPH_CONF``
244 environment variable)
245 #. ``-c path/path`` (*i.e.,* the ``-c`` command line argument)
246 #. ``/etc/ceph/$cluster.conf``
247 #. ``~/.ceph/$cluster.conf``
248 #. ``./$cluster.conf`` (*i.e.,* in the current working directory)
249 #. On FreeBSD systems only, ``/usr/local/etc/ceph/$cluster.conf``
251 where ``$cluster`` is the cluster's name (default ``ceph``).
253 The Ceph configuration file uses an *ini* style syntax. You can add comment
254 text after a pound sign (#) or a semi-colon (;). For example:
258 # <--A number (#) sign precedes a comment.
259 ; A comment may be anything.
260 # Comments always follow a semi-colon (;) or a pound (#) on each line.
261 # The end of the line terminates a comment.
262 # We recommend that you provide comments in your configuration file(s).
265 .. _ceph-conf-settings:
267 Config file section names
268 -------------------------
270 The configuration file is divided into sections. Each section must begin with a
271 valid configuration section name (see `Configuration sections`_, above)
272 surrounded by square brackets. For example,
289 Config file option values
290 -------------------------
292 The value of a configuration option is a string. If it is too long to
293 fit in a single line, you can put a backslash (``\``) at the end of line
294 as the line continuation marker, so the value of the option will be
295 the string after ``=`` in current line combined with the string in the next
302 In the example above, the value of "``foo``" would be "``long long ago long ago``".
304 Normally, the option value ends with a new line, or a comment, like
309 obscure_one = difficult to explain # I will try harder in next release
310 simpler_one = nothing to explain
312 In the example above, the value of "``obscure one``" would be "``difficult to explain``";
313 and the value of "``simpler one`` would be "``nothing to explain``".
315 If an option value contains spaces, and we want to make it explicit, we
316 could quote the value using single or double quotes, like
321 line = "to be, or not to be"
323 Certain characters are not allowed to be present in the option values directly.
324 They are ``=``, ``#``, ``;`` and ``[``. If we have to, we need to escape them,
330 secret = "i love \# and \["
332 Every configuration option is typed with one of the types below:
336 64-bit signed integer, Some SI prefixes are supported, like "K", "M", "G",
337 "T", "P", "E", meaning, respectively, 10\ :sup:`3`, 10\ :sup:`6`,
338 10\ :sup:`9`, etc. And "B" is the only supported unit. So, "1K", "1M", "128B" and "-1" are all valid
339 option values. Some times, a negative value implies "unlimited" when it comes to
340 an option for threshold or limit.
342 :example: ``42``, ``-1``
346 It is almost identical to ``integer``. But a negative value will be rejected.
348 :example: ``256``, ``0``
352 Free style strings encoded in UTF-8, but some characters are not allowed. Please
353 reference the above notes for the details.
355 :example: ``"hello world"``, ``"i love \#"``, ``yet-another-name``
357 .. describe:: boolean
359 one of the two values ``true`` or ``false``. But an integer is also accepted,
360 where "0" implies ``false``, and any non-zero values imply ``true``.
362 :example: ``true``, ``false``, ``1``, ``0``
366 a single address optionally prefixed with ``v1``, ``v2`` or ``any`` for the messenger
367 protocol. If the prefix is not specified, ``v2`` protocol is used. Please see
368 :ref:`address_formats` for more details.
370 :example: ``v1:1.2.3.4:567``, ``v2:1.2.3.4:567``, ``1.2.3.4:567``, ``2409:8a1e:8fb6:aa20:1260:4bff:fe92:18f5::567``, ``[::1]:6789``
372 .. describe:: addrvec
374 a set of addresses separated by ",". The addresses can be optionally quoted with ``[`` and ``]``.
376 :example: ``[v1:1.2.3.4:567,v2:1.2.3.4:568]``, ``v1:1.2.3.4:567,v1:1.2.3.14:567`` ``[2409:8a1e:8fb6:aa20:1260:4bff:fe92:18f5::567], [2409:8a1e:8fb6:aa20:1260:4bff:fe92:18f5::568]``
380 the string format of a uuid defined by `RFC4122 <https://www.ietf.org/rfc/rfc4122.txt>`_.
381 And some variants are also supported, for more details, see
382 `Boost document <https://www.boost.org/doc/libs/1_74_0/libs/uuid/doc/uuid.html#String%20Generator>`_.
384 :example: ``f81d4fae-7dec-11d0-a765-00a0c91e6bf6``
388 denotes a 64-bit unsigned integer. Both SI prefixes and IEC prefixes are
389 supported. And "B" is the only supported unit. A negative value will be
392 :example: ``1Ki``, ``1K``, ``1KiB`` and ``1B``.
396 denotes a duration of time. By default the unit is second if not specified.
397 Following units of time are supported:
399 * second: "s", "sec", "second", "seconds"
400 * minute: "m", "min", "minute", "minutes"
401 * hour: "hs", "hr", "hour", "hours"
402 * day: "d", "day", "days"
403 * week: "w", "wk", "week", "weeks"
404 * month: "mo", "month", "months"
405 * year: "y", "yr", "year", "years"
407 :example: ``1 m``, ``1m`` and ``1 week``
409 .. _ceph-conf-database:
411 Monitor configuration database
412 ==============================
414 The monitor cluster manages a database of configuration options that
415 can be consumed by the entire cluster, enabling streamlined central
416 configuration management for the entire system. The vast majority of
417 configuration options can and should be stored here for ease of
418 administration and transparency.
420 A handful of settings may still need to be stored in local
421 configuration files because they affect the ability to connect to the
422 monitors, authenticate, and fetch configuration information. In most
423 cases this is limited to the ``mon_host`` option, although this can
424 also be avoided through the use of DNS SRV records.
429 Configuration options stored by the monitor can live in a global
430 section, daemon type section, or specific daemon section, just like
431 options in a configuration file can.
433 In addition, options may also have a *mask* associated with them to
434 further restrict which daemons or clients the option applies to.
435 Masks take two forms:
437 #. ``type:location`` where *type* is a CRUSH property like `rack` or
438 `host`, and *location* is a value for that property. For example,
439 ``host:foo`` would limit the option only to daemons or clients
440 running on a particular host.
441 #. ``class:device-class`` where *device-class* is the name of a CRUSH
442 device class (e.g., ``hdd`` or ``ssd``). For example,
443 ``class:ssd`` would limit the option only to OSDs backed by SSDs.
444 (This mask has no effect for non-OSD daemons or clients.)
446 When setting a configuration option, the `who` may be a section name,
447 a mask, or a combination of both separated by a slash (``/``)
448 character. For example, ``osd/rack:foo`` would mean all OSD daemons
451 When viewing configuration options, the section name and mask are
452 generally separated out into separate fields or columns to ease readability.
458 The following CLI commands are used to configure the cluster:
460 * ``ceph config dump`` will dump the entire monitors' configuration
461 database for the cluster.
463 * ``ceph config get <who>`` will dump configuration options stored in
464 the monitors' configuration database for a specific daemon or client
467 * ``ceph config get <who> <option>`` will show a configuration value
468 stored in the monitors' configuration database for a specific daemon
469 or client (e.g., ``mds.a``), or, if not present in the monitors'
470 configuration database, the compiled-in default value.
472 * ``ceph config set <who> <option> <value>`` will set a configuration
473 option in the monitors' configuration database.
475 * ``ceph config show <who>`` will show the reported running
476 configuration for a running daemon. These settings may differ from
477 those stored by the monitors if there are also local configuration
478 files in use or options have been overridden on the command line or
479 at run time. The source of the option values is reported as part
482 * ``ceph config assimilate-conf -i <input file> -o <output file>``
483 will ingest a configuration file from *input file* and move any
484 valid options into the monitors' configuration database. Any
485 settings that are unrecognized, invalid, or cannot be controlled by
486 the monitor will be returned in an abbreviated config file stored in
487 *output file*. This command is useful for transitioning from legacy
488 configuration files to centralized monitor-based configuration.
490 Note that ``ceph config set <who> <option> <value>`` and ``ceph config get
491 <who> <option>`` aren't symmetric because the latter also shows compiled-in
492 default values. In order to determine whether a configuration option is
493 present in the monitors' configuration database, use ``ceph config dump``.
499 You can get help for a particular option with:
503 ceph config help <option>
505 Note that this will use the configuration schema that is compiled into the running monitors. If you have a mixed-version cluster (e.g., during an upgrade), you might also want to query the option schema from a specific running daemon:
509 ceph daemon <name> config help [option]
515 ceph config help log_file
519 log_file - path to log file
521 Default (non-daemon):
522 Default (daemon): /var/log/ceph/$cluster-$name.log
523 Can update at runtime: false
524 See also: [log_to_stderr,err_to_stderr,log_to_syslog,err_to_syslog]
530 ceph config help log_file -f json-pretty
536 "type": "std::string",
538 "desc": "path to log file",
541 "daemon_default": "/var/log/ceph/$cluster-$name.log",
553 "can_update_at_runtime": false
556 The ``level`` property can be any of `basic`, `advanced`, or `dev`.
557 The `dev` options are intended for use by developers, generally for
558 testing purposes, and are not recommended for use by operators.
564 In most cases, Ceph permits changes to the configuration of a daemon at
565 runtime. This can be used for increasing or decreasing the amount of logging
566 output, for enabling or disabling debug settings, and for runtime optimization.
568 Configuration options can be updated via the ``ceph config set`` command. For
569 example, to enable the debug log level on a specific OSD, run a command of this form:
573 ceph config set osd.123 debug_ms 20
575 .. note:: If an option has been customized in a local configuration file, the
577 <https://ceph.io/en/news/blog/2018/new-mimic-centralized-configuration-management/>`_
578 setting will be ignored (it has a lower priority than the local
584 Options can be set temporarily by using the `tell` or `daemon` interfaces on
585 the Ceph CLI. These *override* values are ephemeral, which means that they
586 affect only the current instance of the daemon and revert to persistently
587 configured values when the daemon restarts.
589 Override values can be set in two ways:
591 #. From any host, send a message to a daemon with a command of the following
596 ceph tell <name> config set <option> <value>
602 ceph tell osd.123 config set debug_osd 20
604 The ``tell`` command can also accept a wildcard as the daemon identifier.
605 For example, to adjust the debug level on all OSD daemons, run a command of
610 ceph tell osd.* config set debug_osd 20
612 #. On the host where the daemon is running, connect to the daemon via a socket
613 in ``/var/run/ceph`` by running a command of this form:
617 ceph daemon <name> config set <option> <value>
623 ceph daemon osd.4 config set debug_osd 20
625 .. note:: In the output of the ``ceph config show`` command, these temporary
626 values are shown with a source of ``override``.
629 Viewing runtime settings
630 ========================
632 You can see the current options set for a running daemon with the ``ceph config show`` command. For example:
636 ceph config show osd.0
638 will show you the (non-default) options for that daemon. You can also look at a specific option with:
642 ceph config show osd.0 debug_osd
644 or view all options (even those with default values) with:
648 ceph config show-with-defaults osd.0
650 You can also observe settings for a running daemon by connecting to it from the local host via the admin socket. For example:
654 ceph daemon osd.0 config show
656 will dump all current settings:
660 ceph daemon osd.0 config diff
662 will show only non-default settings (as well as where the value came from: a config file, the monitor, an override, etc.), and:
666 ceph daemon osd.0 config get debug_osd
668 will report the value of a single option.
672 Changes since Nautilus
673 ======================
675 The Octopus release changed the way the configuration file is parsed.
676 These changes are as follows:
678 - Repeated configuration options are allowed, and no warnings will be printed.
679 The value of the last one is used, which means that the setting last in the file
680 is the one that takes effect. Before this change, we would print warning messages
681 when lines with duplicated options were encountered, like::
683 warning line 42: 'foo' in section 'bar' redefined
685 - Invalid UTF-8 options were ignored with warning messages. But since Octopus,
686 they are treated as fatal errors.
688 - Backslash ``\`` is used as the line continuation marker to combine the next
689 line with current one. Before Octopus, it was required to follow a backslash with
690 a non-empty line. But in Octopus, an empty line following a backslash is now allowed.
692 - In the configuration file, each line specifies an individual configuration
693 option. The option's name and its value are separated with ``=``, and the
694 value may be quoted using single or double quotes. If an invalid
695 configuration is specified, we will treat it as an invalid configuration
698 bad option ==== bad value
700 - Before Octopus, if no section name was specified in the configuration file,
701 all options would be set as though they were within the :confsec:`global` section. This is
702 now discouraged. Since Octopus, only a single option is allowed for
703 configuration files without a section name.