7 When Ceph services start, the initialization process activates a set of
8 daemons that run in the background. A :term:`Ceph Storage Cluster` runs at
9 least three types of daemons:
11 - :term:`Ceph Monitor` (``ceph-mon``)
12 - :term:`Ceph Manager` (``ceph-mgr``)
13 - :term:`Ceph OSD Daemon` (``ceph-osd``)
15 Any Ceph Storage Cluster that supports the :term:`Ceph File System` also runs
16 at least one :term:`Ceph Metadata Server` (``ceph-mds``). Any Cluster that
17 supports :term:`Ceph Object Storage` runs Ceph RADOS Gateway daemons
20 Each daemon has a number of configuration options, and each of those options
21 has a default value. Adjust the behavior of the system by changing these
22 configuration options. Make sure to understand the consequences before
23 overriding the default values, as it is possible to significantly degrade the
24 performance and stability of your cluster. Remember that default values
25 sometimes change between releases. For this reason, it is best to review the
26 version of this documentation that applies to your Ceph release.
31 Each of the Ceph configuration options has a unique name that consists of words
32 formed with lowercase characters and connected with underscore characters
35 When option names are specified on the command line, underscore (``_``) and
36 dash (``-``) characters can be used interchangeably (for example,
37 ``--mon-host`` is equivalent to ``--mon_host``).
39 When option names appear in configuration files, spaces can also be used in
40 place of underscores or dashes. However, for the sake of clarity and
41 convenience, we suggest that you consistently use underscores, as we do
42 throughout this documentation.
47 Each Ceph daemon, process, and library pulls its configuration from one or more
48 of the several sources listed below. Sources that occur later in the list
49 override those that occur earlier in the list (when both are present).
51 - the compiled-in default value
52 - the monitor cluster's centralized configuration database
53 - a configuration file stored on the local host
54 - environment variables
55 - command-line arguments
56 - runtime overrides that are set by an administrator
58 One of the first things a Ceph process does on startup is parse the
59 configuration options provided via the command line, via the environment, and
60 via the local configuration file. Next, the process contacts the monitor
61 cluster to retrieve centrally-stored configuration for the entire cluster.
62 After a complete view of the configuration is available, the startup of the
63 daemon or process will commence.
65 .. _bootstrap-options:
70 Bootstrap options are configuration options that affect the process's ability
71 to contact the monitors, to authenticate, and to retrieve the cluster-stored
72 configuration. For this reason, these options might need to be stored locally
73 on the node, and set by means of a local configuration file. These options
74 include the following:
77 .. confval:: mon_host_override
79 - :confval:`mon_dns_srv_name`
80 - :confval:`mon_data`, :confval:`osd_data`, :confval:`mds_data`,
81 :confval:`mgr_data`, and similar options that define which local directory
82 the daemon stores its data in.
83 - :confval:`keyring`, :confval:`keyfile`, and/or :confval:`key`, which can be
84 used to specify the authentication credential to use to authenticate with the
85 monitor. Note that in most cases the default keyring location is in the data
86 directory specified above.
88 In most cases, there is no reason to modify the default values of these
89 options. However, there is one exception to this: the :confval:`mon_host`
90 option that identifies the addresses of the cluster's monitors. But when
91 :ref:`DNS is used to identify monitors<mon-dns-lookup>`, a local Ceph
92 configuration file can be avoided entirely.
95 Skipping monitor config
96 -----------------------
98 The option ``--no-mon-config`` can be passed in any command in order to skip
99 the step that retrieves configuration information from the cluster's monitors.
100 Skipping this retrieval step can be useful in cases where configuration is
101 managed entirely via configuration files, or when maintenance activity needs to
102 be done but the monitor cluster is down.
106 Configuration sections
107 ======================
109 Each of the configuration options associated with a single process or daemon
110 has a single value. However, the values for a configuration option can vary
111 across daemon types, and can vary even across different daemons of the same
112 type. Ceph options that are stored in the monitor configuration database or in
113 local configuration files are grouped into sections |---| so-called "configuration
114 sections" |---| to indicate which daemons or clients they apply to.
117 These sections include the following:
121 Settings under ``global`` affect all daemons and clients
122 in a Ceph Storage Cluster.
124 :example: ``log_file = /var/log/ceph/$cluster-$type.$id.log``
128 Settings under ``mon`` affect all ``ceph-mon`` daemons in
129 the Ceph Storage Cluster, and override the same setting in
132 :example: ``mon_cluster_log_to_syslog = true``
136 Settings in the ``mgr`` section affect all ``ceph-mgr`` daemons in
137 the Ceph Storage Cluster, and override the same setting in
140 :example: ``mgr_stats_period = 10``
144 Settings under ``osd`` affect all ``ceph-osd`` daemons in
145 the Ceph Storage Cluster, and override the same setting in
148 :example: ``osd_op_queue = wpq``
152 Settings in the ``mds`` section affect all ``ceph-mds`` daemons in
153 the Ceph Storage Cluster, and override the same setting in
156 :example: ``mds_cache_memory_limit = 10G``
160 Settings under ``client`` affect all Ceph clients
161 (for example, mounted Ceph File Systems, mounted Ceph Block Devices)
162 as well as RADOS Gateway (RGW) daemons.
164 :example: ``objecter_inflight_ops = 512``
167 Configuration sections can also specify an individual daemon or client name. For example,
168 ``mon.foo``, ``osd.123``, and ``client.smith`` are all valid section names.
171 Any given daemon will draw its settings from the global section, the daemon- or
172 client-type section, and the section sharing its name. Settings in the
173 most-specific section take precedence so precedence: for example, if the same
174 option is specified in both :confsec:`global`, :confsec:`mon`, and ``mon.foo``
175 on the same source (i.e. that is, in the same configuration file), the
176 ``mon.foo`` setting will be used.
178 If multiple values of the same configuration option are specified in the same
179 section, the last value specified takes precedence.
181 Note that values from the local configuration file always take precedence over
182 values from the monitor configuration database, regardless of the section in
185 .. _ceph-metavariables:
190 Metavariables dramatically simplify Ceph storage cluster configuration. When a
191 metavariable is set in a configuration value, Ceph expands the metavariable at
192 the time the configuration value is used. In this way, Ceph metavariables
193 behave similarly to the way that variable expansion works in the Bash shell.
195 Ceph supports the following metavariables:
197 .. describe:: $cluster
199 Expands to the Ceph Storage Cluster name. Useful when running
200 multiple Ceph Storage Clusters on the same hardware.
202 :example: ``/etc/ceph/$cluster.keyring``
207 Expands to a daemon or process type (for example, ``mds``, ``osd``, or ``mon``)
209 :example: ``/var/lib/ceph/$type``
213 Expands to the daemon or client identifier. For
214 ``osd.0``, this would be ``0``; for ``mds.a``, it would
217 :example: ``/var/lib/ceph/$type/$cluster-$id``
221 Expands to the host name where the process is running.
225 Expands to ``$type.$id``.
227 :example: ``/var/run/ceph/$cluster-$name.asok``
231 Expands to daemon pid.
233 :example: ``/var/run/ceph/$cluster-$name-$pid.asok``
236 Ceph configuration file
237 =======================
239 On startup, Ceph processes search for a configuration file in the
242 #. ``$CEPH_CONF`` (that is, the path following the ``$CEPH_CONF``
243 environment variable)
244 #. ``-c path/path`` (that is, the ``-c`` command line argument)
245 #. ``/etc/ceph/$cluster.conf``
246 #. ``~/.ceph/$cluster.conf``
247 #. ``./$cluster.conf`` (that is, in the current working directory)
248 #. On FreeBSD systems only, ``/usr/local/etc/ceph/$cluster.conf``
250 Here ``$cluster`` is the cluster's name (default: ``ceph``).
252 The Ceph configuration file uses an ``ini`` style syntax. You can add "comment
253 text" after a pound sign (#) or a semi-colon semicolon (;). For example:
257 # <--A number (#) sign number sign (#) precedes a comment.
258 ; A comment may be anything.
259 # Comments always follow a semi-colon semicolon (;) or a pound sign (#) on each line.
260 # The end of the line terminates a comment.
261 # We recommend that you provide comments in your configuration file(s).
264 .. _ceph-conf-settings:
266 Config file section names
267 -------------------------
269 The configuration file is divided into sections. Each section must begin with a
270 valid configuration section name (see `Configuration sections`_, above) that is
271 surrounded by square brackets. For example:
287 Config file option values
288 -------------------------
290 The value of a configuration option is a string. If the string is too long to
291 fit on a single line, you can put a backslash (``\``) at the end of the line
292 and the backslash will act as a line continuation marker. In such a case, the
293 value of the option will be the string after ``=`` in the current line,
294 combined with the string in the next line. Here is an example::
300 In this example, the value of the "``foo``" option is "``long long ago long
303 An option value typically ends with either a newline or a comment. For
309 obscure_one = difficult to explain # I will try harder in next release
310 simpler_one = nothing to explain
312 In this example, the value of the "``obscure one``" option is "``difficult to
313 explain``" and the value of the "``simpler one`` options is "``nothing to
316 When an option value contains spaces, it can be enclosed within single quotes
317 or double quotes in order to make its scope clear and in order to make sure
318 that the first space in the value is not interpreted as the end of the value.
324 line = "to be, or not to be"
326 In option values, there are four characters that are treated as escape
327 characters: ``=``, ``#``, ``;`` and ``[``. They are permitted to occur in an
328 option value only if they are immediately preceded by the backslash character
329 (``\``). For example:
334 secret = "i love \# and \["
336 Each configuration option falls under one of the following types:
340 64-bit signed integer. Some SI suffixes are supported, such as "K", "M",
341 "G", "T", "P", and "E" (meaning, respectively, 10\ :sup:`3`, 10\ :sup:`6`,
342 10\ :sup:`9`, etc.). "B" is the only supported unit string. Thus "1K", "1M",
343 "128B" and "-1" are all valid option values. When a negative value is
344 assigned to a threshold option, this can indicate that the option is
345 "unlimited" -- that is, that there is no threshold or limit in effect.
347 :example: ``42``, ``-1``
351 This differs from ``integer`` only in that negative values are not
354 :example: ``256``, ``0``
358 A string encoded in UTF-8. Certain characters are not permitted. Reference
359 the above notes for the details.
361 :example: ``"hello world"``, ``"i love \#"``, ``yet-another-name``
363 .. describe:: boolean
365 Typically either of the two values ``true`` or ``false``. However, any
366 integer is permitted: "0" implies ``false``, and any non-zero value implies
369 :example: ``true``, ``false``, ``1``, ``0``
373 A single address, optionally prefixed with ``v1``, ``v2`` or ``any`` for the
374 messenger protocol. If no prefix is specified, the ``v2`` protocol is used.
375 For more details, see :ref:`address_formats`.
377 :example: ``v1:1.2.3.4:567``, ``v2:1.2.3.4:567``, ``1.2.3.4:567``, ``2409:8a1e:8fb6:aa20:1260:4bff:fe92:18f5::567``, ``[::1]:6789``
379 .. describe:: addrvec
381 A set of addresses separated by ",". The addresses can be optionally quoted
382 with ``[`` and ``]``.
384 :example: ``[v1:1.2.3.4:567,v2:1.2.3.4:568]``, ``v1:1.2.3.4:567,v1:1.2.3.14:567`` ``[2409:8a1e:8fb6:aa20:1260:4bff:fe92:18f5::567], [2409:8a1e:8fb6:aa20:1260:4bff:fe92:18f5::568]``
388 The string format of a uuid defined by `RFC4122
389 <https://www.ietf.org/rfc/rfc4122.txt>`_. Certain variants are also
390 supported: for more details, see `Boost document
391 <https://www.boost.org/doc/libs/1_74_0/libs/uuid/doc/uuid.html#String%20Generator>`_.
393 :example: ``f81d4fae-7dec-11d0-a765-00a0c91e6bf6``
397 64-bit unsigned integer. Both SI prefixes and IEC prefixes are supported.
398 "B" is the only supported unit string. Negative values are not permitted.
400 :example: ``1Ki``, ``1K``, ``1KiB`` and ``1B``.
404 Denotes a duration of time. The default unit of time is the second.
405 The following units of time are supported:
407 * second: ``s``, ``sec``, ``second``, ``seconds``
408 * minute: ``m``, ``min``, ``minute``, ``minutes``
409 * hour: ``hs``, ``hr``, ``hour``, ``hours``
410 * day: ``d``, ``day``, ``days``
411 * week: ``w``, ``wk``, ``week``, ``weeks``
412 * month: ``mo``, ``month``, ``months``
413 * year: ``y``, ``yr``, ``year``, ``years``
415 :example: ``1 m``, ``1m`` and ``1 week``
417 .. _ceph-conf-database:
419 Monitor configuration database
420 ==============================
422 The monitor cluster manages a database of configuration options that can be
423 consumed by the entire cluster. This allows for streamlined central
424 configuration management of the entire system. For ease of administration and
425 transparency, the vast majority of configuration options can and should be
426 stored in this database.
428 Some settings might need to be stored in local configuration files because they
429 affect the ability of the process to connect to the monitors, to authenticate,
430 and to fetch configuration information. In most cases this applies only to the
431 ``mon_host`` option. This issue can be avoided by using :ref:`DNS SRV
432 records<mon-dns-lookup>`.
437 Configuration options stored by the monitor can be stored in a global section,
438 in a daemon-type section, or in a specific daemon section. In this, they are
439 no different from the options in a configuration file.
441 In addition, options may have a *mask* associated with them to further restrict
442 which daemons or clients the option applies to. Masks take two forms:
444 #. ``type:location`` where ``type`` is a CRUSH property like ``rack`` or
445 ``host``, and ``location`` is a value for that property. For example,
446 ``host:foo`` would limit the option only to daemons or clients
447 running on a particular host.
448 #. ``class:device-class`` where ``device-class`` is the name of a CRUSH
449 device class (for example, ``hdd`` or ``ssd``). For example,
450 ``class:ssd`` would limit the option only to OSDs backed by SSDs.
451 (This mask has no effect on non-OSD daemons or clients.)
453 In commands that specify a configuration option, the argument of the option (in
454 the following examples, this is the "who" string) may be a section name, a
455 mask, or a combination of both separated by a slash character (``/``). For
456 example, ``osd/rack:foo`` would refer to all OSD daemons in the ``foo`` rack.
458 When configuration options are shown, the section name and mask are presented
459 in separate fields or columns to make them more readable.
464 The following CLI commands are used to configure the cluster:
466 * ``ceph config dump`` dumps the entire monitor configuration
467 database for the cluster.
469 * ``ceph config get <who>`` dumps the configuration options stored in
470 the monitor configuration database for a specific daemon or client
471 (for example, ``mds.a``).
473 * ``ceph config get <who> <option>`` shows either a configuration value
474 stored in the monitor configuration database for a specific daemon or client
475 (for example, ``mds.a``), or, if that value is not present in the monitor
476 configuration database, the compiled-in default value.
478 * ``ceph config set <who> <option> <value>`` specifies a configuration
479 option in the monitor configuration database.
481 * ``ceph config show <who>`` shows the configuration for a running daemon.
482 These settings might differ from those stored by the monitors if there are
483 also local configuration files in use or if options have been overridden on
484 the command line or at run time. The source of the values of the options is
485 displayed in the output.
487 * ``ceph config assimilate-conf -i <input file> -o <output file>`` ingests a
488 configuration file from *input file* and moves any valid options into the
489 monitor configuration database. Any settings that are unrecognized, are
490 invalid, or cannot be controlled by the monitor will be returned in an
491 abbreviated configuration file stored in *output file*. This command is
492 useful for transitioning from legacy configuration files to centralized
493 monitor-based configuration.
495 Note that ``ceph config set <who> <option> <value>`` and ``ceph config get
496 <who> <option>`` will not necessarily return the same values. The latter
497 command will show compiled-in default values. In order to determine whether a
498 configuration option is present in the monitor configuration database, run
499 ``ceph config dump``.
504 To get help for a particular option, run the following command:
508 ceph config help <option>
514 ceph config help log_file
518 log_file - path to log file
520 Default (non-daemon):
521 Default (daemon): /var/log/ceph/$cluster-$name.log
522 Can update at runtime: false
523 See also: [log_to_stderr,err_to_stderr,log_to_syslog,err_to_syslog]
529 ceph config help log_file -f json-pretty
535 "type": "std::string",
537 "desc": "path to log file",
540 "daemon_default": "/var/log/ceph/$cluster-$name.log",
552 "can_update_at_runtime": false
555 The ``level`` property can be ``basic``, ``advanced``, or ``dev``. The `dev`
556 options are intended for use by developers, generally for testing purposes, and
557 are not recommended for use by operators.
559 .. note:: This command uses the configuration schema that is compiled into the
560 running monitors. If you have a mixed-version cluster (as might exist, for
561 example, during an upgrade), you might want to query the option schema from
562 a specific running daemon by running a command of the following form:
566 ceph daemon <name> config help [option]
571 In most cases, Ceph permits changes to the configuration of a daemon at
572 run time. This can be used for increasing or decreasing the amount of logging
573 output, for enabling or disabling debug settings, and for runtime optimization.
575 Use the ``ceph config set`` command to update configuration options. For
576 example, to enable the most verbose debug log level on a specific OSD, run a
577 command of the following form:
581 ceph config set osd.123 debug_ms 20
583 .. note:: If an option has been customized in a local configuration file, the
585 <https://ceph.io/en/news/blog/2018/new-mimic-centralized-configuration-management/>`_
586 setting will be ignored because it has a lower priority than the local
589 .. note:: Log levels range from 0 to 20.
594 Options can be set temporarily by using the Ceph CLI ``tell`` or ``daemon``
595 interfaces on the Ceph CLI. These *override* values are ephemeral, which means
596 that they affect only the current instance of the daemon and revert to
597 persistently configured values when the daemon restarts.
599 Override values can be set in two ways:
601 #. From any host, send a message to a daemon with a command of the following
606 ceph tell <name> config set <option> <value>
612 ceph tell osd.123 config set debug_osd 20
614 The ``tell`` command can also accept a wildcard as the daemon identifier.
615 For example, to adjust the debug level on all OSD daemons, run a command of
620 ceph tell osd.* config set debug_osd 20
622 #. On the host where the daemon is running, connect to the daemon via a socket
623 in ``/var/run/ceph`` by running a command of the following form:
627 ceph daemon <name> config set <option> <value>
633 ceph daemon osd.4 config set debug_osd 20
635 .. note:: In the output of the ``ceph config show`` command, these temporary
636 values are shown to have a source of ``override``.
639 Viewing runtime settings
640 ========================
642 You can see the current settings specified for a running daemon with the ``ceph
643 config show`` command. For example, to see the (non-default) settings for the
644 daemon ``osd.0``, run the following command:
648 ceph config show osd.0
650 To see a specific setting, run the following command:
654 ceph config show osd.0 debug_osd
656 To see all settings (including those with default values), run the following
661 ceph config show-with-defaults osd.0
663 You can see all settings for a daemon that is currently running by connecting
664 to it on the local host via the admin socket. For example, to dump all
665 current settings, run the following command:
669 ceph daemon osd.0 config show
671 To see non-default settings and to see where each value came from (for example,
672 a config file, the monitor, or an override), run the following command:
676 ceph daemon osd.0 config diff
678 To see the value of a single setting, run the following command:
682 ceph daemon osd.0 config get debug_osd
685 Changes introduced in Octopus
686 =============================
688 The Octopus release changed the way the configuration file is parsed.
689 These changes are as follows:
691 - Repeated configuration options are allowed, and no warnings will be
692 displayed. This means that the setting that comes last in the file is the one
693 that takes effect. Prior to this change, Ceph displayed warning messages when
694 lines containing duplicate options were encountered, such as::
696 warning line 42: 'foo' in section 'bar' redefined
697 - Prior to Octopus, options containing invalid UTF-8 characters were ignored
698 with warning messages. But in Octopus, they are treated as fatal errors.
699 - The backslash character ``\`` is used as the line-continuation marker that
700 combines the next line with the current one. Prior to Octopus, there was a
701 requirement that any end-of-line backslash be followed by a non-empty line.
702 But in Octopus, an empty line following a backslash is allowed.
703 - In the configuration file, each line specifies an individual configuration
704 option. The option's name and its value are separated with ``=``, and the
705 value may be enclosed within single or double quotes. If an invalid
706 configuration is specified, we will treat it as an invalid configuration
709 bad option ==== bad value
710 - Prior to Octopus, if no section name was specified in the configuration file,
711 all options would be set as though they were within the :confsec:`global`
712 section. This approach is discouraged. Since Octopus, any configuration
713 file that has no section name must contain only a single option.
715 .. |---| unicode:: U+2014 .. EM DASH :trim: