7 When you start the Ceph service, the initialization process activates a series
8 of daemons that run in the background. A :term:`Ceph Storage Cluster` runs
9 three types of daemons:
11 - :term:`Ceph Monitor` (``ceph-mon``)
12 - :term:`Ceph Manager` (``ceph-mgr``)
13 - :term:`Ceph OSD Daemon` (``ceph-osd``)
15 Ceph Storage Clusters that support the :term:`Ceph Filesystem` run at
16 least one :term:`Ceph Metadata Server` (``ceph-mds``). Clusters that
17 support :term:`Ceph Object Storage` run Ceph Gateway daemons
20 Each daemon has a series of configuration options, each of which has a
21 default values. You may adjust the behavior of the system by changing these
22 configuration options.
27 All Ceph configuration options have a unique name consisting of words
28 formed with lower-case characters and connected with underscore
31 When option names are specified on the command line, either underscore
32 (``_``) or dash (``-``) characters can be used interchangeable (e.g.,
33 ``--mon-host`` is equivalent to ``--mon_host``).
35 When option names appear in configuration files, spaces can also be
36 used in place of underscore or dash.
41 Each Ceph daemon, process, and library will pull its configuration
42 from several sources, listed below. Sources later in the list will
43 override those earlier in the list when both are present.
45 - the compiled-in default value
46 - the monitor cluster's centralized configuration database
47 - a configuration file stored on the local host
48 - environment variables
49 - command line arguments
50 - runtime overrides set by an administrator
52 One of the first things a Ceph process does on startup is parse the
53 configuration options provided via the command line, environment, and
54 local configuration file. The process will then contact the monitor
55 cluster to retrieve configuration stored centrally for the entire
56 cluster. Once a complete view of the configuration is available, the
57 daemon or process startup will proceed.
62 Because some configuration options affect the process's ability to
63 contact the monitors, authenticate, and retrieve the cluster-stored
64 configuration, they may need to be stored locally on the node and set
65 in a local configuration file. These options include:
67 - ``mon_host``, the list of monitors for the cluster
68 - ``mon_dns_serv_name`` (default: `ceph-mon`), the name of the DNS
69 SRV record to check to identify the cluster monitors via DNS
70 - ``mon_data``, ``osd_data``, ``mds_data``, ``mgr_data``, and
71 similar options that define which local directory the daemon
73 - ``keyring``, ``keyfile``, and/or ``key``, which can be used to
74 specify the authentication credential to use to authenticate with
75 the monitor. Note that in most cases the default keyring location
76 is in the data directory specified above.
78 In the vast majority of cases the default values of these are
79 appropriate, with the exception of the ``mon_host`` option that
80 identifies the addresses of the cluster's monitors. When DNS is used
81 to identify monitors a local ceph configuration file can be avoided
84 Skipping monitor config
85 -----------------------
87 Any process may be passed the option ``--no-mon-config`` to skip the
88 step that retrieves configuration from the cluster monitors. This is
89 useful in cases where configuration is managed entirely via
90 configuration files or where the monitor cluster is currently down but
91 some maintenance activity needs to be done.
97 Configuration sections
98 ======================
100 Any given process or daemon has a single value for each configuration
101 option. However, values for an option may vary across different
102 daemon types even daemons of the same type. Ceph options that are
103 stored in the monitor configuration database or in local configuration
104 files are grouped into sections to indicate which daemons or clients
107 These sections include:
111 :Description: Settings under ``global`` affect all daemons and clients
112 in a Ceph Storage Cluster.
114 :Example: ``log_file = /var/log/ceph/$cluster-$type.$id.log``
118 :Description: Settings under ``mon`` affect all ``ceph-mon`` daemons in
119 the Ceph Storage Cluster, and override the same setting in
122 :Example: ``mon_cluster_log_to_syslog = true``
127 :Description: Settings in the ``mgr`` section affect all ``ceph-mgr`` daemons in
128 the Ceph Storage Cluster, and override the same setting in
131 :Example: ``mgr_stats_period = 10``
135 :Description: Settings under ``osd`` affect all ``ceph-osd`` daemons in
136 the Ceph Storage Cluster, and override the same setting in
139 :Example: ``osd_op_queue = wpq``
143 :Description: Settings in the ``mds`` section affect all ``ceph-mds`` daemons in
144 the Ceph Storage Cluster, and override the same setting in
147 :Example: ``mds_cache_size = 10G``
151 :Description: Settings under ``client`` affect all Ceph Clients
152 (e.g., mounted Ceph Filesystems, mounted Ceph Block Devices,
153 etc.) as well as Rados Gateway (RGW) daemons.
155 :Example: ``objecter_inflight_ops = 512``
158 Sections may also specify an individual daemon or client name. For example,
159 ``mon.foo``, ``osd.123``, and ``client.smith`` are all valid section names.
162 Any given daemon will draw its settings from the global section, the
163 daemon or client type section, and the section sharing its name.
164 Settings in the most-specific section take precedence, so for example
165 if the same option is specified in both ``global``, ``mon``, and
166 ``mon.foo`` on the same source (i.e., in the same configurationfile),
167 the ``mon.foo`` value will be used.
169 Note that values from the local configuration file always take
170 precedence over values from the monitor configuration database,
171 regardless of which section they appear in.
174 .. _ceph-metavariables:
179 Metavariables simplify Ceph Storage Cluster configuration
180 dramatically. When a metavariable is set in a configuration value,
181 Ceph expands the metavariable into a concrete value at the time the
182 configuration value is used. Ceph metavariables are similar to variable expansion in the Bash shell.
184 Ceph supports the following metavariables:
188 :Description: Expands to the Ceph Storage Cluster name. Useful when running
189 multiple Ceph Storage Clusters on the same hardware.
191 :Example: ``/etc/ceph/$cluster.keyring``
197 :Description: Expands to a daemon or process type (e.g., ``mds``, ``osd``, or ``mon``)
199 :Example: ``/var/lib/ceph/$type``
204 :Description: Expands to the daemon or client identifier. For
205 ``osd.0``, this would be ``0``; for ``mds.a``, it would
208 :Example: ``/var/lib/ceph/$type/$cluster-$id``
213 :Description: Expands to the host name where the process is running.
218 :Description: Expands to ``$type.$id``.
219 :Example: ``/var/run/ceph/$cluster-$name.asok``
223 :Description: Expands to daemon pid.
224 :Example: ``/var/run/ceph/$cluster-$name-$pid.asok``
228 The Configuration File
229 ======================
231 On startup, Ceph processes search for a configuration file in the
234 #. ``$CEPH_CONF`` (*i.e.,* the path following the ``$CEPH_CONF``
235 environment variable)
236 #. ``-c path/path`` (*i.e.,* the ``-c`` command line argument)
237 #. ``/etc/ceph/$cluster.conf``
238 #. ``~/.ceph/$cluster.conf``
239 #. ``./$cluster.conf`` (*i.e.,* in the current working directory)
240 #. On FreeBSD systems only, ``/usr/local/etc/ceph/$cluster.conf``
242 where ``$cluster`` is the cluster's name (default ``ceph``).
244 The Ceph configuration file uses an *ini* style syntax. You can add comments
245 by preceding comments with a pound sign (#) or a semi-colon (;). For example:
249 # <--A number (#) sign precedes a comment.
250 ; A comment may be anything.
251 # Comments always follow a semi-colon (;) or a pound (#) on each line.
252 # The end of the line terminates a comment.
253 # We recommend that you provide comments in your configuration file(s).
256 .. _ceph-conf-settings:
258 Config file section names
259 -------------------------
261 The configuration file is divided into sections. Each section must begin with a
262 valid configuration section name (see `Configuration sections`_, above)
263 surrounded by square brackets. For example,
281 Monitor configuration database
282 ==============================
284 The monitor cluster manages a database of configuration options that
285 can be consumed by the entire cluster, enabling streamlined central
286 configuration management for the entire system. The vast majority of
287 configuration options can and should be stored here for ease of
288 administration and transparency.
290 A handful of settings may still need to be stored in local
291 configuration files because they affect the ability to connect to the
292 monitors, authenticate, and fetch configuration information. In most
293 cases this is limited to the ``mon_host`` option, although this can
294 also be avoided through the use of DNS SRV records.
299 Configuration options stored by the monitor can live in a global
300 section, daemon type section, or specific daemon section, just like
301 options in a configuration file can.
303 In addition, options may also have a *mask* associated with them to
304 further restrict which daemons or clients the option applies to.
305 Masks take two forms:
307 #. ``type:location`` where *type* is a CRUSH property like `rack` or
308 `host`, and *location* is a value for that property. For example,
309 ``host:foo`` would limit the option only to daemons or clients
310 running on a particular host.
311 #. ``class:device-class`` where *device-class* is the name of a CRUSH
312 device class (e.g., ``hdd`` or ``ssd``). For example,
313 ``class:ssd`` would limit the option only to OSDs backed by SSDs.
314 (This mask has no effect for non-OSD daemons or clients.)
316 When setting a configuration option, the `who` may be a section name,
317 a mask, or a combination of both separated by a slash (``/``)
318 character. For example, ``osd/rack:foo`` would mean all OSD daemons
321 When viewing configuration options, the section name and mask are
322 generally separated out into separate fields or columns to ease readability.
328 The following CLI commands are used to configure the cluster:
330 * ``ceph config dump`` will dump the entire configuration database for
333 * ``ceph config get <who>`` will dump the configuration for a specific
334 daemon or client (e.g., ``mds.a``), as stored in the monitors'
335 configuration database.
337 * ``ceph config set <who> <option> <value>`` will set a configuration
338 option in the monitors' configuration database.
340 * ``ceph config show <who>`` will show the reported running
341 configuration for a running daemon. These settings may differ from
342 those stored by the monitors if there are also local configuration
343 files in use or options have been overridden on the command line or
344 at run time. The source of the option values is reported as part
347 * ``ceph config assimilate-conf -i <input file> -o <output file>``
348 will ingest a configuration file from *input file* and move any
349 valid options into the monitors' configuration database. Any
350 settings that are unrecognized, invalid, or cannot be controlled by
351 the monitor will be returned in an abbreviated config file stored in
352 *output file*. This command is useful for transitioning from legacy
353 configuration files to centralized monitor-based configuration.
359 You can get help for a particular option with::
361 ceph config help <option>
363 Note that this will use the configuration schema that is compiled into the running monitors. If you have a mixed-version cluster (e.g., during an upgrade), you might also want to query the option schema from a specific running daemon::
365 ceph daemon <name> config help [option]
369 $ ceph config help log_file
370 log_file - path to log file
372 Default (non-daemon):
373 Default (daemon): /var/log/ceph/$cluster-$name.log
374 Can update at runtime: false
375 See also: [log_to_stderr,err_to_stderr,log_to_syslog,err_to_syslog]
379 $ ceph config help log_file -f json-pretty
382 "type": "std::string",
384 "desc": "path to log file",
387 "daemon_default": "/var/log/ceph/$cluster-$name.log",
399 "can_update_at_runtime": false
402 The ``level`` property can be any of `basic`, `advanced`, or `dev`.
403 The `dev` options are intended for use by developers, generally for
404 testing purposes, and are not recommended for use by operators.
410 In most cases, Ceph allows you to make changes to the configuration of
411 a daemon at runtime. This capability is quite useful for
412 increasing/decreasing logging output, enabling/disabling debug
413 settings, and even for runtime optimization.
415 Generally speaking, configuration options can be updated in the usual
416 way via the ``ceph config set`` command. For example, do enable the debug log level on a specific OSD,::
418 ceph config set osd.123 debug_ms 20
420 Note that if the same option is also customized in a local
421 configuration file, the monitor setting will be ignored (it has a
422 lower priority than the local config file).
427 You can also temporarily set an option using the `tell` or `daemon`
428 interfaces on the Ceph CLI. These *override* values are ephemeral in
429 that they only affect the running process and are discarded/lost if
430 the daemon or process restarts.
432 Override values can be set in two ways:
434 #. From any host, we can send a message to a daemon over the network with::
436 ceph tell <name> config set <option> <value>
440 ceph tell osd.123 config set debug_osd 20
442 The `tell` command can also accept a wildcard for the daemon
443 identifier. For example, to adjust the debug level on all OSD
446 ceph tell osd.* config set debug_osd 20
448 #. From the host the process is running on, we can connect directly to
449 the process via a socket in ``/var/run/ceph`` with::
451 ceph daemon <name> config set <option> <value>
455 ceph daemon osd.4 config set debug_osd 20
457 Note that in the ``ceph config show`` command output these temporary
458 values will be shown with a source of ``override``.
461 Viewing runtime settings
462 ========================
464 You can see the current options set for a running daemon with the ``ceph config show`` command. For example,::
466 ceph config show osd.0
468 will show you the (non-default) options for that daemon. You can also look at a specific option with::
470 ceph config show osd.0 debug_osd
472 or view all options (even those with default values) with::
474 ceph config show-with-defaults osd.0
476 You can also observe settings for a running daemon by connecting to it from the local host via the admin socket. For example,::
478 ceph daemon osd.0 config show
480 will dump all current settings,::
482 ceph daemon osd.0 config diff
484 will show only non-default settings (as well as where the value came from: a config file, the monitor, an override, etc.), and::
486 ceph daemon osd.0 config get debug_osd
488 will report the value of a single option.