]>
Commit | Line | Data |
---|---|---|
81eedcae TL |
1 | .. _configuring-ceph: |
2 | ||
7c673cae FG |
3 | ================== |
4 | Configuring Ceph | |
5 | ================== | |
6 | ||
7 | When you start the Ceph service, the initialization process activates a series | |
8 | of daemons that run in the background. A :term:`Ceph Storage Cluster` runs | |
11fdf7f2 | 9 | three types of daemons: |
7c673cae FG |
10 | |
11 | - :term:`Ceph Monitor` (``ceph-mon``) | |
11fdf7f2 | 12 | - :term:`Ceph Manager` (``ceph-mgr``) |
7c673cae FG |
13 | - :term:`Ceph OSD Daemon` (``ceph-osd``) |
14 | ||
11fdf7f2 TL |
15 | Ceph Storage Clusters that support the :term:`Ceph Filesystem` run at |
16 | least one :term:`Ceph Metadata Server` (``ceph-mds``). Clusters that | |
17 | support :term:`Ceph Object Storage` run Ceph Gateway daemons | |
18 | (``radosgw``). | |
7c673cae | 19 | |
11fdf7f2 TL |
20 | Each daemon has a series of configuration options, each of which has a |
21 | default values. You may adjust the behavior of the system by changing these | |
22 | configuration options. | |
7c673cae | 23 | |
11fdf7f2 TL |
24 | Option names |
25 | ============ | |
7c673cae | 26 | |
11fdf7f2 TL |
27 | All Ceph configuration options have a unique name consisting of words |
28 | formed with lower-case characters and connected with underscore | |
29 | (``_``) characters. | |
7c673cae | 30 | |
11fdf7f2 TL |
31 | When option names are specified on the command line, either underscore |
32 | (``_``) or dash (``-``) characters can be used interchangeable (e.g., | |
33 | ``--mon-host`` is equivalent to ``--mon_host``). | |
7c673cae | 34 | |
11fdf7f2 TL |
35 | When option names appear in configuration files, spaces can also be |
36 | used in place of underscore or dash. | |
7c673cae | 37 | |
11fdf7f2 TL |
38 | Config sources |
39 | ============== | |
7c673cae | 40 | |
11fdf7f2 TL |
41 | Each Ceph daemon, process, and library will pull its configuration |
42 | from several sources, listed below. Sources later in the list will | |
43 | override those earlier in the list when both are present. | |
44 | ||
45 | - the compiled-in default value | |
46 | - the monitor cluster's centralized configuration database | |
47 | - a configuration file stored on the local host | |
48 | - environment variables | |
49 | - command line arguments | |
50 | - runtime overrides set by an administrator | |
51 | ||
52 | One of the first things a Ceph process does on startup is parse the | |
53 | configuration options provided via the command line, environment, and | |
54 | local configuration file. The process will then contact the monitor | |
55 | cluster to retrieve configuration stored centrally for the entire | |
56 | cluster. Once a complete view of the configuration is available, the | |
57 | daemon or process startup will proceed. | |
58 | ||
59 | Bootstrap options | |
60 | ----------------- | |
61 | ||
62 | Because some configuration options affect the process's ability to | |
63 | contact the monitors, authenticate, and retrieve the cluster-stored | |
64 | configuration, they may need to be stored locally on the node and set | |
65 | in a local configuration file. These options include: | |
66 | ||
67 | - ``mon_host``, the list of monitors for the cluster | |
68 | - ``mon_dns_serv_name`` (default: `ceph-mon`), the name of the DNS | |
69 | SRV record to check to identify the cluster monitors via DNS | |
70 | - ``mon_data``, ``osd_data``, ``mds_data``, ``mgr_data``, and | |
71 | similar options that define which local directory the daemon | |
72 | stores its data in. | |
73 | - ``keyring``, ``keyfile``, and/or ``key``, which can be used to | |
74 | specify the authentication credential to use to authenticate with | |
75 | the monitor. Note that in most cases the default keyring location | |
76 | is in the data directory specified above. | |
77 | ||
78 | In the vast majority of cases the default values of these are | |
79 | appropriate, with the exception of the ``mon_host`` option that | |
80 | identifies the addresses of the cluster's monitors. When DNS is used | |
81 | to identify monitors a local ceph configuration file can be avoided | |
82 | entirely. | |
83 | ||
84 | Skipping monitor config | |
85 | ----------------------- | |
86 | ||
87 | Any process may be passed the option ``--no-mon-config`` to skip the | |
88 | step that retrieves configuration from the cluster monitors. This is | |
89 | useful in cases where configuration is managed entirely via | |
90 | configuration files or where the monitor cluster is currently down but | |
91 | some maintenance activity needs to be done. | |
7c673cae FG |
92 | |
93 | ||
11fdf7f2 | 94 | .. _ceph-conf-file: |
7c673cae | 95 | |
7c673cae | 96 | |
11fdf7f2 TL |
97 | Configuration sections |
98 | ====================== | |
7c673cae | 99 | |
11fdf7f2 TL |
100 | Any given process or daemon has a single value for each configuration |
101 | option. However, values for an option may vary across different | |
102 | daemon types even daemons of the same type. Ceph options that are | |
103 | stored in the monitor configuration database or in local configuration | |
104 | files are grouped into sections to indicate which daemons or clients | |
105 | they apply to. | |
7c673cae | 106 | |
11fdf7f2 | 107 | These sections include: |
7c673cae | 108 | |
11fdf7f2 | 109 | ``global`` |
7c673cae | 110 | |
11fdf7f2 TL |
111 | :Description: Settings under ``global`` affect all daemons and clients |
112 | in a Ceph Storage Cluster. | |
7c673cae | 113 | |
11fdf7f2 | 114 | :Example: ``log_file = /var/log/ceph/$cluster-$type.$id.log`` |
7c673cae | 115 | |
11fdf7f2 | 116 | ``mon`` |
7c673cae | 117 | |
11fdf7f2 | 118 | :Description: Settings under ``mon`` affect all ``ceph-mon`` daemons in |
7c673cae | 119 | the Ceph Storage Cluster, and override the same setting in |
11fdf7f2 | 120 | ``global``. |
7c673cae | 121 | |
11fdf7f2 | 122 | :Example: ``mon_cluster_log_to_syslog = true`` |
7c673cae FG |
123 | |
124 | ||
11fdf7f2 | 125 | ``mgr`` |
7c673cae | 126 | |
11fdf7f2 | 127 | :Description: Settings in the ``mgr`` section affect all ``ceph-mgr`` daemons in |
7c673cae | 128 | the Ceph Storage Cluster, and override the same setting in |
11fdf7f2 | 129 | ``global``. |
7c673cae | 130 | |
11fdf7f2 | 131 | :Example: ``mgr_stats_period = 10`` |
7c673cae | 132 | |
11fdf7f2 | 133 | ``osd`` |
7c673cae | 134 | |
11fdf7f2 TL |
135 | :Description: Settings under ``osd`` affect all ``ceph-osd`` daemons in |
136 | the Ceph Storage Cluster, and override the same setting in | |
137 | ``global``. | |
7c673cae | 138 | |
11fdf7f2 | 139 | :Example: ``osd_op_queue = wpq`` |
7c673cae | 140 | |
11fdf7f2 | 141 | ``mds`` |
7c673cae | 142 | |
11fdf7f2 TL |
143 | :Description: Settings in the ``mds`` section affect all ``ceph-mds`` daemons in |
144 | the Ceph Storage Cluster, and override the same setting in | |
145 | ``global``. | |
7c673cae | 146 | |
11fdf7f2 | 147 | :Example: ``mds_cache_size = 10G`` |
7c673cae | 148 | |
11fdf7f2 | 149 | ``client`` |
7c673cae | 150 | |
11fdf7f2 TL |
151 | :Description: Settings under ``client`` affect all Ceph Clients |
152 | (e.g., mounted Ceph Filesystems, mounted Ceph Block Devices, | |
153 | etc.) as well as Rados Gateway (RGW) daemons. | |
7c673cae | 154 | |
11fdf7f2 | 155 | :Example: ``objecter_inflight_ops = 512`` |
7c673cae | 156 | |
7c673cae | 157 | |
11fdf7f2 TL |
158 | Sections may also specify an individual daemon or client name. For example, |
159 | ``mon.foo``, ``osd.123``, and ``client.smith`` are all valid section names. | |
7c673cae | 160 | |
7c673cae | 161 | |
11fdf7f2 TL |
162 | Any given daemon will draw its settings from the global section, the |
163 | daemon or client type section, and the section sharing its name. | |
164 | Settings in the most-specific section take precedence, so for example | |
165 | if the same option is specified in both ``global``, ``mon``, and | |
166 | ``mon.foo`` on the same source (i.e., in the same configurationfile), | |
167 | the ``mon.foo`` value will be used. | |
7c673cae | 168 | |
11fdf7f2 TL |
169 | Note that values from the local configuration file always take |
170 | precedence over values from the monitor configuration database, | |
171 | regardless of which section they appear in. | |
7c673cae FG |
172 | |
173 | ||
174 | .. _ceph-metavariables: | |
175 | ||
176 | Metavariables | |
177 | ============= | |
178 | ||
11fdf7f2 TL |
179 | Metavariables simplify Ceph Storage Cluster configuration |
180 | dramatically. When a metavariable is set in a configuration value, | |
181 | Ceph expands the metavariable into a concrete value at the time the | |
182 | configuration value is used. Ceph metavariables are similar to variable expansion in the Bash shell. | |
7c673cae FG |
183 | |
184 | Ceph supports the following metavariables: | |
185 | ||
7c673cae FG |
186 | ``$cluster`` |
187 | ||
188 | :Description: Expands to the Ceph Storage Cluster name. Useful when running | |
189 | multiple Ceph Storage Clusters on the same hardware. | |
190 | ||
191 | :Example: ``/etc/ceph/$cluster.keyring`` | |
192 | :Default: ``ceph`` | |
193 | ||
194 | ||
195 | ``$type`` | |
196 | ||
11fdf7f2 | 197 | :Description: Expands to a daemon or process type (e.g., ``mds``, ``osd``, or ``mon``) |
7c673cae FG |
198 | |
199 | :Example: ``/var/lib/ceph/$type`` | |
200 | ||
201 | ||
202 | ``$id`` | |
203 | ||
11fdf7f2 TL |
204 | :Description: Expands to the daemon or client identifier. For |
205 | ``osd.0``, this would be ``0``; for ``mds.a``, it would | |
206 | be ``a``. | |
7c673cae FG |
207 | |
208 | :Example: ``/var/lib/ceph/$type/$cluster-$id`` | |
209 | ||
210 | ||
211 | ``$host`` | |
212 | ||
11fdf7f2 | 213 | :Description: Expands to the host name where the process is running. |
7c673cae FG |
214 | |
215 | ||
216 | ``$name`` | |
217 | ||
218 | :Description: Expands to ``$type.$id``. | |
219 | :Example: ``/var/run/ceph/$cluster-$name.asok`` | |
220 | ||
221 | ``$pid`` | |
222 | ||
223 | :Description: Expands to daemon pid. | |
224 | :Example: ``/var/run/ceph/$cluster-$name-$pid.asok`` | |
225 | ||
226 | ||
7c673cae | 227 | |
11fdf7f2 TL |
228 | The Configuration File |
229 | ====================== | |
7c673cae | 230 | |
11fdf7f2 TL |
231 | On startup, Ceph processes search for a configuration file in the |
232 | following locations: | |
7c673cae | 233 | |
11fdf7f2 TL |
234 | #. ``$CEPH_CONF`` (*i.e.,* the path following the ``$CEPH_CONF`` |
235 | environment variable) | |
236 | #. ``-c path/path`` (*i.e.,* the ``-c`` command line argument) | |
237 | #. ``/etc/ceph/$cluster.conf`` | |
238 | #. ``~/.ceph/$cluster.conf`` | |
239 | #. ``./$cluster.conf`` (*i.e.,* in the current working directory) | |
240 | #. On FreeBSD systems only, ``/usr/local/etc/ceph/$cluster.conf`` | |
7c673cae | 241 | |
11fdf7f2 | 242 | where ``$cluster`` is the cluster's name (default ``ceph``). |
7c673cae | 243 | |
11fdf7f2 TL |
244 | The Ceph configuration file uses an *ini* style syntax. You can add comments |
245 | by preceding comments with a pound sign (#) or a semi-colon (;). For example: | |
7c673cae | 246 | |
11fdf7f2 | 247 | .. code-block:: ini |
7c673cae | 248 | |
11fdf7f2 TL |
249 | # <--A number (#) sign precedes a comment. |
250 | ; A comment may be anything. | |
251 | # Comments always follow a semi-colon (;) or a pound (#) on each line. | |
252 | # The end of the line terminates a comment. | |
253 | # We recommend that you provide comments in your configuration file(s). | |
7c673cae | 254 | |
7c673cae | 255 | |
11fdf7f2 | 256 | .. _ceph-conf-settings: |
7c673cae | 257 | |
11fdf7f2 TL |
258 | Config file section names |
259 | ------------------------- | |
7c673cae | 260 | |
11fdf7f2 TL |
261 | The configuration file is divided into sections. Each section must begin with a |
262 | valid configuration section name (see `Configuration sections`_, above) | |
263 | surrounded by square brackets. For example, | |
7c673cae | 264 | |
11fdf7f2 | 265 | .. code-block:: ini |
7c673cae | 266 | |
11fdf7f2 TL |
267 | [global] |
268 | debug ms = 0 | |
7c673cae | 269 | |
11fdf7f2 TL |
270 | [osd] |
271 | debug ms = 1 | |
7c673cae | 272 | |
11fdf7f2 TL |
273 | [osd.1] |
274 | debug ms = 10 | |
7c673cae | 275 | |
11fdf7f2 TL |
276 | [osd.2] |
277 | debug ms = 10 | |
7c673cae | 278 | |
7c673cae FG |
279 | |
280 | ||
11fdf7f2 TL |
281 | Monitor configuration database |
282 | ============================== | |
7c673cae | 283 | |
11fdf7f2 TL |
284 | The monitor cluster manages a database of configuration options that |
285 | can be consumed by the entire cluster, enabling streamlined central | |
286 | configuration management for the entire system. The vast majority of | |
287 | configuration options can and should be stored here for ease of | |
288 | administration and transparency. | |
7c673cae | 289 | |
11fdf7f2 TL |
290 | A handful of settings may still need to be stored in local |
291 | configuration files because they affect the ability to connect to the | |
292 | monitors, authenticate, and fetch configuration information. In most | |
293 | cases this is limited to the ``mon_host`` option, although this can | |
294 | also be avoided through the use of DNS SRV records. | |
7c673cae | 295 | |
11fdf7f2 TL |
296 | Sections and masks |
297 | ------------------ | |
7c673cae | 298 | |
11fdf7f2 TL |
299 | Configuration options stored by the monitor can live in a global |
300 | section, daemon type section, or specific daemon section, just like | |
301 | options in a configuration file can. | |
7c673cae | 302 | |
11fdf7f2 TL |
303 | In addition, options may also have a *mask* associated with them to |
304 | further restrict which daemons or clients the option applies to. | |
305 | Masks take two forms: | |
7c673cae | 306 | |
11fdf7f2 TL |
307 | #. ``type:location`` where *type* is a CRUSH property like `rack` or |
308 | `host`, and *location* is a value for that property. For example, | |
309 | ``host:foo`` would limit the option only to daemons or clients | |
310 | running on a particular host. | |
311 | #. ``class:device-class`` where *device-class* is the name of a CRUSH | |
312 | device class (e.g., ``hdd`` or ``ssd``). For example, | |
313 | ``class:ssd`` would limit the option only to OSDs backed by SSDs. | |
314 | (This mask has no effect for non-OSD daemons or clients.) | |
7c673cae | 315 | |
11fdf7f2 TL |
316 | When setting a configuration option, the `who` may be a section name, |
317 | a mask, or a combination of both separated by a slash (``/``) | |
318 | character. For example, ``osd/rack:foo`` would mean all OSD daemons | |
319 | in the ``foo`` rack. | |
7c673cae | 320 | |
11fdf7f2 TL |
321 | When viewing configuration options, the section name and mask are |
322 | generally separated out into separate fields or columns to ease readability. | |
7c673cae FG |
323 | |
324 | ||
11fdf7f2 TL |
325 | Commands |
326 | -------- | |
7c673cae | 327 | |
11fdf7f2 | 328 | The following CLI commands are used to configure the cluster: |
7c673cae | 329 | |
11fdf7f2 TL |
330 | * ``ceph config dump`` will dump the entire configuration database for |
331 | the cluster. | |
7c673cae | 332 | |
11fdf7f2 TL |
333 | * ``ceph config get <who>`` will dump the configuration for a specific |
334 | daemon or client (e.g., ``mds.a``), as stored in the monitors' | |
335 | configuration database. | |
7c673cae | 336 | |
11fdf7f2 TL |
337 | * ``ceph config set <who> <option> <value>`` will set a configuration |
338 | option in the monitors' configuration database. | |
7c673cae | 339 | |
11fdf7f2 TL |
340 | * ``ceph config show <who>`` will show the reported running |
341 | configuration for a running daemon. These settings may differ from | |
342 | those stored by the monitors if there are also local configuration | |
343 | files in use or options have been overridden on the command line or | |
344 | at run time. The source of the option values is reported as part | |
345 | of the output. | |
7c673cae | 346 | |
11fdf7f2 TL |
347 | * ``ceph config assimilate-conf -i <input file> -o <output file>`` |
348 | will ingest a configuration file from *input file* and move any | |
349 | valid options into the monitors' configuration database. Any | |
350 | settings that are unrecognized, invalid, or cannot be controlled by | |
351 | the monitor will be returned in an abbreviated config file stored in | |
352 | *output file*. This command is useful for transitioning from legacy | |
353 | configuration files to centralized monitor-based configuration. | |
7c673cae | 354 | |
7c673cae | 355 | |
11fdf7f2 TL |
356 | Help |
357 | ==== | |
7c673cae | 358 | |
11fdf7f2 TL |
359 | You can get help for a particular option with:: |
360 | ||
361 | ceph config help <option> | |
362 | ||
363 | Note that this will use the configuration schema that is compiled into the running monitors. If you have a mixed-version cluster (e.g., during an upgrade), you might also want to query the option schema from a specific running daemon:: | |
364 | ||
365 | ceph daemon <name> config help [option] | |
366 | ||
367 | For example,:: | |
368 | ||
369 | $ ceph config help log_file | |
370 | log_file - path to log file | |
371 | (std::string, basic) | |
372 | Default (non-daemon): | |
373 | Default (daemon): /var/log/ceph/$cluster-$name.log | |
374 | Can update at runtime: false | |
375 | See also: [log_to_stderr,err_to_stderr,log_to_syslog,err_to_syslog] | |
376 | ||
377 | or:: | |
378 | ||
379 | $ ceph config help log_file -f json-pretty | |
380 | { | |
381 | "name": "log_file", | |
382 | "type": "std::string", | |
383 | "level": "basic", | |
384 | "desc": "path to log file", | |
385 | "long_desc": "", | |
386 | "default": "", | |
387 | "daemon_default": "/var/log/ceph/$cluster-$name.log", | |
388 | "tags": [], | |
389 | "services": [], | |
390 | "see_also": [ | |
391 | "log_to_stderr", | |
392 | "err_to_stderr", | |
393 | "log_to_syslog", | |
394 | "err_to_syslog" | |
395 | ], | |
396 | "enum_values": [], | |
397 | "min": "", | |
398 | "max": "", | |
399 | "can_update_at_runtime": false | |
400 | } | |
401 | ||
402 | The ``level`` property can be any of `basic`, `advanced`, or `dev`. | |
403 | The `dev` options are intended for use by developers, generally for | |
404 | testing purposes, and are not recommended for use by operators. | |
7c673cae | 405 | |
7c673cae FG |
406 | |
407 | Runtime Changes | |
408 | =============== | |
409 | ||
11fdf7f2 TL |
410 | In most cases, Ceph allows you to make changes to the configuration of |
411 | a daemon at runtime. This capability is quite useful for | |
412 | increasing/decreasing logging output, enabling/disabling debug | |
413 | settings, and even for runtime optimization. | |
c07f9fc5 | 414 | |
11fdf7f2 TL |
415 | Generally speaking, configuration options can be updated in the usual |
416 | way via the ``ceph config set`` command. For example, do enable the debug log level on a specific OSD,:: | |
c07f9fc5 | 417 | |
11fdf7f2 | 418 | ceph config set osd.123 debug_ms 20 |
c07f9fc5 | 419 | |
11fdf7f2 TL |
420 | Note that if the same option is also customized in a local |
421 | configuration file, the monitor setting will be ignored (it has a | |
422 | lower priority than the local config file). | |
c07f9fc5 | 423 | |
11fdf7f2 TL |
424 | Override values |
425 | --------------- | |
c07f9fc5 | 426 | |
11fdf7f2 TL |
427 | You can also temporarily set an option using the `tell` or `daemon` |
428 | interfaces on the Ceph CLI. These *override* values are ephemeral in | |
429 | that they only affect the running process and are discarded/lost if | |
430 | the daemon or process restarts. | |
c07f9fc5 | 431 | |
11fdf7f2 | 432 | Override values can be set in two ways: |
c07f9fc5 | 433 | |
11fdf7f2 | 434 | #. From any host, we can send a message to a daemon over the network with:: |
c07f9fc5 | 435 | |
11fdf7f2 | 436 | ceph tell <name> config set <option> <value> |
c07f9fc5 | 437 | |
11fdf7f2 | 438 | For example,:: |
c07f9fc5 | 439 | |
11fdf7f2 | 440 | ceph tell osd.123 config set debug_osd 20 |
c07f9fc5 | 441 | |
11fdf7f2 TL |
442 | The `tell` command can also accept a wildcard for the daemon |
443 | identifier. For example, to adjust the debug level on all OSD | |
444 | daemons,:: | |
c07f9fc5 | 445 | |
11fdf7f2 | 446 | ceph tell osd.* config set debug_osd 20 |
c07f9fc5 | 447 | |
11fdf7f2 TL |
448 | #. From the host the process is running on, we can connect directly to |
449 | the process via a socket in ``/var/run/ceph`` with:: | |
c07f9fc5 | 450 | |
11fdf7f2 | 451 | ceph daemon <name> config set <option> <value> |
c07f9fc5 | 452 | |
11fdf7f2 | 453 | For example,:: |
c07f9fc5 | 454 | |
11fdf7f2 | 455 | ceph daemon osd.4 config set debug_osd 20 |
c07f9fc5 | 456 | |
11fdf7f2 TL |
457 | Note that in the ``ceph config show`` command output these temporary |
458 | values will be shown with a source of ``override``. | |
c07f9fc5 | 459 | |
c07f9fc5 | 460 | |
11fdf7f2 TL |
461 | Viewing runtime settings |
462 | ======================== | |
c07f9fc5 | 463 | |
11fdf7f2 | 464 | You can see the current options set for a running daemon with the ``ceph config show`` command. For example,:: |
c07f9fc5 | 465 | |
11fdf7f2 | 466 | ceph config show osd.0 |
c07f9fc5 | 467 | |
11fdf7f2 | 468 | will show you the (non-default) options for that daemon. You can also look at a specific option with:: |
c07f9fc5 | 469 | |
11fdf7f2 | 470 | ceph config show osd.0 debug_osd |
c07f9fc5 | 471 | |
11fdf7f2 | 472 | or view all options (even those with default values) with:: |
c07f9fc5 | 473 | |
11fdf7f2 | 474 | ceph config show-with-defaults osd.0 |
c07f9fc5 | 475 | |
11fdf7f2 | 476 | You can also observe settings for a running daemon by connecting to it from the local host via the admin socket. For example,:: |
c07f9fc5 | 477 | |
11fdf7f2 | 478 | ceph daemon osd.0 config show |
c07f9fc5 | 479 | |
11fdf7f2 | 480 | will dump all current settings,:: |
c07f9fc5 | 481 | |
11fdf7f2 | 482 | ceph daemon osd.0 config diff |
c07f9fc5 | 483 | |
11fdf7f2 | 484 | will show only non-default settings (as well as where the value came from: a config file, the monitor, an override, etc.), and:: |
c07f9fc5 | 485 | |
11fdf7f2 | 486 | ceph daemon osd.0 config get debug_osd |
7c673cae | 487 | |
11fdf7f2 | 488 | will report the value of a single option. |
7c673cae | 489 | |
7c673cae FG |
490 | |
491 |