1 .. index:: control, commands
11 To issue monitor commands, use the ``ceph`` utility:
15 ceph [-m monhost] {command}
17 In most cases, monitor commands have the following form:
21 ceph {subsystem} {command}
27 To display the current cluster status, run the following commands:
34 To display a running summary of cluster status and major events, run the
41 To display the monitor quorum, including which monitors are participating and
42 which one is the leader, run the following commands:
49 To query the status of a single monitor, including whether it is in the quorum,
50 run the following command:
54 ceph tell mon.[id] mon_status
56 Here the value of ``[id]`` can be found by consulting the output of ``ceph
60 Authentication Subsystem
61 ========================
63 To add an OSD keyring for a specific OSD, run the following command:
67 ceph auth add {osd} {--in-file|-i} {path-to-osd-keyring}
69 To list the cluster's keys and their capabilities, run the following command:
76 Placement Group Subsystem
77 =========================
79 To display the statistics for all placement groups (PGs), run the following
84 ceph pg dump [--format {format}]
86 Here the valid formats are ``plain`` (default), ``json`` ``json-pretty``,
87 ``xml``, and ``xml-pretty``. When implementing monitoring tools and other
88 tools, it is best to use the ``json`` format. JSON parsing is more
89 deterministic than the ``plain`` format (which is more human readable), and the
90 layout is much more consistent from release to release. The ``jq`` utility is
91 very useful for extracting data from JSON output.
93 To display the statistics for all PGs stuck in a specified state, run the
98 ceph pg dump_stuck inactive|unclean|stale|undersized|degraded [--format {format}] [-t|--threshold {seconds}]
100 Here ``--format`` may be ``plain`` (default), ``json``, ``json-pretty``,
101 ``xml``, or ``xml-pretty``.
103 The ``--threshold`` argument determines the time interval (in seconds) for a PG
104 to be considered ``stuck`` (default: 300).
106 PGs might be stuck in any of the following states:
110 PGs are unable to process reads or writes because they are waiting for an
111 OSD that has the most up-to-date data to return to an ``up`` state.
116 PGs contain objects that have not been replicated the desired number of
117 times. These PGs have not yet completed the process of recovering.
122 PGs are in an unknown state, because the OSDs that host them have not
123 reported to the monitor cluster for a certain period of time (specified by
124 the ``mon_osd_report_timeout`` configuration setting).
127 To delete a ``lost`` object or revert an object to its prior state, either by
128 reverting it to its previous version or by deleting it because it was just
129 created and has no previous version, run the following command:
133 ceph pg {pgid} mark_unfound_lost revert|delete
141 To query OSD subsystem status, run the following command:
147 To write a copy of the most recent OSD map to a file (see :ref:`osdmaptool
148 <osdmaptool>`), run the following command:
152 ceph osd getmap -o file
154 To write a copy of the CRUSH map from the most recent OSD map to a file, run
155 the following command:
159 ceph osd getcrushmap -o file
161 Note that this command is functionally equivalent to the following two
166 ceph osd getmap -o /tmp/osdmap
167 osdmaptool /tmp/osdmap --export-crush file
169 To dump the OSD map, run the following command:
173 ceph osd dump [--format {format}]
175 The ``--format`` option accepts the following arguments: ``plain`` (default),
176 ``json``, ``json-pretty``, ``xml``, and ``xml-pretty``. As noted above, JSON is
177 the recommended format for tools, scripting, and other forms of automation.
179 To dump the OSD map as a tree that lists one OSD per line and displays
180 information about the weights and states of the OSDs, run the following
185 ceph osd tree [--format {format}]
187 To find out where a specific RADOS object is stored in the system, run a
188 command of the following form:
192 ceph osd map <pool-name> <object-name>
194 To add or move a new OSD (specified by its ID, name, or weight) to a specific
195 CRUSH location, run the following command:
199 ceph osd crush set {id} {weight} [{loc1} [{loc2} ...]]
201 To remove an existing OSD from the CRUSH map, run the following command:
205 ceph osd crush remove {name}
207 To remove an existing bucket from the CRUSH map, run the following command:
211 ceph osd crush remove {bucket-name}
213 To move an existing bucket from one position in the CRUSH hierarchy to another,
214 run the following command:
218 ceph osd crush move {id} {loc1} [{loc2} ...]
220 To set the CRUSH weight of a specific OSD (specified by ``{name}``) to
221 ``{weight}``, run the following command:
225 ceph osd crush reweight {name} {weight}
227 To mark an OSD as ``lost``, run the following command:
231 ceph osd lost {id} [--yes-i-really-mean-it]
234 This could result in permanent data loss. Use with caution!
236 To create a new OSD, run the following command:
240 ceph osd create [{uuid}]
242 If no UUID is given as part of this command, the UUID will be set automatically
243 when the OSD starts up.
245 To remove one or more specific OSDs, run the following command:
249 ceph osd rm [{id}...]
251 To display the current ``max_osd`` parameter in the OSD map, run the following
258 To import a specific CRUSH map, run the following command:
262 ceph osd setcrushmap -i file
264 To set the ``max_osd`` parameter in the OSD map, run the following command:
270 The parameter has a default value of 10000. Most operators will never need to
273 To mark a specific OSD ``down``, run the following command:
277 ceph osd down {osd-num}
279 To mark a specific OSD ``out`` (so that no data will be allocated to it), run
280 the following command:
284 ceph osd out {osd-num}
286 To mark a specific OSD ``in`` (so that data will be allocated to it), run the
291 ceph osd in {osd-num}
293 By using the "pause flags" in the OSD map, you can pause or unpause I/O
294 requests. If the flags are set, then no I/O requests will be sent to any OSD.
295 When the flags are cleared, then pending I/O requests will be resent. To set or
296 clear pause flags, run one of the following commands:
303 You can assign an override or ``reweight`` weight value to a specific OSD if
304 the normal CRUSH distribution seems to be suboptimal. The weight of an OSD
305 helps determine the extent of its I/O requests and data storage: two OSDs with
306 the same weight will receive approximately the same number of I/O requests and
307 store approximately the same amount of data. The ``ceph osd reweight`` command
308 assigns an override weight to an OSD. The weight value is in the range 0 to 1,
309 and the command forces CRUSH to relocate a certain amount (1 - ``weight``) of
310 the data that would otherwise be on this OSD. The command does not change the
311 weights of the buckets above the OSD in the CRUSH map. Using the command is
312 merely a corrective measure: for example, if one of your OSDs is at 90% and the
313 others are at 50%, you could reduce the outlier weight to correct this
314 imbalance. To assign an override weight to a specific OSD, run the following
319 ceph osd reweight {osd-num} {weight}
321 .. note:: Any assigned override reweight value will conflict with the balancer.
322 This means that if the balancer is in use, all override reweight values
323 should be ``1.0000`` in order to avoid suboptimal cluster behavior.
325 A cluster's OSDs can be reweighted in order to maintain balance if some OSDs
326 are being disproportionately utilized. Note that override or ``reweight``
327 weights have values relative to one another that default to 1.00000; their
328 values are not absolute, and these weights must be distinguished from CRUSH
329 weights (which reflect the absolute capacity of a bucket, as measured in TiB).
330 To reweight OSDs by utilization, run the following command:
334 ceph osd reweight-by-utilization [threshold [max_change [max_osds]]] [--no-increasing]
336 By default, this command adjusts the override weight of OSDs that have ±20% of
337 the average utilization, but you can specify a different percentage in the
338 ``threshold`` argument.
340 To limit the increment by which any OSD's reweight is to be changed, use the
341 ``max_change`` argument (default: 0.05). To limit the number of OSDs that are
342 to be adjusted, use the ``max_osds`` argument (default: 4). Increasing these
343 variables can accelerate the reweighting process, but perhaps at the cost of
344 slower client operations (as a result of the increase in data movement).
346 You can test the ``osd reweight-by-utilization`` command before running it. To
347 find out which and how many PGs and OSDs will be affected by a specific use of
348 the ``osd reweight-by-utilization`` command, run the following command:
352 ceph osd test-reweight-by-utilization [threshold [max_change max_osds]] [--no-increasing]
354 The ``--no-increasing`` option can be added to the ``reweight-by-utilization``
355 and ``test-reweight-by-utilization`` commands in order to prevent any override
356 weights that are currently less than 1.00000 from being increased. This option
357 can be useful in certain circumstances: for example, when you are hastily
358 balancing in order to remedy ``full`` or ``nearfull`` OSDs, or when there are
359 OSDs being evacuated or slowly brought into service.
361 Operators of deployments that utilize Nautilus or newer (or later revisions of
362 Luminous and Mimic) and that have no pre-Luminous clients might likely instead
363 want to enable the `balancer`` module for ``ceph-mgr``.
365 The blocklist can be modified by adding or removing an IP address or a CIDR
366 range. If an address is blocklisted, it will be unable to connect to any OSD.
367 If an OSD is contained within an IP address or CIDR range that has been
368 blocklisted, the OSD will be unable to perform operations on its peers when it
369 acts as a client: such blocked operations include tiering and copy-from
370 functionality. To add or remove an IP address or CIDR range to the blocklist,
371 run one of the following commands:
375 ceph osd blocklist ["range"] add ADDRESS[:source_port][/netmask_bits] [TIME]
376 ceph osd blocklist ["range"] rm ADDRESS[:source_port][/netmask_bits]
378 If you add something to the blocklist with the above ``add`` command, you can
379 use the ``TIME`` keyword to specify the length of time (in seconds) that it
380 will remain on the blocklist (default: one hour). To add or remove a CIDR
381 range, use the ``range`` keyword in the above commands.
383 Note that these commands are useful primarily in failure testing. Under normal
384 conditions, blocklists are maintained automatically and do not need any manual
387 To create or delete a snapshot of a specific storage pool, run one of the
392 ceph osd pool mksnap {pool-name} {snap-name}
393 ceph osd pool rmsnap {pool-name} {snap-name}
395 To create, delete, or rename a specific storage pool, run one of the following
400 ceph osd pool create {pool-name} [pg_num [pgp_num]]
401 ceph osd pool delete {pool-name} [{pool-name} --yes-i-really-really-mean-it]
402 ceph osd pool rename {old-name} {new-name}
404 To change a pool setting, run the following command:
408 ceph osd pool set {pool-name} {field} {value}
410 The following are valid fields:
412 * ``size``: The number of copies of data in the pool.
413 * ``pg_num``: The PG number.
414 * ``pgp_num``: The effective number of PGs when calculating placement.
415 * ``crush_rule``: The rule number for mapping placement.
417 To retrieve the value of a pool setting, run the following command:
421 ceph osd pool get {pool-name} {field}
425 * ``pg_num``: The PG number.
426 * ``pgp_num``: The effective number of PGs when calculating placement.
428 To send a scrub command to a specific OSD, or to all OSDs (by using ``*``), run
429 the following command:
433 ceph osd scrub {osd-num}
435 To send a repair command to a specific OSD, or to all OSDs (by using ``*``),
436 run the following command:
442 You can run a simple throughput benchmark test against a specific OSD. This
443 test writes a total size of ``TOTAL_DATA_BYTES`` (default: 1 GB) incrementally,
444 in multiple write requests that each have a size of ``BYTES_PER_WRITE``
445 (default: 4 MB). The test is not destructive and it will not overwrite existing
446 live OSD data, but it might temporarily affect the performance of clients that
447 are concurrently accessing the OSD. To launch this benchmark test, run the
452 ceph tell osd.N bench [TOTAL_DATA_BYTES] [BYTES_PER_WRITE]
454 To clear the caches of a specific OSD during the interval between one benchmark
455 run and another, run the following command:
459 ceph tell osd.N cache drop
461 To retrieve the cache statistics of a specific OSD, run the following command:
465 ceph tell osd.N cache status
470 To change the configuration parameters of a running metadata server, run the
475 ceph tell mds.{mds-id} config set {setting} {value}
481 ceph tell mds.0 config set debug_ms 1
483 To enable debug messages, run the following command:
489 To display the status of all metadata servers, run the following command:
495 To mark the active metadata server as failed (and to trigger failover to a
496 standby if a standby is present), run the following command:
498 .. todo:: ``ceph mds`` subcommands missing docs: set, dump, getmap, stop, setmap
504 To display monitor statistics, run the following command:
510 This command returns output similar to the following:
514 e2: 3 mons at {a=127.0.0.1:40000/0,b=127.0.0.1:40001/0,c=127.0.0.1:40002/0}, election epoch 6, quorum 0,1,2 a,b,c
516 There is a ``quorum`` list at the end of the output. It lists those monitor
517 nodes that are part of the current quorum.
519 To retrieve this information in a more direct way, run the following command:
523 ceph quorum_status -f json-pretty
525 This command returns output similar to the following:
527 .. code-block:: javascript
541 "quorum_leader_name": "a",
544 "fsid": "ba807e74-b64f-4b72-b43f-597dfe60ddbc",
545 "modified": "2016-12-26 14:42:09.288066",
546 "created": "2016-12-26 14:42:03.573585",
557 "addr": "127.0.0.1:40000\/0",
558 "public_addr": "127.0.0.1:40000\/0"
563 "addr": "127.0.0.1:40001\/0",
564 "public_addr": "127.0.0.1:40001\/0"
569 "addr": "127.0.0.1:40002\/0",
570 "public_addr": "127.0.0.1:40002\/0"
577 The above will block until a quorum is reached.
579 To see the status of a specific monitor, run the following command:
583 ceph tell mon.[name] mon_status
585 Here the value of ``[name]`` can be found by consulting the output of the
586 ``ceph quorum_status`` command. This command returns output similar to the
602 "required_con": "9025616074522624",
606 "quorum_con": "1152921504336314367",
611 "outside_quorum": [],
612 "extra_probe_peers": [],
616 "fsid": "ba807e74-b64f-4b72-b43f-597dfe60ddbc",
617 "modified": "2016-12-26 14:42:09.288066",
618 "created": "2016-12-26 14:42:03.573585",
629 "addr": "127.0.0.1:40000\/0",
630 "public_addr": "127.0.0.1:40000\/0"
635 "addr": "127.0.0.1:40001\/0",
636 "public_addr": "127.0.0.1:40001\/0"
641 "addr": "127.0.0.1:40002\/0",
642 "public_addr": "127.0.0.1:40002\/0"
648 To see a dump of the monitor state, run the following command:
654 This command returns output similar to the following:
658 dumped monmap epoch 2
660 fsid ba807e74-b64f-4b72-b43f-597dfe60ddbc
661 last_changed 2016-12-26 14:42:09.288066
662 created 2016-12-26 14:42:03.573585
663 0: 127.0.0.1:40000/0 mon.a
664 1: 127.0.0.1:40001/0 mon.b
665 2: 127.0.0.1:40002/0 mon.c