2 .. _orchestrator-cli-module:
8 This module provides a command line interface (CLI) to orchestrator
9 modules (ceph-mgr modules which interface with external orchestration services).
11 As the orchestrator CLI unifies different external orchestrators, a common nomenclature
12 for the orchestrator module is needed.
14 +--------------------------------------+---------------------------------------+
15 | *host* | hostname (not DNS name) of the |
16 | | physical host. Not the podname, |
17 | | container name, or hostname inside |
19 +--------------------------------------+---------------------------------------+
20 | *service type* | The type of the service. e.g., nfs, |
21 | | mds, osd, mon, rgw, mgr, iscsi |
22 +--------------------------------------+---------------------------------------+
23 | *service* | A logical service, Typically |
24 | | comprised of multiple service |
25 | | instances on multiple hosts for HA |
27 | | * ``fs_name`` for mds type |
28 | | * ``rgw_zone`` for rgw type |
29 | | * ``ganesha_cluster_id`` for nfs type |
30 +--------------------------------------+---------------------------------------+
31 | *daemon* | A single instance of a service. |
32 | | Usually a daemon, but maybe not |
33 | | (e.g., might be a kernel service |
34 | | like LIO or knfsd or whatever) |
36 | | This identifier should |
37 | | uniquely identify the instance |
38 +--------------------------------------+---------------------------------------+
40 The relation between the names is the following:
42 * A *service* has a specfic *service type*
43 * A *daemon* is a physical instance of a *service type*
48 Orchestrator modules may only implement a subset of the commands listed below.
49 Also, the implementation of the commands are orchestrator module dependent and will
50 differ between implementations.
59 Show current orchestrator mode and high-level status (whether the module able
62 Also show any in-progress actions.
67 List hosts associated with the cluster::
71 Add and remove hosts::
73 ceph orch host add <hostname> [<addr>] [<labels>...]
74 ceph orch host rm <hostname>
79 Many hosts can be added at once using
80 ``ceph orch apply -i`` by submitting a multi-document YAML file::
100 This can be combined with service specifications (below) to create a cluster spec file to deploy a whole cluster in one command. see ``cephadm bootstrap --apply-spec`` also to do this during bootstrap. Cluster SSH Keys must be copied to hosts prior.
108 Print a list of discovered devices, grouped by host and optionally
109 filtered to a particular host:
113 ceph orch device ls [--host=...] [--refresh]
117 HOST PATH TYPE SIZE DEVICE AVAIL REJECT REASONS
118 master /dev/vda hdd 42.0G False locked
119 node1 /dev/vda hdd 42.0G False locked
120 node1 /dev/vdb hdd 8192M 387836 False locked, LVM detected, Insufficient space (<5GB) on vgs
121 node1 /dev/vdc hdd 8192M 450575 False locked, LVM detected, Insufficient space (<5GB) on vgs
122 node3 /dev/vda hdd 42.0G False locked
123 node3 /dev/vdb hdd 8192M 395145 False LVM detected, locked, Insufficient space (<5GB) on vgs
124 node3 /dev/vdc hdd 8192M 165562 False LVM detected, locked, Insufficient space (<5GB) on vgs
125 node2 /dev/vda hdd 42.0G False locked
126 node2 /dev/vdb hdd 8192M 672147 False LVM detected, Insufficient space (<5GB) on vgs, locked
127 node2 /dev/vdc hdd 8192M 228094 False LVM detected, Insufficient space (<5GB) on vgs, locked
132 Erase Devices (Zap Devices)
133 ---------------------------
135 Erase (zap) a device so that it can be resued. ``zap`` calls ``ceph-volume zap`` on the remote host.
139 orch device zap <hostname> <path>
143 ceph orch device zap my_hostname /dev/sdx
149 Create OSDs on a group of devices on a single host::
151 ceph orch daemon add osd <host>:device1,device2
155 ceph orch apply osd -i <json_file/yaml_file>
160 ceph orch apply osd --all-available-devices
163 For a more in-depth guide to DriveGroups please refer to :ref:`drivegroups`
167 # ceph orch daemon add osd node1:/dev/vdd
168 Created osd(s) 6 on host 'node1'
171 If the 'apply' method is used. You will be presented with a preview of what will happen.
175 # ceph orch apply osd --all-available-devices
176 NAME HOST DATA DB WAL
177 all-available-devices node1 /dev/vdb - -
178 all-available-devices node2 /dev/vdc - -
179 all-available-devices node3 /dev/vdd - -
183 Output form Cephadm orchestrator
189 ceph orch osd rm <svc_id>... [--replace] [--force]
191 Removes one or more OSDs from the cluster.
196 Scheduled OSD(s) for removal
199 OSDs that are not safe-to-destroy will be rejected.
201 You can query the state of the operation with::
203 # ceph orch osd rm status
204 NAME HOST PGS STARTED_AT
205 osd.7 node1 55 2020-04-22 19:28:38.785761
206 osd.5 node3 3 2020-04-22 19:28:34.201685
207 osd.3 node2 0 2020-04-22 19:28:34.201695
210 When no PGs are left on the osd, it will be decommissioned and removed from the cluster.
217 orch osd rm <svc_id>... --replace [--force]
221 # ceph orch osd rm 4 --replace
222 Scheduled OSD(s) for replacement
225 This follows the same procedure as the "Remove OSD" part with the exception that the OSD is not permanently removed
226 from the crush hierarchy, but is assigned a 'destroyed' flag.
228 **Preserving the OSD ID**
230 The previously set the 'destroyed' flag is used to determined osd ids that will be reused in the next osd deployment.
232 If you use OSDSpecs for osd deployment, your newly added disks will be assigned with the osd ids of their replaced
233 counterpart, granted the new disk still match the OSDSpecs.
235 For assistance in this process you can use the 'preview' feature:
240 ceph orch apply osd --service-name <name_of_osd_spec> --preview
241 NAME HOST DATA DB WAL
242 <name_of_osd_spec> node1 /dev/vdb - -
244 Tip: The name of your OSDSpec can be retrieved from **ceph orch ls**
246 Alternatively, you can use your OSDSpec file::
248 ceph orch apply osd -i <osd_spec_file> --preview
249 NAME HOST DATA DB WAL
250 <name_of_osd_spec> node1 /dev/vdb - -
253 If this matches your anticipated behavior, just omit the --preview flag to execute the deployment.
261 ceph orch device ident-on <dev_id>
262 ceph orch device ident-on <dev_name> <host>
263 ceph orch device fault-on <dev_id>
264 ceph orch device fault-on <dev_name> <host>
266 ceph orch device ident-off <dev_id> [--force=true]
267 ceph orch device ident-off <dev_id> <host> [--force=true]
268 ceph orch device fault-off <dev_id> [--force=true]
269 ceph orch device fault-off <dev_id> <host> [--force=true]
271 where ``dev_id`` is the device id as listed in ``osd metadata``,
272 ``dev_name`` is the name of the device on the system and ``host`` is the host as
273 returned by ``orchestrator host ls``
275 ceph orch osd ident-on {primary,journal,db,wal,all} <osd-id>
276 ceph orch osd ident-off {primary,journal,db,wal,all} <osd-id>
277 ceph orch osd fault-on {primary,journal,db,wal,all} <osd-id>
278 ceph orch osd fault-off {primary,journal,db,wal,all} <osd-id>
280 Where ``journal`` is the filestore journal, ``wal`` is the write ahead log of
281 bluestore and ``all`` stands for all devices associated with the osd
284 Monitor and manager management
285 ==============================
287 Creates or removes MONs or MGRs from the cluster. Orchestrator may return an
288 error if it doesn't know how to do this transition.
290 Update the number of monitor hosts::
292 ceph orch apply mon <num> [host, host:network...]
294 Each host can optionally specify a network for the monitor to listen on.
296 Update the number of manager hosts::
298 ceph orch apply mgr <num> [host...]
303 The host lists are the new full list of mon/mgr hosts
307 specifying hosts is optional for some orchestrator modules
308 and mandatory for others (e.g. Ansible).
314 Print a list of services known to the orchestrator. The list can be limited to
315 services on a particular host with the optional --host parameter and/or
316 services of a particular type via optional --type parameter
317 (mon, osd, mgr, mds, rgw):
321 ceph orch ls [--service_type type] [--service_name name] [--export] [--format f] [--refresh]
323 Discover the status of a particular service or daemons::
325 ceph orch ls --service_type type --service_name <name> [--refresh]
327 Export the service specs known to the orchestrator as yaml in format
328 that is compatible to ``ceph orch apply -i``::
330 ceph orch ls --export
336 Print a list of all daemons known to the orchestrator::
338 ceph orch ps [--hostname host] [--daemon_type type] [--service_name name] [--daemon_id id] [--format f] [--refresh]
340 Query the status of a particular service instance (mon, osd, mds, rgw). For OSDs
341 the id is the numeric OSD ID, for MDS services it is the file system name::
343 ceph orch ps --daemon_type osd --daemon_id 0
346 .. _orchestrator-cli-cephfs:
351 In order to set up a :term:`CephFS`, execute::
353 ceph fs volume create <fs_name> <placement spec>
355 Where ``name`` is the name of the CephFS, ``placement`` is a
356 :ref:`orchestrator-cli-placement-spec`.
358 This command will create the required Ceph pools, create the new
359 CephFS, and deploy mds servers.
361 Stateless services (MDS/RGW/NFS/rbd-mirror/iSCSI)
362 =================================================
364 The orchestrator is not responsible for configuring the services. Please look into the corresponding
365 documentation for details.
367 The ``name`` parameter is an identifier of the group of instances:
369 * a CephFS file system for a group of MDS daemons,
370 * a zone name for a group of RGWs
372 Sizing: the ``size`` parameter gives the number of daemons in the cluster
373 (e.g. the number of MDS daemons for a particular CephFS file system).
375 Creating/growing/shrinking/removing services::
377 ceph orch {mds,rgw} update <name> <size> [host…]
378 ceph orch {mds,rgw} add <name>
379 ceph orch nfs update <name> <size> [host…]
380 ceph orch nfs add <name> <pool> [--namespace=<namespace>]
381 ceph orch {mds,rgw,nfs} rm <name>
383 e.g., ``ceph orch mds update myfs 3 host1 host2 host3``
387 ceph orch service {stop,start,reload} <type> <name>
389 ceph orch daemon {start,stop,reload} <type> <daemon-id>
391 .. _orchestrator-cli-service-spec:
393 Service Specification
394 =====================
396 As *Service Specification* is a data structure often represented as YAML
397 to specify the deployment of services. For example:
402 service_id: realm.zone
411 Where the properties of a service specification are the following:
413 * ``service_type`` is the type of the service. Needs to be either a Ceph
414 service (``mon``, ``crash``, ``mds``, ``mgr``, ``osd`` or
415 ``rbd-mirror``), a gateway (``nfs`` or ``rgw``), or part of the
416 monitoring stack (``alertmanager``, ``grafana``, ``node-exporter`` or
418 * ``service_id`` is the name of the service. Omit the service time
419 * ``placement`` is a :ref:`orchestrator-cli-placement-spec`
420 * ``spec``: additional specifications for a specific service.
421 * ``unmanaged``: If set to ``true``, the orchestrator will not deploy nor
422 remove any daemon associated with this service. Placement and all other
423 properties will be ignored. This is useful, if this service should not
424 be managed temporarily.
426 Each service type can have different requirements for the spec.
428 Service specifications of type ``mon``, ``mgr``, and the monitoring
429 types do not require a ``service_id``
431 A service of type ``nfs`` requires a pool name and contain
432 an optional namespace:
444 namespace: mynamespace
446 Where ``pool`` is a RADOS pool where NFS client recovery data is stored
447 and ``namespace`` is a RADOS namespace where NFS client recovery
448 data is stored in the pool.
450 A service of type ``osd`` is in detail described in :ref:`drivegroups`
452 Many service specifications can then be applied at once using
453 ``ceph orch apply -i`` by submitting a multi-document YAML file::
455 cat <<EOF | ceph orch apply -i -
471 .. _orchestrator-cli-placement-spec:
473 Placement Specification
474 =======================
476 In order to allow the orchestrator to deploy a *service*, it needs to
477 know how many and where it should deploy *daemons*. The orchestrator
478 defines a placement specification that can either be passed as a command line argument.
483 Daemons can be explictly placed on hosts by simply specifying them::
485 orch apply prometheus "host1 host2 host3"
491 service_type: prometheus
498 MONs and other services may require some enhanced network specifications::
500 orch daemon add mon myhost:[v2:1.2.3.4:3000,v1:1.2.3.4:6789]=name
502 Where ``[v2:1.2.3.4:3000,v1:1.2.3.4:6789]`` is the network address of the monitor
503 and ``=name`` specifies the name of the new monitor.
508 Daemons can be explictly placed on hosts that match a specifc label::
510 orch apply prometheus label:mylabel
516 service_type: prometheus
521 Placement by pattern matching
522 -----------------------------
524 Daemons can be placed on hosts as well::
526 orch apply prometheus 'myhost[1-3]'
532 service_type: prometheus
534 host_pattern: "myhost[1-3]"
536 To place a service on *all* hosts, use ``"*"``::
544 service_type: node-exporter
552 By specifying ``count``, only that number of daemons will be created::
554 orch apply prometheus 3
556 To deploy *daemons* on a subset of hosts, also specify the count::
558 orch apply prometheus "2 host1 host2 host3"
560 If the count is bigger than the amount of hosts, cephadm still deploys two daemons::
562 orch apply prometheus "3 host1 host2"
568 service_type: prometheus
576 service_type: prometheus
584 Configuring the Orchestrator CLI
585 ================================
587 To enable the orchestrator, select the orchestrator module to use
588 with the ``set backend`` command::
590 ceph orch set backend <module>
592 For example, to enable the Rook orchestrator module and use it with the CLI::
594 ceph mgr module enable rook
595 ceph orch set backend rook
597 Check the backend is properly configured::
601 Disable the Orchestrator
602 ------------------------
604 To disable the orchestrator, use the empty string ``""``::
606 ceph orch set backend ""
607 ceph mgr module disable rook
609 Current Implementation Status
610 =============================
612 This is an overview of the current implementation status of the orchestrators.
614 =================================== ====== =========
616 =================================== ====== =========
629 daemon {stop,start,...} ⚪ ✔
630 device {ident,fault}-(on,off} ⚪ ✔
638 =================================== ====== =========
642 * ⚪ = not yet implemented