[ceph.git] / ceph / doc / mgr / orchestrator_cli.rst


.. _orchestrator-cli-module:

================
Orchestrator CLI
================

This module provides a command line interface (CLI) to orchestrator
modules (ceph-mgr modules which interface with external orchestation services)

As the orchestrator CLI unifies different external orchestrators, a common nomenclature
for the orchestrator module is needed.

+--------------------------------------+---------------------------------------+
| host                                 | hostname (not DNS name) of the        |
|                                      | physical host. Not the podname,       |
|                                      | container name, or hostname inside    |
|                                      | the container.                        |
+--------------------------------------+---------------------------------------+
| service type                         | The type of the service. e.g., nfs,   |
|                                      | mds, osd, mon, rgw, mgr, iscsi        |
+--------------------------------------+---------------------------------------+
| service                              | A logical service, Typically          |
|                                      | comprised of multiple service         |
|                                      | instances on multiple hosts for HA    |
|                                      |                                       |
|                                      | * ``fs_name`` for mds type            |
|                                      | * ``rgw_zone`` for rgw type           |
|                                      | * ``ganesha_cluster_id`` for nfs type |
+--------------------------------------+---------------------------------------+
| service instance                     | A single instance of a service.       |
|                                      |  Usually a daemon, but maybe not      |
|                                      | (e.g., might be a kernel service      |
|                                      | like LIO or knfsd or whatever)        |
|                                      |                                       |
|                                      | This identifier should                |
|                                      | uniquely identify the instance        |
+--------------------------------------+---------------------------------------+
| daemon                               | A running process on a host; use      |
|                                      | “service instance” instead            |
+--------------------------------------+---------------------------------------+

The relation between the names is the following:

* a service belongs to a service type
* a service instance belongs to a service type
* a service instance belongs to a single service group

Configuration
=============

To enable the orchestrator, please select the orchestrator module to use
with the ``set backend`` command::

    ceph orchestrator set backend <module>

For example, to enable the Rook orchestrator module and use it with the CLI::

    ceph mgr module enable rook
    ceph orchestrator set backend rook

You can then check backend is properly configured::

    ceph orchestrator status

Disable the Orchestrator
~~~~~~~~~~~~~~~~~~~~~~~~

To disable the orchestrator again, use the empty string ``""``::

    ceph orchestrator set backend ""``
    ceph mgr module disable rook

Usage
=====

.. warning::

    The orchestrator CLI is unfinished and work in progress. Some commands will not
    exist, or return a different result.

.. note::

    Orchestrator modules may only implement a subset of the commands listed below.
    Also, the implementation of the commands are orchestrator module dependent and will
    differ between implementations.

Status
~~~~~~

::

    ceph orchestrator status

Show current orchestrator mode and high-level status (whether the module able
to talk to it)

Also show any in-progress actions.

Host Management
~~~~~~~~~~~~~~~

List hosts associated with the cluster::

    ceph orchestrator host ls

Add and remove hosts::

    ceph orchestrator host add <host>
    ceph orchestrator host rm <host>

OSD Management
~~~~~~~~~~~~~~

List Devices
^^^^^^^^^^^^

Print a list of discovered devices, grouped by node and optionally
filtered to a particular node:

::

    ceph orchestrator device ls [--host=...] [--refresh]

Create OSDs
^^^^^^^^^^^

Create OSDs on a group of devices on a single host::

    ceph orchestrator osd create <host>:<drive>
    ceph orchestrator osd create -i <path-to-drive-group.json>


The output of ``osd create`` is not specified and may vary between orchestrator backends.

Where ``drive.group.json`` is a JSON file containing the fields defined in :class:`orchestrator.DriveGroupSpec`


Decommission an OSD
^^^^^^^^^^^^^^^^^^^
::

    ceph orchestrator osd rm <osd-id> [osd-id...]

Removes one or more OSDs from the cluster and the host, if the OSDs are marked as
``destroyed``.


..
    Blink Device Lights
    ^^^^^^^^^^^^^^^^^^^
    ::

        ceph orchestrator device ident-on <host> <devname>
        ceph orchestrator device ident-off <host> <devname>
        ceph orchestrator device fault-on <host> <devname>
        ceph orchestrator device fault-off <host> <devname>

        ceph orchestrator osd ident-on {primary,journal,db,wal,all} <osd-id>
        ceph orchestrator osd ident-off {primary,journal,db,wal,all} <osd-id>
        ceph orchestrator osd fault-on {primary,journal,db,wal,all} <osd-id>
        ceph orchestrator osd fault-off {primary,journal,db,wal,all} <osd-id>

    Where ``journal`` is the filestore journal, ``wal`` is the write ahead log of
    bluestore and ``all`` stands for all devices associated with the osd


Monitor and manager management
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Creates or removes MONs or MGRs from the cluster. Orchestrator may return an
error if it doesn't know how to do this transition.

Update the number of monitor nodes::

    ceph orchestrator mon update <num> [host, host:network...]

Each host can optionally specificy a network for the monitor to listen on.

Update the number of manager nodes::

    ceph orchestrator mgr update <num> [host...]

..
    .. note::

        The host lists are the new full list of mon/mgr hosts

    .. note::

        specifying hosts is optional for some orchestrator modules
        and mandatory for others (e.g. Ansible).


Service Status
~~~~~~~~~~~~~~

Print a list of services known to the orchestrator. The list can be limited to
services on a particular host with the optional --host parameter and/or
services of a particular type via optional --type parameter
(mon, osd, mgr, mds, rgw):

::

    ceph orchestrator service ls [--host host] [--svc_type type] [--refresh]

Discover the status of a particular service::

    ceph orchestrator service ls --svc_type type --svc_id <name> [--refresh]


Query the status of a particular service instance (mon, osd, mds, rgw).  For OSDs
the id is the numeric OSD ID, for MDS services it is the filesystem name::

    ceph orchestrator service-instance status <type> <instance-name> [--refresh]


Stateless services (MDS/RGW/NFS/rbd-mirror/iSCSI)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The orchestrator is not responsible for configuring the services. Please look into the corresponding
documentation for details.

The ``name`` parameter is an identifier of the group of instances:

* a CephFS filesystem for a group of MDS daemons,
* a zone name for a group of RGWs

Sizing: the ``size`` parameter gives the number of daemons in the cluster
(e.g. the number of MDS daemons for a particular CephFS filesystem).

Creating/growing/shrinking/removing services::

    ceph orchestrator {mds,rgw} update <name> <size> [host…]
    ceph orchestrator {mds,rgw} add <name>
    ceph orchestrator nfs update <name> <size> [host…]
    ceph orchestrator nfs add <name> <pool> [--namespace=<namespace>]
    ceph orchestrator {mds,rgw,nfs} rm <name>

e.g., ``ceph orchestrator mds update myfs 3 host1 host2 host3``

Start/stop/reload::

    ceph orchestrator service {stop,start,reload} <type> <name>

    ceph orchestrator service-instance {start,stop,reload} <type> <instance-name>


Current Implementation Status
=============================

This is an overview of the current implementation status of the orchestrators.

=================================== ========= ====== ========= =====
 Command                             Ansible   Rook   DeepSea   SSH
=================================== ========= ====== ========= =====
 host add                            ⚪         ⚪       ⚪         ✔️
 host ls                             ⚪         ⚪       ⚪         ✔️
 host rm                             ⚪         ⚪       ⚪         ✔️
 mgr update                          ⚪         ⚪       ⚪         ✔️
 mon update                          ⚪         ✔️       ⚪         ✔️
 osd create                          ✔️         ✔️       ⚪         ✔️
 osd device {ident,fault}-{on,off}   ⚪         ⚪       ⚪         ⚪
 osd rm                              ✔️         ⚪       ⚪         ⚪
 device {ident,fault}-(on,off}       ⚪         ⚪       ⚪         ⚪
 device ls                           ✔️         ✔️       ✔️         ✔️
 service ls                          ⚪         ✔️       ✔️         ⚪
 service-instance status             ⚪         ⚪       ⚪         ⚪
 iscsi {stop,start,reload}           ⚪         ⚪       ⚪         ⚪
 iscsi add                           ⚪         ⚪       ⚪         ⚪
 iscsi rm                            ⚪         ⚪       ⚪         ⚪
 iscsi update                        ⚪         ⚪       ⚪         ⚪
 mds {stop,start,reload}             ⚪         ⚪       ⚪         ⚪
 mds add                             ⚪         ✔️       ⚪         ⚪
 mds rm                              ⚪         ✔️       ⚪         ⚪
 mds update                          ⚪         ⚪       ⚪         ⚪
 nfs {stop,start,reload}             ⚪         ⚪       ⚪         ⚪
 nfs add                             ⚪         ✔️       ⚪         ⚪
 nfs rm                              ⚪         ✔️       ⚪         ⚪
 nfs update                          ⚪         ⚪       ⚪         ⚪
 rbd-mirror {stop,start,reload}      ⚪         ⚪       ⚪         ⚪
 rbd-mirror add                      ⚪         ⚪       ⚪         ⚪
 rbd-mirror rm                       ⚪         ⚪       ⚪         ⚪
 rbd-mirror update                   ⚪         ⚪       ⚪         ⚪
 rgw {stop,start,reload}             ⚪         ⚪       ⚪         ⚪
 rgw add                             ⚪         ✔️       ⚪         ⚪
 rgw rm                              ⚪         ✔️       ⚪         ⚪
 rgw update                          ⚪         ⚪       ⚪         ⚪
=================================== ========= ====== ========= =====

where

* ⚪ = not yet implemented
* ❌ = not applicable
* ✔ = implemented
Commit	Line	Data
11fdf7f2 TL	1
	2	.. _orchestrator-cli-module:
	3
	4	================
	5	Orchestrator CLI
	6	================
	7
	8	This module provides a command line interface (CLI) to orchestrator
	9	modules (ceph-mgr modules which interface with external orchestation services)
	10
	11	As the orchestrator CLI unifies different external orchestrators, a common nomenclature
	12	for the orchestrator module is needed.
	13
	14	+--------------------------------------+---------------------------------------+
	15	\| host \| hostname (not DNS name) of the \|
	16	\| \| physical host. Not the podname, \|
	17	\| \| container name, or hostname inside \|
	18	\| \| the container. \|
	19	+--------------------------------------+---------------------------------------+
	20	\| service type \| The type of the service. e.g., nfs, \|
	21	\| \| mds, osd, mon, rgw, mgr, iscsi \|
	22	+--------------------------------------+---------------------------------------+
	23	\| service \| A logical service, Typically \|
	24	\| \| comprised of multiple service \|
	25	\| \| instances on multiple hosts for HA \|
	26	\| \| \|
	27	\| \| * ``fs_name`` for mds type \|
	28	\| \| * ``rgw_zone`` for rgw type \|
	29	\| \| * ``ganesha_cluster_id`` for nfs type \|
	30	+--------------------------------------+---------------------------------------+
	31	\| service instance \| A single instance of a service. \|
	32	\| \| Usually a daemon, but maybe not \|
	33	\| \| (e.g., might be a kernel service \|
	34	\| \| like LIO or knfsd or whatever) \|
	35	\| \| \|
	36	\| \| This identifier should \|
	37	\| \| uniquely identify the instance \|
	38	+--------------------------------------+---------------------------------------+
	39	\| daemon \| A running process on a host; use \|
	40	\| \| “service instance” instead \|
	41	+--------------------------------------+---------------------------------------+
	42
	43	The relation between the names is the following:
	44
	45	* a service belongs to a service type
	46	* a service instance belongs to a service type
	47	* a service instance belongs to a single service group
	48
	49	Configuration
	50	=============
	51
eafe8130 TL	52	To enable the orchestrator, please select the orchestrator module to use
eafe8130 TL	53	with the ``set backend`` command::
11fdf7f2 TL	54
	55	ceph orchestrator set backend <module>
	56
	57	For example, to enable the Rook orchestrator module and use it with the CLI::
	58
	59	ceph mgr module enable rook
	60	ceph orchestrator set backend rook
	61
	62	You can then check backend is properly configured::
	63
	64	ceph orchestrator status
	65
eafe8130 TL	66	Disable the Orchestrator
	67	~~~~~~~~~~~~~~~~~~~~~~~~
	68
	69	To disable the orchestrator again, use the empty string ``""``::
	70
	71	ceph orchestrator set backend ""``
	72	ceph mgr module disable rook
	73
11fdf7f2 TL	74	Usage
	75	=====
	76
	77	.. warning::
	78
	79	The orchestrator CLI is unfinished and work in progress. Some commands will not
	80	exist, or return a different result.
	81
	82	.. note::
	83
	84	Orchestrator modules may only implement a subset of the commands listed below.
	85	Also, the implementation of the commands are orchestrator module dependent and will
	86	differ between implementations.
	87
	88	Status
	89	~~~~~~
	90
	91	::
	92
	93	ceph orchestrator status
	94
	95	Show current orchestrator mode and high-level status (whether the module able
	96	to talk to it)
	97
	98	Also show any in-progress actions.
	99
	100	Host Management
	101	~~~~~~~~~~~~~~~
	102
	103	List hosts associated with the cluster::
	104
	105	ceph orchestrator host ls
	106
	107	Add and remove hosts::
	108
	109	ceph orchestrator host add <host>
	110	ceph orchestrator host rm <host>
	111
	112	OSD Management
	113	~~~~~~~~~~~~~~
	114
	115	List Devices
	116	^^^^^^^^^^^^
	117
	118	Print a list of discovered devices, grouped by node and optionally
	119	filtered to a particular node:
	120
	121	::
	122
	123	ceph orchestrator device ls [--host=...] [--refresh]
	124
	125	Create OSDs
	126	^^^^^^^^^^^
	127
	128	Create OSDs on a group of devices on a single host::
	129
	130	ceph orchestrator osd create <host>:<drive>
	131	ceph orchestrator osd create -i <path-to-drive-group.json>
	132
	133
	134	The output of ``osd create`` is not specified and may vary between orchestrator backends.
	135
	136	Where ``drive.group.json`` is a JSON file containing the fields defined in :class:`orchestrator.DriveGroupSpec`
	137
138
139	Decommission an OSD
140	^^^^^^^^^^^^^^^^^^^
141	::
142
143	ceph orchestrator osd rm <osd-id> [osd-id...]
144
145	Removes one or more OSDs from the cluster and the host, if the OSDs are marked as
146	``destroyed``.
147
148
149	..
150	Blink Device Lights
151	^^^^^^^^^^^^^^^^^^^
152	::
153
154	ceph orchestrator device ident-on <host> <devname>
155	ceph orchestrator device ident-off <host> <devname>
156	ceph orchestrator device fault-on <host> <devname>
157	ceph orchestrator device fault-off <host> <devname>
158
159	ceph orchestrator osd ident-on {primary,journal,db,wal,all} <osd-id>
160	ceph orchestrator osd ident-off {primary,journal,db,wal,all} <osd-id>
161	ceph orchestrator osd fault-on {primary,journal,db,wal,all} <osd-id>
162	ceph orchestrator osd fault-off {primary,journal,db,wal,all} <osd-id>
163
164	Where ``journal`` is the filestore journal, ``wal`` is the write ahead log of
165	bluestore and ``all`` stands for all devices associated with the osd
166
167
168	Monitor and manager management
169	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
170
171	Creates or removes MONs or MGRs from the cluster. Orchestrator may return an
172	error if it doesn't know how to do this transition.
173
174	Update the number of monitor nodes::
175
176	ceph orchestrator mon update <num> [host, host:network...]
177
178	Each host can optionally specificy a network for the monitor to listen on.
179
180	Update the number of manager nodes::
181
182	ceph orchestrator mgr update <num> [host...]
183
184	..
185	.. note::
186
187	The host lists are the new full list of mon/mgr hosts
188
189	.. note::
190
191	specifying hosts is optional for some orchestrator modules
192	and mandatory for others (e.g. Ansible).
193
194
195	Service Status
196	~~~~~~~~~~~~~~
197
198	Print a list of services known to the orchestrator. The list can be limited to
199	services on a particular host with the optional --host parameter and/or
200	services of a particular type via optional --type parameter
201	(mon, osd, mgr, mds, rgw):
202
203	::
204
eafe8130	205	ceph orchestrator service ls [--host host] [--svc_type type] [--refresh]
11fdf7f2 TL	206
	207	Discover the status of a particular service::
	208
eafe8130	209	ceph orchestrator service ls --svc_type type --svc_id <name> [--refresh]
11fdf7f2 TL	210
	211
	212	Query the status of a particular service instance (mon, osd, mds, rgw). For OSDs
	213	the id is the numeric OSD ID, for MDS services it is the filesystem name::
	214
	215	ceph orchestrator service-instance status <type> <instance-name> [--refresh]
	216
	217
	218
	219	Stateless services (MDS/RGW/NFS/rbd-mirror/iSCSI)
	220	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
	221	The orchestrator is not responsible for configuring the services. Please look into the corresponding
	222	documentation for details.
	223
	224	The ``name`` parameter is an identifier of the group of instances:
	225
	226	* a CephFS filesystem for a group of MDS daemons,
	227	* a zone name for a group of RGWs
	228
	229	Sizing: the ``size`` parameter gives the number of daemons in the cluster
	230	(e.g. the number of MDS daemons for a particular CephFS filesystem).
	231
	232	Creating/growing/shrinking/removing services::
	233
	234	ceph orchestrator {mds,rgw} update <name> <size> [host…]
	235	ceph orchestrator {mds,rgw} add <name>
	236	ceph orchestrator nfs update <name> <size> [host…]
	237	ceph orchestrator nfs add <name> <pool> [--namespace=<namespace>]
	238	ceph orchestrator {mds,rgw,nfs} rm <name>
	239
	240	e.g., ``ceph orchestrator mds update myfs 3 host1 host2 host3``
	241
	242	Start/stop/reload::
	243
	244	ceph orchestrator service {stop,start,reload} <type> <name>
	245
	246	ceph orchestrator service-instance {start,stop,reload} <type> <instance-name>
	247
	248
	249	Current Implementation Status
	250	=============================
	251
	252	This is an overview of the current implementation status of the orchestrators.
	253
	254	=================================== ========= ====== ========= =====
	255	Command Ansible Rook DeepSea SSH
	256	=================================== ========= ====== ========= =====
	257	host add ⚪ ⚪ ⚪ ✔️
	258	host ls ⚪ ⚪ ⚪ ✔️
	259	host rm ⚪ ⚪ ⚪ ✔️
	260	mgr update ⚪ ⚪ ⚪ ✔️
81eedcae	261	mon update ⚪ ✔️ ⚪ ✔️
11fdf7f2 TL	262	osd create ✔️ ✔️ ⚪ ✔️
	263	osd device {ident,fault}-{on,off} ⚪ ⚪ ⚪ ⚪
	264	osd rm ✔️ ⚪ ⚪ ⚪
	265	device {ident,fault}-(on,off} ⚪ ⚪ ⚪ ⚪
	266	device ls ✔️ ✔️ ✔️ ✔️
	267	service ls ⚪ ✔️ ✔️ ⚪
11fdf7f2 TL	268	service-instance status ⚪ ⚪ ⚪ ⚪
	269	iscsi {stop,start,reload} ⚪ ⚪ ⚪ ⚪
	270	iscsi add ⚪ ⚪ ⚪ ⚪
	271	iscsi rm ⚪ ⚪ ⚪ ⚪
	272	iscsi update ⚪ ⚪ ⚪ ⚪
	273	mds {stop,start,reload} ⚪ ⚪ ⚪ ⚪
	274	mds add ⚪ ✔️ ⚪ ⚪
	275	mds rm ⚪ ✔️ ⚪ ⚪
	276	mds update ⚪ ⚪ ⚪ ⚪
	277	nfs {stop,start,reload} ⚪ ⚪ ⚪ ⚪
	278	nfs add ⚪ ✔️ ⚪ ⚪
	279	nfs rm ⚪ ✔️ ⚪ ⚪
	280	nfs update ⚪ ⚪ ⚪ ⚪
	281	rbd-mirror {stop,start,reload} ⚪ ⚪ ⚪ ⚪
	282	rbd-mirror add ⚪ ⚪ ⚪ ⚪
	283	rbd-mirror rm ⚪ ⚪ ⚪ ⚪
	284	rbd-mirror update ⚪ ⚪ ⚪ ⚪
	285	rgw {stop,start,reload} ⚪ ⚪ ⚪ ⚪
	286	rgw add ⚪ ✔️ ⚪ ⚪
	287	rgw rm ⚪ ✔️ ⚪ ⚪
	288	rgw update ⚪ ⚪ ⚪ ⚪
	289	=================================== ========= ====== ========= =====
	290
	291	where
	292
	293	* ⚪ = not yet implemented
	294	* ❌ = not applicable
	295	* ✔ = implemented