]>
Commit | Line | Data |
---|---|---|
11fdf7f2 TL |
1 | |
2 | ||
3 | .. _orchestrator-modules: | |
4 | ||
5 | .. py:currentmodule:: orchestrator | |
6 | ||
7 | ceph-mgr orchestrator modules | |
8 | ============================= | |
9 | ||
10 | .. warning:: | |
11 | ||
12 | This is developer documentation, describing Ceph internals that | |
13 | are only relevant to people writing ceph-mgr orchestrator modules. | |
14 | ||
15 | In this context, *orchestrator* refers to some external service that | |
16 | provides the ability to discover devices and create Ceph services. This | |
9f95a23c | 17 | includes external projects such as Rook. |
11fdf7f2 TL |
18 | |
19 | An *orchestrator module* is a ceph-mgr module (:ref:`mgr-module-dev`) | |
20 | which implements common management operations using a particular | |
21 | orchestrator. | |
22 | ||
23 | Orchestrator modules subclass the ``Orchestrator`` class: this class is | |
24 | an interface, it only provides method definitions to be implemented | |
25 | by subclasses. The purpose of defining this common interface | |
26 | for different orchestrators is to enable common UI code, such as | |
27 | the dashboard, to work with various different backends. | |
28 | ||
29 | ||
30 | .. graphviz:: | |
31 | ||
32 | digraph G { | |
33 | subgraph cluster_1 { | |
34 | volumes [label="mgr/volumes"] | |
35 | rook [label="mgr/rook"] | |
36 | dashboard [label="mgr/dashboard"] | |
9f95a23c | 37 | orchestrator_cli [label="mgr/orchestrator"] |
11fdf7f2 | 38 | orchestrator [label="Orchestrator Interface"] |
9f95a23c | 39 | cephadm [label="mgr/cephadm"] |
11fdf7f2 TL |
40 | |
41 | label = "ceph-mgr"; | |
42 | } | |
43 | ||
44 | volumes -> orchestrator | |
45 | dashboard -> orchestrator | |
46 | orchestrator_cli -> orchestrator | |
47 | orchestrator -> rook -> rook_io | |
9f95a23c | 48 | orchestrator -> cephadm |
11fdf7f2 TL |
49 | |
50 | ||
51 | rook_io [label="Rook"] | |
11fdf7f2 TL |
52 | |
53 | rankdir="TB"; | |
54 | } | |
55 | ||
56 | Behind all the abstraction, the purpose of orchestrator modules is simple: | |
57 | enable Ceph to do things like discover available hardware, create and | |
58 | destroy OSDs, and run MDS and RGW services. | |
59 | ||
60 | A tutorial is not included here: for full and concrete examples, see | |
61 | the existing implemented orchestrator modules in the Ceph source tree. | |
62 | ||
63 | Glossary | |
64 | -------- | |
65 | ||
66 | Stateful service | |
67 | a daemon that uses local storage, such as OSD or mon. | |
68 | ||
69 | Stateless service | |
70 | a daemon that doesn't use any local storage, such | |
71 | as an MDS, RGW, nfs-ganesha, iSCSI gateway. | |
72 | ||
73 | Label | |
74 | arbitrary string tags that may be applied by administrators | |
9f95a23c TL |
75 | to hosts. Typically administrators use labels to indicate |
76 | which hosts should run which kinds of service. Labels are | |
77 | advisory (from human input) and do not guarantee that hosts | |
11fdf7f2 TL |
78 | have particular physical capabilities. |
79 | ||
80 | Drive group | |
81 | collection of block devices with common/shared OSD | |
82 | formatting (typically one or more SSDs acting as | |
83 | journals/dbs for a group of HDDs). | |
84 | ||
85 | Placement | |
9f95a23c | 86 | choice of which host is used to run a service. |
11fdf7f2 TL |
87 | |
88 | Key Concepts | |
89 | ------------ | |
90 | ||
91 | The underlying orchestrator remains the source of truth for information | |
92 | about whether a service is running, what is running where, which | |
9f95a23c | 93 | hosts are available, etc. Orchestrator modules should avoid taking |
11fdf7f2 TL |
94 | any internal copies of this information, and read it directly from |
95 | the orchestrator backend as much as possible. | |
96 | ||
9f95a23c | 97 | Bootstrapping hosts and adding them to the underlying orchestration |
11fdf7f2 | 98 | system is outside the scope of Ceph's orchestrator interface. Ceph |
9f95a23c | 99 | can only work on hosts when the orchestrator is already aware of them. |
11fdf7f2 TL |
100 | |
101 | Calls to orchestrator modules are all asynchronous, and return *completion* | |
102 | objects (see below) rather than returning values immediately. | |
103 | ||
104 | Where possible, placement of stateless services should be left up to the | |
105 | orchestrator. | |
106 | ||
107 | Completions and batching | |
108 | ------------------------ | |
109 | ||
110 | All methods that read or modify the state of the system can potentially | |
9f95a23c TL |
111 | be long running. To handle that, all such methods return a *Completion* |
112 | object. Orchestrator modules | |
113 | must implement the *process* method: this takes a list of completions, and | |
11fdf7f2 TL |
114 | is responsible for checking if they're finished, and advancing the underlying |
115 | operations as needed. | |
116 | ||
117 | Each orchestrator module implements its own underlying mechanisms | |
118 | for completions. This might involve running the underlying operations | |
119 | in threads, or batching the operations up before later executing | |
120 | in one go in the background. If implementing such a batching pattern, the | |
121 | module would do no work on any operation until it appeared in a list | |
9f95a23c | 122 | of completions passed into *process*. |
11fdf7f2 | 123 | |
9f95a23c TL |
124 | Some operations need to show a progress. Those operations need to add |
125 | a *ProgressReference* to the completion. At some point, the progress reference | |
126 | becomes *effective*, meaning that the operation has really happened | |
127 | (e.g. a service has actually been started). | |
11fdf7f2 | 128 | |
9f95a23c | 129 | .. automethod:: Orchestrator.process |
11fdf7f2 | 130 | |
9f95a23c | 131 | .. autoclass:: Completion |
11fdf7f2 TL |
132 | :members: |
133 | ||
9f95a23c | 134 | .. autoclass:: ProgressReference |
81eedcae TL |
135 | :members: |
136 | ||
11fdf7f2 TL |
137 | Error Handling |
138 | -------------- | |
139 | ||
140 | The main goal of error handling within orchestrator modules is to provide debug information to | |
141 | assist users when dealing with deployment errors. | |
142 | ||
143 | .. autoclass:: OrchestratorError | |
144 | .. autoclass:: NoOrchestrator | |
145 | .. autoclass:: OrchestratorValidationError | |
146 | ||
147 | ||
148 | In detail, orchestrators need to explicitly deal with different kinds of errors: | |
149 | ||
150 | 1. No orchestrator configured | |
151 | ||
152 | See :class:`NoOrchestrator`. | |
153 | ||
154 | 2. An orchestrator doesn't implement a specific method. | |
155 | ||
156 | For example, an Orchestrator doesn't support ``add_host``. | |
157 | ||
158 | In this case, a ``NotImplementedError`` is raised. | |
159 | ||
160 | 3. Missing features within implemented methods. | |
161 | ||
162 | E.g. optional parameters to a command that are not supported by the | |
9f95a23c | 163 | backend (e.g. the hosts field in :func:`Orchestrator.apply_mons` command with the rook backend). |
11fdf7f2 TL |
164 | |
165 | See :class:`OrchestratorValidationError`. | |
166 | ||
167 | 4. Input validation errors | |
168 | ||
9f95a23c | 169 | The ``orchestrator`` module and other calling modules are supposed to |
11fdf7f2 TL |
170 | provide meaningful error messages. |
171 | ||
172 | See :class:`OrchestratorValidationError`. | |
173 | ||
174 | 5. Errors when actually executing commands | |
175 | ||
176 | The resulting Completion should contain an error string that assists in understanding the | |
9f95a23c | 177 | problem. In addition, :func:`Completion.is_errored` is set to ``True`` |
11fdf7f2 TL |
178 | |
179 | 6. Invalid configuration in the orchestrator modules | |
180 | ||
181 | This can be tackled similar to 5. | |
182 | ||
183 | ||
184 | All other errors are unexpected orchestrator issues and thus should raise an exception that are then | |
185 | logged into the mgr log file. If there is a completion object at that point, | |
9f95a23c | 186 | :func:`Completion.result` may contain an error message. |
11fdf7f2 TL |
187 | |
188 | ||
189 | Excluded functionality | |
190 | ---------------------- | |
191 | ||
192 | - Ceph's orchestrator interface is not a general purpose framework for | |
193 | managing linux servers -- it is deliberately constrained to manage | |
194 | the Ceph cluster's services only. | |
195 | - Multipathed storage is not handled (multipathing is unnecessary for | |
196 | Ceph clusters). Each drive is assumed to be visible only on | |
9f95a23c | 197 | a single host. |
11fdf7f2 TL |
198 | |
199 | Host management | |
200 | --------------- | |
201 | ||
202 | .. automethod:: Orchestrator.add_host | |
203 | .. automethod:: Orchestrator.remove_host | |
204 | .. automethod:: Orchestrator.get_hosts | |
9f95a23c TL |
205 | .. automethod:: Orchestrator.update_host_addr |
206 | .. automethod:: Orchestrator.add_host_label | |
207 | .. automethod:: Orchestrator.remove_host_label | |
11fdf7f2 | 208 | |
9f95a23c TL |
209 | .. autoclass:: HostSpec |
210 | ||
211 | Devices | |
212 | ------- | |
11fdf7f2 TL |
213 | |
214 | .. automethod:: Orchestrator.get_inventory | |
215 | .. autoclass:: InventoryFilter | |
11fdf7f2 | 216 | |
9f95a23c TL |
217 | .. py:currentmodule:: ceph.deployment.inventory |
218 | ||
219 | .. autoclass:: Devices | |
11fdf7f2 TL |
220 | :members: |
221 | ||
9f95a23c TL |
222 | .. autoclass:: Device |
223 | :members: | |
224 | ||
225 | .. py:currentmodule:: orchestrator | |
226 | ||
227 | Placement | |
228 | --------- | |
229 | ||
230 | A :ref:`orchestrator-cli-placement-spec` defines the placement of | |
231 | daemons of a specifc service. | |
232 | ||
233 | In general, stateless services do not require any specific placement | |
234 | rules as they can run anywhere that sufficient system resources | |
235 | are available. However, some orchestrators may not include the | |
236 | functionality to choose a location in this way. Optionally, you can | |
237 | specify a location when creating a stateless service. | |
238 | ||
239 | ||
240 | .. py:currentmodule:: ceph.deployment.service_spec | |
241 | ||
242 | .. autoclass:: PlacementSpec | |
243 | :members: | |
244 | ||
245 | .. py:currentmodule:: orchestrator | |
246 | ||
247 | ||
248 | Services | |
249 | -------- | |
250 | ||
11fdf7f2 TL |
251 | .. autoclass:: ServiceDescription |
252 | ||
9f95a23c TL |
253 | .. py:currentmodule:: ceph.deployment.service_spec |
254 | ||
255 | .. autoclass:: ServiceSpec | |
256 | ||
257 | .. py:currentmodule:: orchestrator | |
258 | ||
259 | .. automethod:: Orchestrator.describe_service | |
11fdf7f2 TL |
260 | |
261 | .. automethod:: Orchestrator.service_action | |
9f95a23c TL |
262 | .. automethod:: Orchestrator.remove_service |
263 | ||
264 | ||
265 | Daemons | |
266 | ------- | |
267 | ||
268 | .. automethod:: Orchestrator.list_daemons | |
269 | .. automethod:: Orchestrator.remove_daemons | |
270 | .. automethod:: Orchestrator.daemon_action | |
11fdf7f2 TL |
271 | |
272 | OSD management | |
273 | -------------- | |
274 | ||
275 | .. automethod:: Orchestrator.create_osds | |
11fdf7f2 | 276 | |
9f95a23c TL |
277 | .. automethod:: Orchestrator.blink_device_light |
278 | .. autoclass:: DeviceLightLoc | |
11fdf7f2 | 279 | |
9f95a23c TL |
280 | .. _orchestrator-osd-replace: |
281 | ||
282 | OSD Replacement | |
283 | ^^^^^^^^^^^^^^^ | |
284 | ||
285 | See :ref:`rados-replacing-an-osd` for the underlying process. | |
286 | ||
287 | Replacing OSDs is fundamentally a two-staged process, as users need to | |
288 | physically replace drives. The orchestrator therefor exposes this two-staged process. | |
289 | ||
290 | Phase one is a call to :meth:`Orchestrator.remove_daemons` with ``destroy=True`` in order to mark | |
291 | the OSD as destroyed. | |
292 | ||
293 | ||
294 | Phase two is a call to :meth:`Orchestrator.create_osds` with a Drive Group with | |
295 | ||
296 | .. py:currentmodule:: ceph.deployment.drive_group | |
297 | ||
298 | :attr:`DriveGroupSpec.osd_id_claims` set to the destroyed OSD ids. | |
299 | ||
300 | .. py:currentmodule:: orchestrator | |
301 | ||
302 | Monitors | |
303 | -------- | |
304 | ||
305 | .. automethod:: Orchestrator.add_mon | |
306 | .. automethod:: Orchestrator.apply_mon | |
11fdf7f2 TL |
307 | |
308 | Stateless Services | |
309 | ------------------ | |
310 | ||
9f95a23c TL |
311 | .. automethod:: Orchestrator.add_mgr |
312 | .. automethod:: Orchestrator.apply_mgr | |
313 | .. automethod:: Orchestrator.add_mds | |
314 | .. automethod:: Orchestrator.apply_mds | |
315 | .. automethod:: Orchestrator.add_rbd_mirror | |
316 | .. automethod:: Orchestrator.apply_rbd_mirror | |
317 | ||
318 | .. py:currentmodule:: ceph.deployment.service_spec | |
319 | ||
320 | .. autoclass:: RGWSpec | |
321 | ||
322 | .. py:currentmodule:: orchestrator | |
323 | ||
324 | .. automethod:: Orchestrator.add_rgw | |
325 | .. automethod:: Orchestrator.apply_rgw | |
326 | ||
327 | .. py:currentmodule:: ceph.deployment.service_spec | |
328 | ||
329 | .. autoclass:: NFSServiceSpec | |
330 | ||
331 | .. py:currentmodule:: orchestrator | |
332 | ||
333 | .. automethod:: Orchestrator.add_nfs | |
334 | .. automethod:: Orchestrator.apply_nfs | |
11fdf7f2 TL |
335 | |
336 | Upgrades | |
337 | -------- | |
338 | ||
339 | .. automethod:: Orchestrator.upgrade_available | |
340 | .. automethod:: Orchestrator.upgrade_start | |
341 | .. automethod:: Orchestrator.upgrade_status | |
11fdf7f2 TL |
342 | .. autoclass:: UpgradeStatusSpec |
343 | ||
344 | Utility | |
345 | ------- | |
346 | ||
347 | .. automethod:: Orchestrator.available | |
9f95a23c | 348 | .. automethod:: Orchestrator.get_feature_set |
11fdf7f2 TL |
349 | |
350 | Client Modules | |
351 | -------------- | |
352 | ||
353 | .. autoclass:: OrchestratorClientMixin | |
354 | :members: |