]>
Commit | Line | Data |
---|---|---|
9f95a23c TL |
1 | .. _drivegroups: |
2 | ||
801d1391 TL |
3 | ========================= |
4 | OSD Service Specification | |
5 | ========================= | |
9f95a23c | 6 | |
801d1391 | 7 | :ref:`orchestrator-cli-service-spec` of type ``osd`` are a way to describe a cluster layout using the properties of disks. |
9f95a23c TL |
8 | It gives the user an abstract way tell ceph which disks should turn into an OSD |
9 | with which configuration without knowing the specifics of device names and paths. | |
10 | ||
11 | Instead of doing this:: | |
12 | ||
801d1391 | 13 | [monitor 1] # ceph orch daemon add osd *<host>*:*<path-to-device>* |
9f95a23c TL |
14 | |
15 | for each device and each host, we can define a yaml|json file that allows us to describe | |
16 | the layout. Here's the most basic example. | |
17 | ||
801d1391 | 18 | Create a file called i.e. osd_spec.yml |
9f95a23c TL |
19 | |
20 | .. code-block:: yaml | |
21 | ||
801d1391 TL |
22 | service_type: osd |
23 | service_id: default_drive_group <- name of the drive_group (name can be custom) | |
24 | placement: | |
25 | host_pattern: '*' <- which hosts to target, currently only supports globs | |
26 | data_devices: <- the type of devices you are applying specs to | |
27 | all: true <- a filter, check below for a full list | |
9f95a23c TL |
28 | |
29 | This would translate to: | |
30 | ||
31 | Turn any available(ceph-volume decides what 'available' is) into an OSD on all hosts that match | |
32 | the glob pattern '*'. (The glob pattern matches against the registered hosts from `host ls`) | |
33 | There will be a more detailed section on host_pattern down below. | |
34 | ||
35 | and pass it to `osd create` like so:: | |
36 | ||
801d1391 | 37 | [monitor 1] # ceph orch apply osd -i /path/to/osd_spec.yml |
9f95a23c TL |
38 | |
39 | This will go out on all the matching hosts and deploy these OSDs. | |
40 | ||
41 | Since we want to have more complex setups, there are more filters than just the 'all' filter. | |
42 | ||
f6b5b4d7 TL |
43 | Also, there is a `--dry-run` flag that can be passed to the `apply osd` command, which gives you a synopsis |
44 | of the proposed layout. | |
45 | ||
46 | Example:: | |
47 | ||
48 | [monitor 1] # ceph orch apply osd -i /path/to/osd_spec.yml --dry-run | |
49 | ||
50 | ||
9f95a23c TL |
51 | |
52 | Filters | |
53 | ======= | |
54 | ||
f6b5b4d7 TL |
55 | .. note:: |
56 | Filters are applied using a `AND` gate by default. This essentially means that a drive needs to fulfill all filter | |
57 | criteria in order to get selected. | |
58 | If you wish to change this behavior you can adjust this behavior by setting | |
59 | ||
60 | `filter_logic: OR` # valid arguments are `AND`, `OR` | |
61 | ||
62 | in the OSD Specification. | |
63 | ||
9f95a23c TL |
64 | You can assign disks to certain groups by their attributes using filters. |
65 | ||
66 | The attributes are based off of ceph-volume's disk query. You can retrieve the information | |
67 | with:: | |
68 | ||
69 | ceph-volume inventory </path/to/disk> | |
70 | ||
71 | Vendor or Model: | |
72 | ------------------- | |
73 | ||
74 | You can target specific disks by their Vendor or by their Model | |
75 | ||
76 | .. code-block:: yaml | |
77 | ||
78 | model: disk_model_name | |
79 | ||
80 | or | |
81 | ||
82 | .. code-block:: yaml | |
83 | ||
84 | vendor: disk_vendor_name | |
85 | ||
86 | ||
87 | Size: | |
88 | -------------- | |
89 | ||
90 | You can also match by disk `Size`. | |
91 | ||
92 | .. code-block:: yaml | |
93 | ||
94 | size: size_spec | |
95 | ||
96 | Size specs: | |
97 | ___________ | |
98 | ||
99 | Size specification of format can be of form: | |
100 | ||
101 | * LOW:HIGH | |
102 | * :HIGH | |
103 | * LOW: | |
104 | * EXACT | |
105 | ||
106 | Concrete examples: | |
107 | ||
108 | Includes disks of an exact size:: | |
109 | ||
110 | size: '10G' | |
111 | ||
112 | Includes disks which size is within the range:: | |
113 | ||
114 | size: '10G:40G' | |
115 | ||
116 | Includes disks less than or equal to 10G in size:: | |
117 | ||
118 | size: ':10G' | |
119 | ||
120 | ||
121 | Includes disks equal to or greater than 40G in size:: | |
122 | ||
123 | size: '40G:' | |
124 | ||
125 | Sizes don't have to be exclusively in Gigabyte(G). | |
126 | ||
127 | Supported units are Megabyte(M), Gigabyte(G) and Terrabyte(T). Also appending the (B) for byte is supported. MB, GB, TB | |
128 | ||
129 | ||
130 | Rotational: | |
131 | ----------- | |
132 | ||
133 | This operates on the 'rotational' attribute of the disk. | |
134 | ||
135 | .. code-block:: yaml | |
136 | ||
137 | rotational: 0 | 1 | |
138 | ||
139 | `1` to match all disks that are rotational | |
140 | ||
141 | `0` to match all disks that are non-rotational (SSD, NVME etc) | |
142 | ||
143 | ||
144 | All: | |
145 | ------------ | |
146 | ||
147 | This will take all disks that are 'available' | |
148 | ||
149 | Note: This is exclusive for the data_devices section. | |
150 | ||
151 | .. code-block:: yaml | |
152 | ||
153 | all: true | |
154 | ||
155 | ||
156 | Limiter: | |
157 | -------- | |
158 | ||
159 | When you specified valid filters but want to limit the amount of matching disks you can use the 'limit' directive. | |
160 | ||
161 | .. code-block:: yaml | |
162 | ||
163 | limit: 2 | |
164 | ||
165 | For example, if you used `vendor` to match all disks that are from `VendorA` but only want to use the first two | |
166 | you could use `limit`. | |
167 | ||
168 | .. code-block:: yaml | |
169 | ||
170 | data_devices: | |
171 | vendor: VendorA | |
172 | limit: 2 | |
173 | ||
174 | Note: Be aware that `limit` is really just a last resort and shouldn't be used if it can be avoided. | |
175 | ||
176 | ||
177 | Additional Options | |
178 | =================== | |
179 | ||
180 | There are multiple optional settings you can use to change the way OSDs are deployed. | |
181 | You can add these options to the base level of a DriveGroup for it to take effect. | |
182 | ||
183 | This example would deploy all OSDs with encryption enabled. | |
184 | ||
185 | .. code-block:: yaml | |
186 | ||
801d1391 TL |
187 | service_type: osd |
188 | service_id: example_osd_spec | |
189 | placement: | |
190 | host_pattern: '*' | |
191 | data_devices: | |
192 | all: true | |
1911f103 | 193 | encrypted: true |
9f95a23c TL |
194 | |
195 | See a full list in the DriveGroupSpecs | |
196 | ||
197 | .. py:currentmodule:: ceph.deployment.drive_group | |
198 | ||
199 | .. autoclass:: DriveGroupSpec | |
200 | :members: | |
201 | :exclude-members: from_json | |
202 | ||
203 | Examples | |
204 | ======== | |
205 | ||
206 | The simple case | |
207 | --------------- | |
208 | ||
209 | All nodes with the same setup:: | |
210 | ||
211 | 20 HDDs | |
212 | Vendor: VendorA | |
213 | Model: HDD-123-foo | |
214 | Size: 4TB | |
215 | ||
216 | 2 SSDs | |
217 | Vendor: VendorB | |
218 | Model: MC-55-44-ZX | |
219 | Size: 512GB | |
220 | ||
221 | This is a common setup and can be described quite easily: | |
222 | ||
223 | .. code-block:: yaml | |
224 | ||
801d1391 TL |
225 | service_type: osd |
226 | service_id: osd_spec_default | |
227 | placement: | |
9f95a23c | 228 | host_pattern: '*' |
801d1391 TL |
229 | data_devices: |
230 | model: HDD-123-foo <- note that HDD-123 would also be valid | |
231 | db_devices: | |
232 | model: MC-55-44-XZ <- same here, MC-55-44 is valid | |
9f95a23c TL |
233 | |
234 | However, we can improve it by reducing the filters on core properties of the drives: | |
235 | ||
236 | .. code-block:: yaml | |
237 | ||
801d1391 TL |
238 | service_type: osd |
239 | service_id: osd_spec_default | |
240 | placement: | |
9f95a23c | 241 | host_pattern: '*' |
801d1391 TL |
242 | data_devices: |
243 | rotational: 1 | |
244 | db_devices: | |
245 | rotational: 0 | |
9f95a23c TL |
246 | |
247 | Now, we enforce all rotating devices to be declared as 'data devices' and all non-rotating devices will be used as shared_devices (wal, db) | |
248 | ||
249 | If you know that drives with more than 2TB will always be the slower data devices, you can also filter by size: | |
250 | ||
251 | .. code-block:: yaml | |
252 | ||
801d1391 TL |
253 | service_type: osd |
254 | service_id: osd_spec_default | |
255 | placement: | |
9f95a23c | 256 | host_pattern: '*' |
801d1391 TL |
257 | data_devices: |
258 | size: '2TB:' | |
259 | db_devices: | |
260 | size: ':2TB' | |
9f95a23c TL |
261 | |
262 | Note: All of the above DriveGroups are equally valid. Which of those you want to use depends on taste and on how much you expect your node layout to change. | |
263 | ||
264 | ||
265 | The advanced case | |
266 | ----------------- | |
267 | ||
268 | Here we have two distinct setups:: | |
269 | ||
270 | 20 HDDs | |
271 | Vendor: VendorA | |
272 | Model: HDD-123-foo | |
273 | Size: 4TB | |
274 | ||
275 | 12 SSDs | |
276 | Vendor: VendorB | |
277 | Model: MC-55-44-ZX | |
278 | Size: 512GB | |
279 | ||
280 | 2 NVMEs | |
281 | Vendor: VendorC | |
282 | Model: NVME-QQQQ-987 | |
283 | Size: 256GB | |
284 | ||
285 | ||
286 | * 20 HDDs should share 2 SSDs | |
287 | * 10 SSDs should share 2 NVMes | |
288 | ||
289 | This can be described with two layouts. | |
290 | ||
291 | .. code-block:: yaml | |
292 | ||
801d1391 TL |
293 | service_type: osd |
294 | service_id: osd_spec_hdd | |
295 | placement: | |
9f95a23c | 296 | host_pattern: '*' |
801d1391 TL |
297 | data_devices: |
298 | rotational: 0 | |
299 | db_devices: | |
300 | model: MC-55-44-XZ | |
301 | limit: 2 (db_slots is actually to be favoured here, but it's not implemented yet) | |
302 | ||
303 | service_type: osd | |
304 | service_id: osd_spec_ssd | |
305 | placement: | |
9f95a23c | 306 | host_pattern: '*' |
801d1391 TL |
307 | data_devices: |
308 | model: MC-55-44-XZ | |
309 | db_devices: | |
310 | vendor: VendorC | |
9f95a23c TL |
311 | |
312 | This would create the desired layout by using all HDDs as data_devices with two SSD assigned as dedicated db/wal devices. | |
313 | The remaining SSDs(8) will be data_devices that have the 'VendorC' NVMEs assigned as dedicated db/wal devices. | |
314 | ||
315 | The advanced case (with non-uniform nodes) | |
316 | ------------------------------------------ | |
317 | ||
318 | The examples above assumed that all nodes have the same drives. That's however not always the case. | |
319 | ||
320 | Node1-5:: | |
321 | ||
322 | 20 HDDs | |
323 | Vendor: Intel | |
324 | Model: SSD-123-foo | |
325 | Size: 4TB | |
326 | 2 SSDs | |
327 | Vendor: VendorA | |
328 | Model: MC-55-44-ZX | |
329 | Size: 512GB | |
330 | ||
331 | Node6-10:: | |
332 | ||
333 | 5 NVMEs | |
334 | Vendor: Intel | |
335 | Model: SSD-123-foo | |
336 | Size: 4TB | |
337 | 20 SSDs | |
338 | Vendor: VendorA | |
339 | Model: MC-55-44-ZX | |
340 | Size: 512GB | |
341 | ||
342 | You can use the 'host_pattern' key in the layout to target certain nodes. Salt target notation helps to keep things easy. | |
343 | ||
344 | ||
345 | .. code-block:: yaml | |
346 | ||
801d1391 TL |
347 | service_type: osd |
348 | service_id: osd_spec_node_one_to_five | |
349 | placement: | |
9f95a23c | 350 | host_pattern: 'node[1-5]' |
801d1391 TL |
351 | data_devices: |
352 | rotational: 1 | |
353 | db_devices: | |
354 | rotational: 0 | |
355 | ||
356 | ||
357 | service_type: osd | |
358 | service_id: osd_spec_six_to_ten | |
359 | placement: | |
9f95a23c | 360 | host_pattern: 'node[6-10]' |
801d1391 TL |
361 | data_devices: |
362 | model: MC-55-44-XZ | |
363 | db_devices: | |
364 | model: SSD-123-foo | |
9f95a23c | 365 | |
801d1391 | 366 | This applies different OSD specs to different hosts depending on the `host_pattern` key. |
9f95a23c TL |
367 | |
368 | Dedicated wal + db | |
369 | ------------------ | |
370 | ||
371 | All previous cases co-located the WALs with the DBs. | |
372 | It's however possible to deploy the WAL on a dedicated device as well, if it makes sense. | |
373 | ||
374 | :: | |
375 | ||
376 | 20 HDDs | |
377 | Vendor: VendorA | |
378 | Model: SSD-123-foo | |
379 | Size: 4TB | |
380 | ||
381 | 2 SSDs | |
382 | Vendor: VendorB | |
383 | Model: MC-55-44-ZX | |
384 | Size: 512GB | |
385 | ||
386 | 2 NVMEs | |
387 | Vendor: VendorC | |
388 | Model: NVME-QQQQ-987 | |
389 | Size: 256GB | |
390 | ||
391 | ||
801d1391 | 392 | The OSD spec for this case would look like the following (using the `model` filter): |
9f95a23c TL |
393 | |
394 | .. code-block:: yaml | |
395 | ||
801d1391 TL |
396 | service_type: osd |
397 | service_id: osd_spec_default | |
398 | placement: | |
9f95a23c | 399 | host_pattern: '*' |
801d1391 TL |
400 | data_devices: |
401 | model: MC-55-44-XZ | |
402 | db_devices: | |
403 | model: SSD-123-foo | |
404 | wal_devices: | |
405 | model: NVME-QQQQ-987 | |
9f95a23c TL |
406 | |
407 | ||
408 | This can easily be done with other filters, like `size` or `vendor` as well. | |
409 |