]>
Commit | Line | Data |
---|---|---|
1911f103 TL |
1 | ======================= |
2 | Developing with cephadm | |
3 | ======================= | |
4 | ||
5 | There are several ways to develop with cephadm. Which you use depends | |
6 | on what you're trying to accomplish. | |
7 | ||
8 | vstart --cephadm | |
9 | ================ | |
10 | ||
11 | - Start a cluster with vstart, with cephadm configured | |
12 | - Manage any additional daemons with cephadm | |
13 | ||
14 | In this case, the mon and manager at a minimum are running in the usual | |
15 | vstart way, not managed by cephadm. But cephadm is enabled and the local | |
16 | host is added, so you can deploy additional daemons or add additional hosts. | |
17 | ||
18 | This works well for developing cephadm itself, because any mgr/cephadm | |
19 | or cephadm/cephadm code changes can be applied by kicking ceph-mgr | |
20 | with ``ceph mgr fail x``. (When the mgr (re)starts, it loads the | |
21 | cephadm/cephadm script into memory.) | |
22 | ||
23 | :: | |
24 | ||
25 | MON=1 MGR=1 OSD=0 MDS=0 ../src/vstart.sh -d -n -x --cephadm | |
26 | ||
27 | - ``~/.ssh/id_dsa[.pub]`` is used as the cluster key. It is assumed that | |
28 | this key is authorized to ssh to root@`hostname`. | |
29 | - No service spec is defined for mon or mgr, which means that cephadm | |
30 | does not try to manage them. | |
31 | - You'll see health warnings from cephadm about stray daemons--that's because | |
32 | the vstart-launched mon and mgr aren't controlled by cephadm. | |
33 | - The default image is ``quay.io/ceph-ci/ceph:master``, but you can change | |
34 | this by passing ``-o container_image=...`` or ``ceph config set global container_image ...``. | |
35 | ||
36 | ||
37 | cstart and cpatch | |
38 | ================= | |
39 | ||
40 | The ``cstart.sh`` script will launch a cluster using cephadm and put the | |
41 | conf and keyring in your build dir, so that the ``bin/ceph ...`` CLI works | |
42 | (just like with vstart). The ``ckill.sh`` script will tear it down. | |
43 | ||
44 | - A unique but stable fsid is stored in ``fsid`` (in the build dir). | |
45 | - The mon port is random, just like with vstart. | |
46 | - The container image is ``quay.io/ceph-ci/ceph:$tag`` where $tag is | |
47 | the first 8 chars of the fsid. | |
48 | - If the container image doesn't exist yet when you run cstart for the | |
49 | first time, it is built with cpatch. | |
50 | ||
51 | There are a few advantages here: | |
52 | ||
53 | - The cluster is a "normal" cephadm cluster that looks and behaves | |
54 | just like a user's cluster would. In contract, vstart and teuthology | |
55 | clusters tend to be special in subtle (and not-so-subtle) ways. | |
56 | ||
57 | To start a test cluster:: | |
58 | ||
59 | sudo ../src/cstart.sh | |
60 | ||
61 | The last line of this will be a line you can cut+paste to update the | |
62 | container image. For instance:: | |
63 | ||
64 | sudo ../src/script/cpach -t quay.io/ceph-ci/ceph:8f509f4e | |
65 | ||
66 | By default, cpatch will patch everything it can think of from the local | |
67 | build dir into the container image. If you are working on a specific | |
68 | part of the system, though, can you get away with smaller changes so that | |
69 | cpatch runs faster. For instance:: | |
70 | ||
71 | sudo ../src/script/cpach -t quay.io/ceph-ci/ceph:8f509f4e --py | |
72 | ||
73 | will update the mgr modules (minus the dashboard). Or:: | |
74 | ||
75 | sudo ../src/script/cpach -t quay.io/ceph-ci/ceph:8f509f4e --core | |
76 | ||
77 | will do most binaries and libraries. Pass ``-h`` to cpatch for all options. | |
78 | ||
79 | Once the container is updated, you can refresh/restart daemons by bouncing | |
80 | them with:: | |
81 | ||
82 | sudo systemctl restart ceph-`cat fsid`.target | |
83 | ||
84 | When you're done, you can tear down the cluster with:: | |
85 | ||
86 | sudo ../src/ckill.sh # or, | |
87 | sudo ../src/cephadm/cephadm rm-cluster --force --fsid `cat fsid` | |
e306af50 TL |
88 | |
89 | Note regarding network calls from CLI handlers | |
90 | ============================================== | |
91 | ||
92 | Executing any cephadm CLI commands like ``ceph orch ls`` will block | |
93 | the mon command handler thread within the MGR, thus preventing any | |
94 | concurrent CLI calls. Note that pressing ``^C`` will not resolve this | |
95 | situation, as *only* the client will be aborted, but not exceution | |
96 | itself. This means, cephadm will be completely unresonsive, until the | |
97 | execution of the CLI handler is fully completed. Note that even | |
98 | ``ceph orch ps`` will not respond, while another handler is executed. | |
99 | ||
100 | This means, we should only do very few calls to remote hosts synchronously. | |
101 | As a guideline, cephadm should do at most ``O(1)`` network calls in CLI handlers. | |
102 | Everything else should be done asynchronously in other threads, like ``serve()``. |