]>
Commit | Line | Data |
---|---|---|
11fdf7f2 TL |
1 | |
2 | .. _kubernetes-dev: | |
3 | ||
4 | ======================================= | |
5 | Hacking on Ceph in Kubernetes with Rook | |
6 | ======================================= | |
7 | ||
8 | .. warning:: | |
9 | ||
10 | This is *not* official user documentation for setting up production | |
11 | Ceph clusters with Kubernetes. It is aimed at developers who want | |
12 | to hack on Ceph in Kubernetes. | |
13 | ||
14 | This guide is aimed at Ceph developers getting started with running | |
15 | in a Kubernetes environment. It assumes that you may be hacking on Rook, | |
16 | Ceph or both, so everything is built from source. | |
17 | ||
9f95a23c TL |
18 | TL;DR for hacking on MGR modules |
19 | ================================ | |
20 | ||
21 | Make your changes to the Python code base and then from Ceph's | |
22 | ``build`` directory, run:: | |
23 | ||
24 | ../src/script/kubejacker/kubejacker.sh '192.168.122.1:5000' | |
25 | ||
26 | where ``'192.168.122.1:5000'`` is a local docker registry and | |
27 | Rook's ``CephCluster`` CR uses ``image: 192.168.122.1:5000/ceph/ceph:latest``. | |
28 | ||
11fdf7f2 TL |
29 | 1. Build a kubernetes cluster |
30 | ============================= | |
31 | ||
32 | Before installing Ceph/Rook, make sure you've got a working kubernetes | |
33 | cluster with some nodes added (i.e. ``kubectl get nodes`` shows you something). | |
34 | The rest of this guide assumes that your development workstation has network | |
35 | access to your kubernetes cluster, such that ``kubectl`` works from your | |
36 | workstation. | |
37 | ||
9f95a23c | 38 | `There are many ways <https://kubernetes.io/docs/setup/>`_ |
11fdf7f2 TL |
39 | to build a kubernetes cluster: here we include some tips/pointers on where |
40 | to get started. | |
41 | ||
9f95a23c TL |
42 | `kubic-terraform-kvm <https://github.com/kubic-project/kubic-terraform-kvm>`_ |
43 | might also be an option. | |
11fdf7f2 | 44 | |
9f95a23c TL |
45 | Or `Host your own <https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/>`_ with |
46 | ``kubeadm``. | |
11fdf7f2 | 47 | |
9f95a23c TL |
48 | Some Tips |
49 | --------- | |
11fdf7f2 TL |
50 | |
51 | Here are some tips for a smoother ride with ``kubeadm``: | |
52 | ||
11fdf7f2 TL |
53 | - If you have previously added any yum/deb repos for kubernetes packages, |
54 | disable them before trying to use the packages.cloud.google.com repository. | |
55 | If you don't, you'll get quite confusing conflicts. | |
56 | - Even if your distro already has docker, make sure you're installing it | |
57 | a version from docker.com that is within the range mentioned in the | |
58 | kubeadm install instructions. Especially, note that the docker in CentOS 7 | |
59 | will *not* work. | |
60 | ||
61 | Hosted elsewhere | |
62 | ---------------- | |
63 | ||
64 | If you do not have any servers to hand, you might try a pure | |
65 | container provider such as Google Compute Engine. Your mileage may | |
66 | vary when it comes to what kinds of storage devices are visible | |
67 | to your kubernetes cluster. | |
68 | ||
69 | Make sure you check how much it's costing you before you spin up a big cluster! | |
70 | ||
71 | ||
9f95a23c TL |
72 | 2. Run a docker registry |
73 | ======================== | |
11fdf7f2 | 74 | |
9f95a23c | 75 | Run this somewhere accessible from both your workstation and your |
11fdf7f2 TL |
76 | kubernetes cluster (i.e. so that ``docker push/pull`` just works everywhere). |
77 | This is likely to be the same host you're using as your kubernetes master. | |
78 | ||
79 | 1. Install the ``docker-distribution`` package. | |
80 | 2. If you want to configure the port, edit ``/etc/docker-distribution/registry/config.yml`` | |
81 | 3. Enable the registry service: | |
82 | ||
83 | :: | |
84 | ||
85 | systemctl enable docker-distribution | |
86 | systemctl start docker-distribution | |
87 | ||
9f95a23c | 88 | You may need to mark the registry as **insecure**. |
11fdf7f2 TL |
89 | |
90 | 3. Build Rook | |
91 | ============= | |
92 | ||
93 | .. note:: | |
94 | ||
9f95a23c | 95 | Building Rook is **not required** to make changes to Ceph. |
11fdf7f2 TL |
96 | |
97 | Install Go if you don't already have it. | |
98 | ||
99 | Download the Rook source code: | |
100 | ||
101 | :: | |
102 | ||
103 | go get github.com/rook/rook | |
104 | ||
105 | # Ignore this warning, as Rook is not a conventional go package | |
106 | can't load package: package github.com/rook/rook: no Go files in /home/jspray/go/src/github.com/rook/rook | |
107 | ||
108 | You will now have a Rook source tree in ~/go/src/github.com/rook/rook -- you may | |
109 | be tempted to clone it elsewhere, but your life will be easier if you | |
110 | leave it in your GOPATH. | |
111 | ||
112 | Run ``make`` in the root of your Rook tree to build its binaries and containers: | |
113 | ||
114 | :: | |
115 | ||
116 | make | |
117 | ... | |
118 | === saving image build-9204c79b/ceph-amd64 | |
119 | === docker build build-9204c79b/ceph-toolbox-base-amd64 | |
120 | sha256:653bb4f8d26d6178570f146fe637278957e9371014ea9fce79d8935d108f1eaa | |
121 | === docker build build-9204c79b/ceph-toolbox-amd64 | |
122 | sha256:445d97b71e6f8de68ca1c40793058db0b7dd1ebb5d05789694307fd567e13863 | |
123 | === caching image build-9204c79b/ceph-toolbox-base-amd64 | |
124 | ||
125 | You can use ``docker image ls`` to see the resulting built images. The | |
126 | images you care about are the ones with tags ending "ceph-amd64" (used | |
127 | for the Rook operator and Ceph daemons) and "ceph-toolbox-amd64" (used | |
128 | for the "toolbox" container where the CLI is run). | |
129 | ||
11fdf7f2 TL |
130 | 4. Build Ceph |
131 | ============= | |
132 | ||
9f95a23c TL |
133 | .. note:: |
134 | ||
135 | Building Ceph is **not required** to make changes to MGR modules | |
136 | written in Python. | |
137 | ||
11fdf7f2 | 138 | |
9f95a23c TL |
139 | The Rook containers and the Ceph containers are independent now. Note that |
140 | Rook's Ceph client libraries need to communicate with the Ceph cluster, | |
141 | therefore a compatible major version is required. | |
142 | ||
143 | You can run a CentOS docker container with access to your Ceph source | |
11fdf7f2 TL |
144 | tree using a command like: |
145 | ||
146 | :: | |
147 | ||
148 | docker run -i -v /my/ceph/src:/my/ceph/src -t centos:7 /bin/bash | |
149 | ||
150 | Once you have built Ceph, you can inject the resulting binaries into | |
151 | the Rook container image using the ``kubejacker.sh`` script (run from | |
152 | your build directory but from *outside* your build container). | |
153 | ||
9f95a23c TL |
154 | 5. Run Kubejacker |
155 | ================= | |
156 | ||
157 | ``kubejacker`` needs access to your docker registry. Execute the script | |
158 | to build a docker image containing your latest Ceph binaries: | |
11fdf7f2 TL |
159 | |
160 | :: | |
161 | ||
9f95a23c | 162 | build$ ../src/script/kubejacker/kubejacker.sh "<host>:<port>" |
11fdf7f2 | 163 | |
11fdf7f2 TL |
164 | |
165 | Now you've got your freshly built Rook and freshly built Ceph into | |
166 | a single container image, ready to run. Next time you change something | |
167 | in Ceph, you can re-run this to update your image and restart your | |
168 | kubernetes containers. If you change something in Rook, then re-run the Rook | |
169 | build, and the Ceph build too. | |
170 | ||
171 | 5. Run a Rook cluster | |
172 | ===================== | |
173 | ||
9f95a23c TL |
174 | Please refer to `Rook's documentation <https://rook.io/docs/rook/master/ceph-quickstart.html>`_ |
175 | for setting up a Rook operator, a Ceph cluster and the toolbox. | |
11fdf7f2 TL |
176 | |
177 | The Rook source tree includes example .yaml files in | |
9f95a23c TL |
178 | ``cluster/examples/kubernetes/ceph/``. Copy these into |
179 | a working directory, and edit as necessary to configure | |
180 | the setup you want: | |
11fdf7f2 | 181 | |
9f95a23c | 182 | - Ensure that ``spec.cephVersion.image`` points to your docker registry:: |
11fdf7f2 | 183 | |
9f95a23c TL |
184 | spec: |
185 | cephVersion: | |
186 | allowUnsupported: true | |
187 | image: 192.168.122.1:5000/ceph/ceph:latest | |
11fdf7f2 | 188 | |
11fdf7f2 TL |
189 | Then, load the configuration into the kubernetes API using ``kubectl``: |
190 | ||
191 | :: | |
192 | ||
9f95a23c TL |
193 | kubectl apply -f ./cluster-test.yaml |
194 | ||
195 | Use ``kubectl -n rook-ceph get pods`` to check the operator | |
196 | pod the Ceph daemons and toolbox are is coming up. | |
11fdf7f2 | 197 | |
9f95a23c | 198 | Once everything is up and running, |
11fdf7f2 TL |
199 | you should be able to open a shell in the toolbox container and |
200 | run ``ceph status``. | |
201 | ||
202 | If your mon services start but the rest don't, it could be that they're | |
203 | unable to form a quorum due to a Kubernetes networking issue: check that | |
204 | containers in your Kubernetes cluster can ping containers on other nodes. | |
205 | ||
206 | Cheat sheet | |
207 | =========== | |
208 | ||
209 | Open a shell in your toolbox container:: | |
210 | ||
9f95a23c | 211 | kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath="{.items[0].metadata.name}") -- bash |
11fdf7f2 TL |
212 | |
213 | Inspect the Rook operator container's logs:: | |
214 | ||
9f95a23c | 215 | kubectl -n rook-ceph logs -l app=rook-ceph-operator |
11fdf7f2 TL |
216 | |
217 | Inspect the ceph-mgr container's logs:: | |
218 | ||
219 | kubectl -n rook-ceph logs -l app=rook-ceph-mgr | |
220 |