]>
Commit | Line | Data |
---|---|---|
20effc67 TL |
1 | Continuous Integration Architecture |
2 | =================================== | |
3 | ||
4 | In Ceph, we rely on multiple CI pipelines in our development. Most of these pipelines | |
5 | are centered around Jenkins. And their configurations are generated using `Jenkins Job Builder`_. | |
6 | ||
7 | .. _Jenkins Job Builder: https://docs.openstack.org/infra/jenkins-job-builder/ | |
8 | ||
9 | Let's take the ``make check`` performed by Jenkins as an example. | |
10 | ||
11 | ceph-pull-requests | |
12 | ------------------ | |
13 | ||
14 | ``ceph-pull-requests`` is a jenkins job which gets triggered by a GitHub pull | |
15 | request or a trigger phrase like:: | |
16 | ||
17 | jenkins test make check | |
18 | ||
19 | There are multiple parties involved in this jenkins job: | |
20 | ||
21 | .. graphviz:: | |
22 | ||
23 | digraph { | |
24 | rankdir="LR"; | |
25 | github [ | |
26 | label="<git> git_repo | <webhooks> webhooks | <api> api"; | |
27 | shape=record; | |
28 | href="https://github.com/ceph/ceph"; | |
29 | ]; | |
30 | subgraph cluster_lab { | |
31 | label="Sepia Lab"; | |
32 | href="https://wiki.sepia.ceph.com/doku.php"; | |
33 | shape=circle; | |
34 | apt_mirror [ | |
35 | href="http://apt-mirror.front.sepia.ceph.com"; | |
36 | ]; | |
37 | shaman [ | |
38 | href="https://shaman.ceph.com"; | |
39 | ]; | |
40 | chacra [ | |
41 | peripheries=3; | |
42 | href="https://chacra.ceph.com"; | |
43 | ]; | |
44 | subgraph cluster_jenkins { | |
45 | label="jenkins"; | |
46 | href="https://jenkins.ceph.com"; | |
47 | jenkins_controller [ label = "controller" ]; | |
48 | jenkins_agents [ label = "agents", peripheries=3 ]; | |
49 | }; | |
50 | }; | |
51 | { | |
52 | rank=same; | |
53 | package_repos [ peripheries=3 ]; | |
54 | pypi; | |
55 | npm; | |
56 | } | |
57 | github:webhooks -> jenkins_controller [ label = "notify", color = "crimson" ]; | |
58 | jenkins_controller -> jenkins_agents [ label = "schedule jobs" ]; | |
59 | jenkins_agents -> github:git [ label = "git pull" ]; | |
60 | jenkins_agents -> shaman [ label = "query for chacra repo URL" ]; | |
61 | jenkins_agents -> chacra [ label = "pull build dependencies" ]; | |
62 | jenkins_agents -> package_repos [ label = "pull build dependencies" ]; | |
63 | jenkins_agents -> pypi [ label = "pull Python packages" ]; | |
64 | jenkins_agents -> npm [ label = "pull JavaScript packages" ]; | |
65 | jenkins_agents -> apt_mirror [ label = "pull build dependencies" ]; | |
66 | jenkins_agents -> github:api [ label = "update", color = "crimson" ]; | |
67 | } | |
68 | ||
69 | Where | |
70 | ||
71 | Sepia Lab | |
72 | `Sepia Lab`_ is a test lab used by the Ceph project. This lab offers | |
73 | the storage and computing resources required by our CI infra. | |
74 | ||
75 | Jenkins agents | |
76 | are a set of machines which perform the CI jobs. In this case, they | |
77 | ||
78 | #. pull the git repo from GitHub and | |
79 | #. rebase the pull request against the latest master | |
80 | #. set necessary environment variables | |
81 | #. run ``run-make-check.sh`` | |
82 | ||
83 | Chacra | |
84 | is a server offering RESTful API allowing the clients to store and | |
85 | retrieve binary packages. It also creates the repo for uploaded | |
86 | packages automatically. Once a certain repo is created on chacra, the | |
87 | configured shaman server is updated as well, then we can query shaman | |
88 | for the corresponding repo address. Chacra not only hosts Ceph packages, | |
89 | it also hosts quite a few other packages like various build dependencies. | |
90 | ||
91 | Shaman | |
92 | is a server offering RESTful API allowing the clients to query the | |
93 | information of repos hosted by chacra nodes. Shaman is also known | |
94 | for its `Web UI`_. But please note, shaman does not build the | |
95 | packages, it just offers information on the builds. | |
96 | ||
97 | As the following shows, `chacra`_ manages multiple projects whose metadata | |
98 | are stored in a database. These metadata are exposed via Shaman as a web | |
99 | service. `chacractl`_ is a utility to interact with the `chacra`_ service. | |
100 | ||
101 | .. graphviz:: | |
102 | ||
103 | digraph { | |
104 | libboost [ | |
105 | shape=cylinder; | |
106 | ]; | |
107 | libzbd [ | |
108 | shape=cylinder; | |
109 | ]; | |
110 | other_repos [ | |
111 | label="..."; | |
112 | shape=cylinder; | |
113 | ]; | |
114 | postgresql [ | |
115 | shape=cylinder; | |
116 | style=filled; | |
117 | ] | |
118 | shaman -> postgresql; | |
119 | chacra -> postgresql; | |
120 | chacractl -> chacra; | |
121 | chacra -> libboost; | |
122 | chacra -> libzbd; | |
123 | chacra -> other_repos; | |
124 | } | |
125 | ||
126 | .. _Sepia Lab: https://wiki.sepia.ceph.com/doku.php | |
127 | .. _Web UI: https://shaman.ceph.com | |
128 | ||
129 | build dependencies | |
130 | ------------------ | |
131 | ||
132 | Just like lots of other software projects, Ceph has both build-time and | |
133 | run-time dependencies. Most of time, we are inclined to use the packages | |
134 | prebuilt by the distro. But there are cases where | |
135 | ||
136 | - the necessary dependencies are either missing in the distro, or | |
137 | - their versions are too old, or | |
138 | - they are packaged without some important feature enabled. | |
139 | - we want to ensure that the version of a certain runtime dependency is | |
140 | identical to the one we tested in our lab. | |
141 | ||
142 | No matter what the reason is, we either need to build them from source, or | |
143 | to package them as binary packages instead of using the ones shipped by the | |
144 | distro. Quite a few build-time dependencies are included as git submodules, | |
145 | but in order to avoid rebuilding these dependencies repeatedly, we pre-built | |
146 | some of them and uploaded them to our own repos. So, when performing | |
147 | ``make check``, the building hosts in our CI just pull them from our internal | |
148 | repos hosting these packages instead of building them. | |
149 | ||
150 | So far, following packages are prebuilt for ubuntu focal, and then uploaded to | |
151 | `chacra`_: | |
152 | ||
153 | libboost | |
154 | packages `boost`_. The packages' names are changed from ``libboost-*`` to | |
155 | ``ceph-libboost-*``, and they are instead installed into ``/opt/ceph``, so | |
156 | they don't interfere with the official ``libboost`` packages shipped by | |
2a845540 TL |
157 | distro. Its build scripts are hosted at https://github.com/ceph/ceph-boost. |
158 | See https://github.com/ceph/ceph-boost/commit/2a8ae02932b2a1fd6a68072da8ca0df2b99b805c | |
159 | for an example of how to bump the version number. The commands used to | |
160 | build 1.79 on a vanilla Ubuntu Focal OS are below. | |
20effc67 TL |
161 | |
162 | .. prompt:: bash $ | |
163 | ||
2a845540 TL |
164 | sudo apt install debhelper dctrl-tools chrpath libbz2-dev libicu-dev bison \ |
165 | flex docbook-to-man help2man xsltproc doxygen dh-python python3-all-dev graphviz | |
166 | wget http://download.ceph.com/qa/boost_1_79_0.tar.bz2 | |
20effc67 | 167 | git clone https://github.com/ceph/ceph-boost |
2a845540 TL |
168 | tar xjf boost_1_79_0.tar.bz2 |
169 | cp -ra ceph-boost/debian boost_1_79_0/ | |
170 | pushd boost_1_79_0 | |
20effc67 TL |
171 | export DEB_BUILD_OPTIONS='parallel=6 nodoc' |
172 | dpkg-buildpackage -us -uc -b | |
2a845540 TL |
173 | popd |
174 | BOOST_SHA=$(git ls-remote https://github.com/ceph/ceph-boost main | awk '{ print $1 }') | |
175 | ls *.deb | chacractl binary create \ | |
176 | libboost/master/$BOOST_SHA/ubuntu/focal/amd64/flavors/default | |
20effc67 TL |
177 | |
178 | libzbd | |
179 | packages `libzbd`_ . The upstream libzbd includes debian packaging already. | |
180 | ||
181 | libpmem | |
182 | packages `pmdk`_ . Please note, ``ndctl`` is one of the build dependencies of | |
183 | pmdk, for an updated debian packaging, please see | |
184 | https://github.com/ceph/ceph-ndctl . | |
185 | ||
186 | .. note:: | |
187 | ||
188 | please ensure that the package version and the release number of the | |
189 | packaging are properly updated when updating/upgrading the packaging, | |
190 | otherwise it would be difficult to tell which version of the package | |
191 | is installed. We check the package version before trying to upgrade | |
192 | it in ``install-deps.sh``. | |
193 | ||
194 | .. _boost: https://www.boost.org | |
195 | .. _libzbd: https://github.com/westerndigitalcorporation/libzbd | |
196 | .. _pmdk: https://github.com/pmem/pmdk | |
197 | ||
198 | But in addition to these libraries, ``ceph-mgr-dashboard``'s frontend uses lots of | |
199 | JavaScript packages. Quite a few of them are not packaged by distros. Not to | |
200 | mention the trouble of testing different combination of versions of these | |
201 | packages. So we decided to include these JavaScript packages in our dist tarball | |
202 | using ``make-dist``. | |
203 | ||
204 | Also, because our downstream might not want to use the prepackaged binaries when | |
205 | redistributing the precompiled Ceph packages, we also need to include these | |
206 | libraries in our dist tarball. They are | |
207 | ||
208 | - boost | |
209 | - liburing | |
210 | - pmdk | |
211 | ||
212 | ``make-dist`` is a script used by our CI pipeline to create dist tarball so the | |
213 | tarball can be used to build the Ceph packages in a clean room environment. When | |
214 | we need to upgrade these third party libraries, we should | |
215 | ||
216 | - update the CMake script | |
217 | - rebuild the prebuilt packages and | |
218 | - update this script to reflect the change. | |
219 | ||
220 | Uploading Dependencies | |
221 | ---------------------- | |
222 | ||
223 | To ensure that prebuilt packages are available by the jenkins agents, we need to | |
224 | upload them to either ``apt-mirror.front.sepia.ceph.com`` or `chacra`_. To upload | |
225 | packages to the former would require the help our our lab administrator, so if we | |
226 | want to maintain the package repositories on regular basis, a better choice would be | |
227 | to manage them using `chacractl`_. `chacra`_ represents packages repositories using | |
228 | a resource hierarchy, like:: | |
229 | ||
230 | <project>/<branch>/<ref>/<distro>/<distro-version>/<arch> | |
231 | ||
232 | In which: | |
233 | ||
234 | project | |
235 | in general, it is used for denoting a set of related packages. For instance, | |
236 | ``libboost``. | |
237 | ||
238 | branch | |
239 | branch of project. This mirrors the concept of a Git repo. | |
240 | ||
241 | ref | |
242 | a unique id of a given version of a set packages. This id is used to reference | |
243 | the set packages under the ``<project>/<branch>``. It is a good practice to | |
244 | version the packaging recipes, like the ``debian`` directory for building deb | |
245 | packages and the ``spec`` for building rpm packages, and use the sha1 of the | |
246 | packaging receipe for the ``ref``. But you could also use a random string for | |
247 | ``ref``, like the tag name of the built source tree. | |
248 | ||
249 | distro | |
250 | the distro name for which the packages are built. Currently, following distros are | |
251 | supported: | |
252 | ||
253 | - centos | |
254 | - debian | |
255 | - fedora | |
256 | - rhel | |
257 | - ubuntu | |
258 | ||
259 | distro-version | |
260 | the version of the distro. For instance, if a package is built on ubuntu focal, | |
261 | the ``distro-version`` should be ``20.04``. | |
262 | ||
263 | arch | |
264 | the architecture of the packages. It could be: | |
265 | ||
266 | - arm64 | |
267 | - amd64 | |
268 | - noarch | |
269 | ||
270 | So, for example, we can upload the prebuilt boost packages to chacra like | |
271 | ||
272 | .. prompt:: bash $ | |
273 | ||
274 | ls *.deb | chacractl binary create \ | |
275 | libboost/master/099c0fd56b4a54457e288a2eff8fffdc0d416f7a/ubuntu/focal/amd64/flavors/default | |
276 | ||
277 | .. _chacra: https://github.com/ceph/chacra | |
278 | .. _chacractl: https://github.com/ceph/chacractl | |
279 | ||
280 | Update ``install-deps.sh`` | |
281 | -------------------------- | |
282 | ||
283 | We also need to update ``install-deps.sh`` to point the built script to the new | |
284 | repo. Please refer to the `script <https://github.com/ceph/ceph/blob/master/install-deps.sh>`_, | |
285 | for more details. |