ceph/doc/dev/continuous-integration.rst

   1 Continuous Integration Architecture
   2 ===================================
   3
   4 In Ceph, we rely on multiple CI pipelines in our development. Most of these pipelines
   5 are centered around Jenkins. And their configurations are generated using `Jenkins Job Builder`_.
   6
   7 .. _Jenkins Job Builder: https://docs.openstack.org/infra/jenkins-job-builder/
   8
   9 Let's take the ``make check`` performed by Jenkins as an example.
  10
  11 ceph-pull-requests
  12 ------------------
  13
  14 ``ceph-pull-requests`` is a jenkins job which gets triggered by a GitHub pull
  15 request or a trigger phrase like::
  16
  17     jenkins test make check
  18
  19 There are multiple parties involved in this jenkins job:
  20
  21 .. graphviz::
  22
  23    digraph  {
  24      rankdir="LR";
  25      github [
  26       label="<git> git_repo | <webhooks> webhooks | <api> api";
  27       shape=record;
  28       href="https://github.com/ceph/ceph";
  29      ];
  30      subgraph cluster_lab {
  31        label="Sepia Lab";
  32        href="https://wiki.sepia.ceph.com/doku.php";
  33        shape=circle;
  34        apt_mirror [
  35          href="http://apt-mirror.front.sepia.ceph.com";
  36        ];
  37        shaman [
  38          href="https://shaman.ceph.com";
  39        ];
  40        chacra [
  41          peripheries=3;
  42          href="https://chacra.ceph.com";
  43        ];
  44        subgraph cluster_jenkins {
  45          label="jenkins";
  46          href="https://jenkins.ceph.com";
  47          jenkins_controller [ label = "controller" ];
  48          jenkins_agents [ label = "agents", peripheries=3 ];
  49        };
  50      };
  51      {
  52        rank=same;
  53        package_repos [ peripheries=3 ];
  54        pypi;
  55        npm;
  56      }
  57      github:webhooks -> jenkins_controller [ label = "notify", color = "crimson" ];
  58      jenkins_controller -> jenkins_agents [ label = "schedule jobs" ];
  59      jenkins_agents -> github:git [ label = "git pull" ];
  60      jenkins_agents -> shaman [ label = "query for chacra repo URL" ];
  61      jenkins_agents -> chacra [ label = "pull build dependencies" ];
  62      jenkins_agents -> package_repos [ label = "pull build dependencies" ];
  63      jenkins_agents -> pypi [ label = "pull Python packages" ];
  64      jenkins_agents -> npm [ label = "pull JavaScript packages" ];
  65      jenkins_agents -> apt_mirror [ label = "pull build dependencies" ];
  66      jenkins_agents -> github:api [ label = "update", color = "crimson" ];
  67    }
  68
  69 Where
  70
  71 Sepia Lab
  72    `Sepia Lab`_ is a test lab used by the Ceph project. This lab offers
  73    the storage and computing resources required by our CI infra.
  74
  75 Jenkins agents
  76    are a set of machines which perform the CI jobs. In this case, they
  77
  78    #. pull the git repo from GitHub and
  79    #. rebase the pull request against the latest master
  80    #. set necessary environment variables
  81    #. run ``run-make-check.sh``
  82
  83 Chacra
  84    is a server offering RESTful API allowing the clients to store and
  85    retrieve binary packages. It also creates the repo for uploaded
  86    packages automatically. Once a certain repo is created on chacra, the
  87    configured shaman server is updated as well, then we can query shaman
  88    for the corresponding repo address. Chacra not only hosts Ceph packages,
  89    it also hosts quite a few other packages like various build dependencies.
  90
  91 Shaman
  92    is a server offering RESTful API allowing the clients to query the
  93    information of repos hosted by chacra nodes. Shaman is also known
  94    for its `Web UI`_. But please note, shaman does not build the
  95    packages, it just offers information on the builds.
  96
  97 As the following shows, `chacra`_ manages multiple projects whose metadata
  98 are stored in a database. These metadata are exposed via Shaman as a web
  99 service. `chacractl`_ is a utility to interact with the `chacra`_ service.
 100
 101 .. graphviz::
 102
 103    digraph  {
 104      libboost [
 105        shape=cylinder;
 106      ];
 107      libzbd [
 108        shape=cylinder;
 109      ];
 110      other_repos [
 111        label="...";
 112        shape=cylinder;
 113      ];
 114      postgresql [
 115        shape=cylinder;
 116        style=filled;
 117      ]
 118      shaman -> postgresql;
 119      chacra -> postgresql;
 120      chacractl -> chacra;
 121      chacra -> libboost;
 122      chacra -> libzbd;
 123      chacra -> other_repos;
 124    }
 125
 126 .. _Sepia Lab: https://wiki.sepia.ceph.com/doku.php
 127 .. _Web UI: https://shaman.ceph.com
 128
 129 build dependencies
 130 ------------------
 131
 132 Just like lots of other software projects, Ceph has both build-time and
 133 run-time dependencies. Most of time, we are inclined to use the packages
 134 prebuilt by the distro. But there are cases where
 135
 136 - the necessary dependencies are either missing in the distro, or
 137 - their versions are too old, or
 138 - they are packaged without some important feature enabled.
 139 - we want to ensure that the version of a certain runtime dependency is
 140   identical to the one we tested in our lab.
 141
 142 No matter what the reason is, we either need to build them from source, or
 143 to package them as binary packages instead of using the ones shipped by the
 144 distro. Quite a few build-time dependencies are included as git submodules,
 145 but in order to avoid rebuilding these dependencies repeatedly, we pre-built
 146 some of them and uploaded them to our own repos. So, when performing
 147 ``make check``, the building hosts in our CI just pull them from our internal
 148 repos hosting these packages instead of building them.
 149
 150 So far, following packages are prebuilt for ubuntu focal, and then uploaded to
 151 `chacra`_:
 152
 153 libboost
 154     packages `boost`_. The packages' names are changed from ``libboost-*`` to
 155     ``ceph-libboost-*``, and they are instead installed into ``/opt/ceph``, so
 156     they don't interfere with the official ``libboost`` packages shipped by
 157     distro. Its build scripts are hosted at https://github.com/ceph/ceph-boost.
 158     See https://github.com/ceph/ceph-boost/commit/2a8ae02932b2a1fd6a68072da8ca0df2b99b805c
 159     for an example of how to bump the version number. The commands used to
 160     build 1.79 on a vanilla Ubuntu Focal OS are below.
 161
 162     .. prompt:: bash $
 163
 164        sudo apt install debhelper dctrl-tools chrpath libbz2-dev libicu-dev bison \
 165          flex docbook-to-man help2man xsltproc doxygen dh-python python3-all-dev graphviz
 166        wget http://download.ceph.com/qa/boost_1_79_0.tar.bz2
 167        git clone https://github.com/ceph/ceph-boost
 168        tar xjf boost_1_79_0.tar.bz2
 169        cp -ra ceph-boost/debian boost_1_79_0/
 170        pushd boost_1_79_0
 171        export DEB_BUILD_OPTIONS='parallel=6 nodoc'
 172        dpkg-buildpackage -us -uc -b
 173        popd
 174        BOOST_SHA=$(git ls-remote https://github.com/ceph/ceph-boost main | awk '{ print $1 }')
 175        ls *.deb | chacractl binary create \
 176          libboost/master/$BOOST_SHA/ubuntu/focal/amd64/flavors/default
 177
 178 libzbd
 179     packages `libzbd`_ . The upstream libzbd includes debian packaging already.
 180
 181 libpmem
 182     packages `pmdk`_ . Please note, ``ndctl`` is one of the build dependencies of
 183     pmdk, for an updated debian packaging, please see
 184     https://github.com/ceph/ceph-ndctl .
 185
 186 .. note::
 187
 188    please ensure that the package version and the release number of the
 189    packaging are properly updated when updating/upgrading the packaging,
 190    otherwise it would be difficult to tell which version of the package
 191    is installed. We check the package version before trying to upgrade
 192    it in ``install-deps.sh``.
 193
 194 .. _boost: https://www.boost.org
 195 .. _libzbd: https://github.com/westerndigitalcorporation/libzbd
 196 .. _pmdk: https://github.com/pmem/pmdk
 197
 198 But in addition to these libraries, ``ceph-mgr-dashboard``'s frontend uses lots of
 199 JavaScript packages. Quite a few of them are not packaged by distros. Not to
 200 mention the trouble of testing different combination of versions of these
 201 packages. So we decided to include these JavaScript packages in our dist tarball
 202 using ``make-dist``.
 203
 204 Also, because our downstream might not want to use the prepackaged binaries when
 205 redistributing the precompiled Ceph packages, we also need to include these
 206 libraries in our dist tarball. They are
 207
 208 - boost
 209 - liburing
 210 - pmdk
 211
 212 ``make-dist`` is a script used by our CI pipeline to create dist tarball so the
 213 tarball can be used to build the Ceph packages in a clean room environment. When
 214 we need to upgrade these third party libraries, we should
 215
 216 - update the CMake script
 217 - rebuild the prebuilt packages and
 218 - update this script to reflect the change.
 219
 220 Uploading Dependencies
 221 ----------------------
 222
 223 To ensure that prebuilt packages are available by the jenkins agents, we need to
 224 upload them to either ``apt-mirror.front.sepia.ceph.com`` or `chacra`_. To upload
 225 packages to the former would require the help our our lab administrator, so if we
 226 want to maintain the package repositories on regular basis, a better choice would be
 227 to manage them using `chacractl`_. `chacra`_ represents packages repositories using
 228 a resource hierarchy, like::
 229
 230   <project>/<branch>/<ref>/<distro>/<distro-version>/<arch>
 231
 232 In which:
 233
 234 project
 235     in general, it is used for denoting a set of related packages. For instance,
 236     ``libboost``.
 237
 238 branch
 239     branch of project. This mirrors the concept of a Git repo.
 240
 241 ref
 242     a unique id of a given version of a set packages. This id is used to reference
 243     the set packages under the ``<project>/<branch>``. It is a good practice to
 244     version the packaging recipes, like the ``debian`` directory for building deb
 245     packages and the ``spec`` for building rpm packages, and use the sha1 of the
 246     packaging receipe for the ``ref``. But you could also use a random string for
 247     ``ref``, like the tag name of the built source tree.
 248
 249 distro
 250     the distro name for which the packages are built. Currently, following distros are
 251     supported:
 252
 253     - centos
 254     - debian
 255     - fedora
 256     - rhel
 257     - ubuntu
 258
 259 distro-version
 260     the version of the distro. For instance, if a package is built on ubuntu focal,
 261     the ``distro-version`` should be ``20.04``.
 262
 263 arch
 264     the architecture of the packages. It could be:
 265
 266     - arm64
 267     - amd64
 268     - noarch
 269
 270 So, for example, we can upload the prebuilt boost packages to chacra like
 271
 272 .. prompt:: bash $
 273
 274    ls *.deb | chacractl binary create \
 275      libboost/master/099c0fd56b4a54457e288a2eff8fffdc0d416f7a/ubuntu/focal/amd64/flavors/default
 276
 277 .. _chacra: https://github.com/ceph/chacra
 278 .. _chacractl: https://github.com/ceph/chacractl
 279
 280 Update ``install-deps.sh``
 281 --------------------------
 282
 283 We also need to update ``install-deps.sh`` to point the built script to the new
 284 repo. Please refer to the `script <https://github.com/ceph/ceph/blob/master/install-deps.sh>`_,
 285 for more details.