ceph/doc/dev/release-process.rst

   1 ======================
   2   Ceph Release Process
   3 ======================
   4
   5 Prerequisites
   6 =============
   7
   8 Signing Machine
   9 ---------------
  10 The signing machine is a virtual machine in the `Sepia lab
  11 <https://wiki.sepia.ceph.com/doku.php?id=start>`_. SSH access to the signing
  12 machine is limited to the usual Infrastructure Admins along with a few other
  13 component leads (e.g., nfs-ganesha, ceph-iscsi).
  14
  15 The ``ubuntu`` user on the machine has some `build scripts <https://github.com/ceph/ceph-build/tree/main/scripts>`_ that help with pulling, pushing, and signing packages.
  16
  17 The GPG signing key permanently lives on a `Nitrokey Pro <https://shop.nitrokey.com/shop/product/nkpr2-nitrokey-pro-2-3>`_ and is passed through to the VM via RHV. This helps to ensure that the key cannot be exported or leave the datacenter in any way.
  18
  19 New Major Releases
  20 ------------------
  21 For each new major (alphabetical) release, you must create one ``ceph-release`` RPM for each RPM repo (e.g., one for el8 and one for el9). `chacra <https://github.com/ceph/chacra>`_ is a python service we use to store DEB and RPM repos. The chacra repos are configured to include this ceph-release RPM, but it must be built separately. You must make sure that chacra is properly configured to include this RPM for each particular release.
  22
  23 1. Update chacra so it is aware of the new Ceph release.  See `this PR <https://github.com/ceph/chacra/pull/219>`_ for an example.
  24 2. Redeploy chacra (e.g., ``ansible-playbook chacra.ceph.com.yml``)
  25 3. Run https://jenkins.ceph.com/view/all/job/ceph-release-rpm/
  26
  27 Summarized build process
  28 ========================
  29
  30 1. QE finishes testing and finds a stopping point.  That commit is pushed to the ``$release-release`` branch in ceph.git (e.g., ``quincy-release``).  This allows work to continue in the working ``$release`` branch without having to freeze it during the release process.
  31 2. The Ceph Council approves and notifies the "Build Lead".
  32 3. The "Build Lead" starts the `Jenkins multijob <https://jenkins.ceph.com/view/all/job/ceph>`_, which triggers all builds.
  33 4. Packages are pushed to chacra.ceph.com.
  34 5. Packages are pulled from chacra.ceph.com to the Signer VM.
  35 6. Packages are signed.
  36 7. Packages are pushed to download.ceph.com.
  37 8. Release containers are built and pushed to quay.io.
  38
  39 Hotfix Release Process Deviation
  40 --------------------------------
  41
  42 A hotfix release has a couple differences.
  43
  44 1. Check out the most recent tag. For example, if we're releasing a hotfix on top of 17.2.3, ``git checkout -f -B quincy-release origin/v17.2.3``
  45 2. ``git cherry-pick -x`` the necessary hotfix commits
  46 3. ``git push -f origin quincy-release``
  47 4. Notify the "Build Lead" to start the build.
  48 5. The "Build Lead" should set ``RELEASE_TYPE=HOTFIX`` instead of ``STABLE``.
  49
  50 Security Release Process Deviation
  51 ----------------------------------
  52
  53 A security/CVE release is similar to a hotfix release with two differences:
  54
  55     1. The fix should be pushed to the `ceph-private <https://github.com/ceph/ceph-private>`_ repo instead of ceph.git (requires GitHub Admin Role).
  56     2. The tags (e.g., v17.2.4) must be manually pushed to ceph.git by the "Build Lead."
  57
  58 1. Check out the most recent tag. For example, if we're releasing a security fix on top of 17.2.3, ``git checkout -f -B quincy-release origin/v17.2.3``
  59 2. ``git cherry-pick -x`` the necessary security fix commits
  60 3. ``git remote add security git@github.com:ceph/ceph-private.git``
  61 4. ``git push -f security quincy-release``
  62 5. Notify the "Build Lead" to start the build.
  63 6. The "Build Lead" should set ``RELEASE_TYPE=SECURITY`` instead of ``STABLE``.
  64 7. Finally, the `ceph-tag <https://github.com/ceph/ceph-build/blob/main/ansible/roles/ceph-release/tasks/push.yml>`_ steps need to be manually run by the "Build Lead" as close to the Announcement time as possible::
  65
  66     # Example using quincy pretending 17.2.4 is the security release version
  67     # Add the ceph-releases repo (also requires GitHub Admin Role). The `ceph-setup <https://jenkins.ceph.com/job/ceph-setup>`_ job will have already created and pushed the tag to ceph-releases.git.
  68     git remote add releases git@github.com:ceph/ceph-releases.git
  69     git fetch --all
  70     # Check out the version commit
  71     git checkout -f -B quincy-release releases/quincy-release
  72     git push -f origin quincy-release
  73     git push origin v17.2.4
  74     # Now create a Pull Request of quincy-release targeting quincy to merge the version commit and security fixes back into the quincy branch
  75
  76 1. Preparing the release branch
  77 ===============================
  78
  79 Once QE has determined a stopping point in the working (e.g., ``quincy``) branch, that commit should be pushed to the corresponding ``quincy-release`` branch.
  80
  81 Notify the "Build Lead" that the release branch is ready.
  82
  83 2. Starting the build
  84 =====================
  85
  86 We'll use a stable/regular 15.2.17 release of Octopus as an example throughout this document.
  87
  88 1. Browse to https://jenkins.ceph.com/view/all/job/ceph/build?delay=0sec
  89 2. Log in with GitHub OAuth
  90 3. Set the parameters as necessary::
  91
  92     BRANCH=octopus
  93     TAG=checked
  94     VERSION=15.2.17
  95     RELEASE_TYPE=STABLE
  96     ARCHS=x86_64 arm64
  97
  98 4. Use https://docs.ceph.com/en/latest/start/os-recommendations/?highlight=debian#platforms to determine the ``DISTROS`` parameter.  For example,
  99
 100     +-------------------+-------------------------------------------+
 101     | Release           | Distro Codemap                            |
 102     +===================+===========================================+
 103     | octopus (15.X.X)  | ``focal bionic centos7 centos8 buster``   |
 104     +-------------------+-------------------------------------------+
 105     | pacific (16.X.X)  | ``focal bionic centos8 buster bullseye``  |
 106     +-------------------+-------------------------------------------+
 107     | quincy (17.X.X)   | ``focal centos8 centos9 bullseye``        |
 108     +-------------------+-------------------------------------------+
 109
 110 5. Click ``Build``.
 111
 112 3. Release Notes
 113 ================
 114
 115 Packages take hours to build. Use those hours to create the Release Notes and Announcements:
 116
 117 1. ceph.git Release Notes (e.g., `v15.2.17's ceph.git (docs.ceph.com) PR <https://github.com/ceph/ceph/pull/47198>`_)
 118 2. ceph.io Release Notes (e.g., `v15.2.17's ceph.io.git (www.ceph.io) PR <https://github.com/ceph/ceph.io/pull/427>`_)
 119 3. E-mail announcement
 120
 121 See `the Ceph Tracker wiki page that explains how to write the release notes <https://tracker.ceph.com/projects/ceph-releases/wiki/HOWTO_write_the_release_notes>`_.
 122
 123 4. Signing and Publishing the Build
 124 ===================================
 125
 126 #. Obtain the sha1 of the version commit from the `build job <https://jenkins.ceph.com/view/all/job/ceph>`_ or the ``sha1`` file created by the `ceph-setup <https://jenkins.ceph.com/job/ceph-setup/>`_ job.
 127
 128 #. Download the packages from chacra.ceph.com to the signing virtual machine. These packages get downloaded to ``/opt/repos`` where the `Sepia Lab Long Running (Ceph) Cluster <https://wiki.sepia.ceph.com/doku.php?id=services:longrunningcluster>`_ is mounted.
 129
 130    .. prompt:: bash $
 131
 132       ssh ubuntu@signer.front.sepia.ceph.com
 133       sync-pull ceph [pacific|quincy|etc] <sha1>
 134
 135    Example::
 136
 137       $ sync-pull ceph octopus 8a82819d84cf884bd39c17e3236e0632ac146dc4
 138       sync for: ceph octopus
 139       ********************************************
 140       Found the most packages (332) in ubuntu/bionic.
 141       No JSON object could be decoded
 142       No JSON object could be decoded
 143       ubuntu@chacra.ceph.com:/opt/repos/ceph/octopus/8a82819d84cf884bd39c17e3236e0632ac146dc4/ubuntu/bionic/flavors/default/* /opt/repos/ceph/octopus-15.2.17/debian/jessie/
 144       --------------------------------------------
 145       receiving incremental file list
 146       db/
 147        db/checksums.db
 148               180.22K 100%    2.23MB/s    0:00:00 (xfr#1, to-chk=463/467)
 149       db/contents.cache.db
 150               507.90K 100%    1.95MB/s    0:00:00 (xfr#2, to-chk=462/467)
 151       db/packages.db
 152
 153       etc...
 154
 155 #. Sign the DEBs:
 156
 157    .. prompt:: bash
 158
 159       merfi gpg /opt/repos/ceph/octopus-15.2.17/debian
 160
 161    Example::
 162
 163       $ merfi gpg /opt/repos/ceph/octopus-15.2.17/debian
 164       --> Starting path collection, looking for files to sign
 165       --> 18 matching paths found
 166       --> will sign with the following commands:
 167       --> gpg --batch --yes --armor --detach-sig --output Release.gpg Release
 168       --> gpg --batch --yes --clearsign --output InRelease Release
 169       --> signing: /opt/repos/ceph/octopus-15.2.17/debian/jessie/dists/bionic/Release
 170       --> Running command: gpg --batch --yes --armor --detach-sig --output Release.gpg Release
 171       --> Running command: gpg --batch --yes --clearsign --output InRelease Release
 172       --> signing: /opt/repos/ceph/octopus-15.2.17/debian/jessie/dists/focal/Release
 173       --> Running command: gpg --batch --yes --armor --detach-sig --output Release.gpg Release
 174       --> Running command: gpg --batch --yes --clearsign --output InRelease Release
 175
 176       etc...
 177
 178 #. Sign the RPMs:
 179
 180    .. prompt:: bash
 181
 182       sign-rpms octopus
 183
 184    Example::
 185
 186       $ sign-rpms octopus
 187       Checking packages in: /opt/repos/ceph/octopus-15.2.17/centos/7
 188       signing:  /opt/repos/ceph/octopus-15.2.17/centos/7/SRPMS/ceph-release-1-1.el7.src.rpm
 189       /opt/repos/ceph/octopus-15.2.17/centos/7/SRPMS/ceph-release-1-1.el7.src.rpm:
 190       signing:  /opt/repos/ceph/octopus-15.2.17/centos/7/SRPMS/ceph-15.2.17-0.el7.src.rpm
 191       /opt/repos/ceph/octopus-15.2.17/centos/7/SRPMS/ceph-15.2.17-0.el7.src.rpm:
 192       signing:  /opt/repos/ceph/octopus-15.2.17/centos/7/noarch/ceph-mgr-modules-core-15.2.17-0.el7.noarch.rpm
 193
 194       etc...
 195
 196 5. Publish the packages to download.ceph.com:
 197
 198    .. prompt:: bash $
 199
 200       sync-push octopus
 201
 202 5. Build Containers
 203 ===================
 204
 205 Start the following two jobs:
 206
 207 #. https://2.jenkins.ceph.com/job/ceph-container-build-ceph-base-push-imgs/
 208 #. https://2.jenkins.ceph.com/job/ceph-container-build-ceph-base-push-imgs-arm64/
 209
 210 6. Announce the Release
 211 =======================
 212
 213 Version Commit PR
 214 -----------------
 215
 216 The `ceph-tag Jenkins job <https://jenkins.ceph.com/job/ceph-tag>`_ creates a Pull Request in ceph.git that targets the release branch.
 217
 218 If this was a regular release (not a hotfix release or a security release), the only commit in that Pull Request should be the version commit.  For example, see `v15.2.17's version commit PR <https://github.com/ceph/ceph/pull/47520>`_.
 219
 220 Request a review and then merge the Pull Request.
 221
 222 Announcing
 223 ----------
 224
 225 Publish the Release Notes on ceph.io before announcing the release by email, because the e-mail announcement references the ceph.io blog post.