]>
Commit | Line | Data |
---|---|---|
7c673cae FG |
1 | ===================== |
2 | Development workflows | |
3 | ===================== | |
4 | ||
5 | This page explains the workflows a developer is expected to follow to | |
6 | implement the goals that are part of the Ceph release cycle. It does not | |
7 | go into technical details and is designed to provide a high level view | |
8 | instead. Each chapter is about a given goal such as ``Merging bug | |
9 | fixes or features`` or ``Publishing point releases and backporting``. | |
10 | ||
11 | A key aspect of all workflows is that none of them blocks another. For | |
12 | instance, a bug fix can be backported and merged to a stable branch | |
13 | while the next point release is being published. For that specific | |
14 | example to work, a branch should be created to avoid any | |
15 | interference. In practice it is not necessary for Ceph because: | |
16 | ||
17 | * there are few people involved | |
18 | * the frequency of backports is not too high | |
19 | * the reviewers, who know a release is being published, are unlikely | |
20 | to merge anything that may cause issues | |
21 | ||
22 | This ad-hoc approach implies the workflows are changed on a regular | |
23 | basis to adapt. For instance, ``quality engineers`` were not involved | |
24 | in the workflow to publish ``dumpling`` point releases. The number of | |
25 | commits being backported to ``firefly`` made it impractical for developers | |
26 | tasked to write code or fix bugs to also run and verify the full suite | |
27 | of integration tests. Inserting ``quality engineers`` makes it | |
28 | possible for someone to participate in the workflow by analyzing test | |
29 | results. | |
30 | ||
31 | The workflows are not enforced when they impose an overhead that does | |
32 | not make sense. For instance, if the release notes for a point release | |
33 | were not written prior to checking all integration tests, they can be | |
11fdf7f2 | 34 | committed to the stable branch and the result sent for publication |
7c673cae FG |
35 | without going through another run of integration tests. |
36 | ||
37 | Release Cycle | |
38 | ============= | |
39 | ||
40 | :: | |
41 | ||
42 | Ceph hammer infernalis | |
43 | Developer CDS CDS | |
44 | Summit | | | |
45 | | | | |
46 | development | | | |
47 | release | v0.88 v0.89 v0.90 ... | v9.0.0 | |
48 | --v--^----^--v---^------^--v- ---v----^----^--- 2015 | |
49 | | | | | | |
50 | stable giant | | hammer | |
51 | release v0.87 | | v0.94 | |
52 | | | | |
53 | point firefly dumpling | |
54 | release v0.80.8 v0.67.12 | |
55 | ||
56 | ||
57 | Four times a year, the development roadmap is discussed online during | |
31f18b77 | 58 | the `Ceph Developer Summit <http://tracker.ceph.com/projects/ceph/wiki/Planning#Ceph-Developer-Summit>`_. A |
7c673cae FG |
59 | new stable release (hammer, infernalis, jewel ...) is published at the same |
60 | frequency. Every other release (firefly, hammer, jewel...) is a `Long Term | |
61 | Stable (LTS) <../../releases>`_. See `Understanding the release cycle | |
62 | <../../releases#understanding-the-release-cycle>`_ for more information. | |
63 | ||
64 | Merging bug fixes or features | |
65 | ============================= | |
66 | ||
67 | The development branch is ``master`` and the workflow followed by all | |
68 | developers can be summarized as follows: | |
69 | ||
70 | * The developer prepares a series of commits | |
71 | * The developer submits the series of commits via a pull request | |
72 | * A reviewer is assigned the pull request | |
73 | * When the pull request looks good to the reviewer, it is merged into | |
74 | an integration branch by the tester | |
75 | * After a successful run of integration tests, the pull request is | |
76 | merged by the tester | |
77 | ||
78 | The ``developer`` is the author of a series of commits. The | |
79 | ``reviewer`` is responsible for providing feedback to the developer on | |
80 | a regular basis and the developer is invited to ping the reviewer if | |
81 | nothing happened after a week. After the ``reviewer`` is satisfied | |
82 | with the pull request, (s)he passes it to the ``tester``. The | |
83 | ``tester`` is responsible for running teuthology integration tests on | |
84 | the pull request. If nothing happens within a month the ``reviewer`` is | |
85 | invited to ping the ``tester``. | |
86 | ||
87 | Resolving bug reports and implementing features | |
88 | =============================================== | |
89 | ||
90 | All bug reports and feature requests are in the `issue tracker | |
91 | <http://tracker.ceph.com>`_ and the workflow can be summarized as | |
92 | follows: | |
93 | ||
94 | * The reporter creates the issue with priority ``Normal`` | |
95 | * A developer may pick the issue right away | |
96 | * During a bi-weekly bug scrub, the team goes over all new issue and | |
97 | assign them a priority | |
98 | * The bugs with higher priority are worked on first | |
99 | ||
11fdf7f2 | 100 | Each ``team`` is responsible for a project, managed by :ref:`leads <governance>`. |
7c673cae FG |
101 | |
102 | The ``developer`` assigned to an issue is responsible for it. The | |
103 | status of an open issue can be: | |
104 | ||
105 | * ``New``: it is unclear if the issue needs work. | |
106 | * ``Verified``: the bug can be reproduced or showed up multiple times | |
107 | * ``In Progress``: the developer is working on it this week | |
108 | * ``Pending Backport``: the fix needs to be backported to the stable | |
109 | releases listed in the backport field | |
110 | ||
111 | For each ``Pending Backport`` issue, there exists at least one issue | |
112 | in the ``Backport`` tracker to record the work done to cherry pick the | |
113 | necessary commits from the master branch to the target stable branch. | |
114 | See `the backporter manual | |
115 | <http://tracker.ceph.com/projects/ceph-releases/wiki/HOWTO>`_ for more | |
116 | information. | |
117 | ||
118 | Running and interpreting teuthology integration tests | |
119 | ===================================================== | |
120 | ||
121 | The :doc:`/dev/sepia` runs `teuthology | |
122 | <https://github.com/ceph/teuthology/>`_ integration tests `on a regular basis <http://tracker.ceph.com/projects/ceph-releases/wiki/HOWTO_monitor_the_automated_tests_AKA_nightlies#Automated-tests-AKA-nightlies>`_ and the | |
123 | results are posted on `pulpito <http://pulpito.ceph.com/>`_ and the | |
31f18b77 | 124 | `ceph-qa mailing list <https://ceph.com/irc/>`_. |
7c673cae FG |
125 | |
126 | * The job failures are `analyzed by quality engineers and developers | |
127 | <http://tracker.ceph.com/projects/ceph-releases/wiki/HOWTO_monitor_the_automated_tests_AKA_nightlies#List-of-suites-and-watchers>`_ | |
128 | * If the cause is environmental (e.g. network connectivity), an issue | |
129 | is created in the `sepia lab project | |
130 | <http://tracker.ceph.com/projects/lab/issues/new>`_ | |
131 | * If the bug is known, a pulpito URL to the failed job is added to the issue | |
132 | * If the bug is new, an issue is created | |
133 | ||
134 | The ``quality engineer`` is either a developer or a member of the QE | |
135 | team. There is at least one integration test suite per project: | |
136 | ||
137 | * `rgw <https://github.com/ceph/ceph/tree/master/qa/suites/rgw>`_ suite | |
138 | * `CephFS <https://github.com/ceph/ceph/tree/master/qa/suites/fs>`_ suite | |
139 | * `rados <https://github.com/ceph/ceph/tree/master/qa/suites/rados>`_ suite | |
140 | * `rbd <https://github.com/ceph/ceph/tree/master/qa/suites/rbd>`_ suite | |
141 | ||
142 | and many others such as | |
143 | ||
144 | * `upgrade <https://github.com/ceph/ceph/tree/master/qa/suites/upgrade>`_ suites | |
145 | * `power-cyle <https://github.com/ceph/ceph/tree/master/qa/suites/powercycle>`_ suite | |
146 | * ... | |
147 | ||
148 | Preparing a new release | |
149 | ======================= | |
150 | ||
151 | A release is prepared in a dedicated branch, different from the | |
152 | ``master`` branch. | |
153 | ||
154 | * For a stable releases it is the branch matching the release code | |
155 | name (dumpling, firefly, etc.) | |
156 | * For a development release it is the ``next`` branch | |
157 | ||
158 | The workflow expected of all developers to stabilize the release | |
159 | candidate is the same as the normal development workflow with the | |
160 | following differences: | |
161 | ||
162 | * The pull requests must target the stable branch or next instead of | |
163 | master | |
164 | * The reviewer rejects pull requests that are not bug fixes | |
165 | * The ``Backport`` issues matching a teuthology test failure and set | |
166 | with priority ``Urgent`` must be fixed before the release | |
167 | ||
168 | Cutting a new stable release | |
169 | ============================ | |
170 | ||
171 | A new stable release can be cut when: | |
172 | ||
173 | * all ``Backport`` issues with priority ``Urgent`` are fixed | |
174 | * integration and upgrade tests run successfully | |
175 | ||
176 | Publishing a new stable release implies a risk of regression or | |
177 | discovering new bugs during the upgrade, no matter how carefully it is | |
178 | tested. The decision to cut a release must take this into account: it | |
179 | may not be wise to publish a stable release that only fixes a few | |
180 | minor bugs. For instance if only one commit has been backported to a | |
181 | stable release that is not a LTS, it is better to wait until there are | |
182 | more. | |
183 | ||
184 | When a stable release is to be retired, it may be safer to | |
185 | recommend an upgrade to the next LTS release instead of | |
186 | proposing a new point release to fix a problem. For instance, the | |
187 | ``dumpling`` v0.67.11 release has bugs related to backfilling which have | |
188 | been fixed in ``firefly`` v0.80.x. A backport fixing these backfilling | |
189 | bugs has been tested in the draft point release ``dumpling`` v0.67.12 but | |
190 | they are large enough to introduce a risk of regression. As ``dumpling`` | |
191 | is to be retired, users suffering from this bug can | |
192 | upgrade to ``firefly`` to fix it. Unless users manifest themselves and ask | |
193 | for ``dumpling`` v0.67.12, this draft release may never be published. | |
194 | ||
195 | * The ``Ceph lead`` decides a new stable release must be published | |
196 | * The ``release master`` gets approval from all leads | |
197 | * The ``release master`` writes and commits the release notes | |
198 | * The ``release master`` informs the ``quality engineer`` that the | |
199 | branch is ready for testing | |
200 | * The ``quality engineer`` runs additional integration tests | |
201 | * If the ``quality engineer`` discovers new bugs that require an | |
202 | ``Urgent Backport``, the release goes back to being prepared, it | |
203 | was not ready after all | |
204 | * The ``quality engineer`` informs the ``publisher`` that the branch | |
205 | is ready for release | |
206 | * The ``publisher`` `creates the packages and sets the release tag | |
207 | <../release-process>`_ | |
208 | ||
209 | The person responsible for each role is: | |
210 | ||
211 | * Sage Weil is the ``Ceph lead`` | |
212 | * Sage Weil is the ``release master`` for major stable releases | |
213 | (``firefly`` 0.80, ``hammer`` 0.94 etc.) | |
214 | * Loic Dachary is the ``release master`` for stable point releases | |
215 | (``firefly`` 0.80.10, ``hammer`` 0.94.1 etc.) | |
216 | * Yuri Weinstein is the ``quality engineer`` | |
217 | * Alfredo Deza is the ``publisher`` | |
218 | ||
219 | Cutting a new development release | |
220 | ================================= | |
221 | ||
222 | The publication workflow of a development release is the same as | |
223 | preparing a new release and cutting it, with the following | |
224 | differences: | |
225 | ||
226 | * The ``next`` branch is reset to the tip of ``master`` after | |
227 | publication | |
228 | * The ``quality engineer`` is not required to run additional tests, | |
229 | the ``release master`` directly informs the ``publisher`` that the | |
230 | release is ready to be published. | |
231 | ||
232 | Publishing point releases and backporting | |
233 | ========================================= | |
234 | ||
235 | The publication workflow of the point releases is the same as | |
236 | preparing a new release and cutting it, with the following | |
237 | differences: | |
238 | ||
239 | * The ``backport`` field of each issue contains the code name of the | |
240 | stable release | |
241 | * There is exactly one issue in the ``Backport`` tracker for each | |
242 | stable release to which the issue is backported | |
243 | * All commits are cherry-picked with ``git cherry-pick -x`` to | |
244 | reference the original commit | |
245 | ||
246 | See `the backporter manual | |
247 | <http://tracker.ceph.com/projects/ceph-releases/wiki/HOWTO>`_ for more | |
248 | information. |