1 .. _basic workflow dev guide:
6 The following chart illustrates the basic Ceph development workflow:
10 Upstream Code Your Local Environment
12 /----------\ git clone /-------------\
13 | Ceph | -------------------------> | ceph/main |
14 \----------/ \-------------/
19 /----------------\ git commit --amend /-------------\
20 | ninja check |---------------------> | ceph/fix_1 |
21 | ceph--qa--suite| \-------------/
28 /--------------\ /-------------\
29 | github |<---------------------- | ceph/fix_1 |
30 | pull request | git push \-------------/
33 This page assumes that you are a new contributor with an idea for a bugfix or
34 an enhancement, but you do not know how to proceed. Watch the `Getting Started
35 with Ceph Development <https://www.youtube.com/watch?v=t5UIehZ1oLs>`_ video for
36 a practical summary of this workflow.
41 Find the :ref:`issue-tracker` (Redmine) number of the bug you intend to fix. If
42 no tracker issue exists, create one. There is only one case in which you do not
43 have to create a Redmine tracker issue: the case of minor documentation changes.
45 Simple documentation cleanup does not require a corresponding tracker issue.
46 Major documenatation changes do require a tracker issue. Major documentation
47 changes include adding new documentation chapters or files, and making
48 substantial changes to the structure or content of the documentation.
50 A (Redmine) tracker ticket explains the issue (bug) to other Ceph developers to
51 keep them informed as the bug nears resolution. Provide a useful, clear title
52 and include detailed information in the description. When composing the title
53 of the ticket, ask yourself "If I need to search for this ticket two years from
54 now, which keywords am I likely to search for?" Then include those keywords in
57 If your tracker permissions are elevated, assign the bug to yourself by setting
58 the ``Assignee`` field. If your tracker permissions have not been elevated,
59 just add a comment with a short message that says "I am working on this issue".
61 Ceph Workflow Overview
62 ----------------------
64 Three repositories are involved in the Ceph workflow. They are:
66 1. The upstream repository (ceph/ceph)
67 2. Your fork of the upstream repository (your_github_id/ceph)
68 3. Your local working copy of the repository (on your workstation)
70 The procedure for making changes to the Ceph repository is as follows:
72 #. Configure your local environment
74 #. :ref:`Create a fork<forking>` of the "upstream Ceph"
77 #. :ref:`Clone the fork<cloning>` to your local filesystem.
81 #. :ref:`Synchronize local main with upstream main<synchronizing>`.
83 #. :ref:`Create a bugfix branch<bugfix_branch>` in your local working copy.
85 #. :ref:`Make alterations to the local working copy of the repository in your
86 local filesystem<fixing_bug_locally>`.
88 #. :ref:`Push the changes in your local working copy to your fork<push_changes>`.
90 #. Create a Pull Request to push the change upstream
92 #. Create a Pull Request that asks for your changes to be added into the
93 "upstream Ceph" repository.
95 Preparing Your Local Working Copy of the Ceph Repository
96 --------------------------------------------------------
98 The procedures in this section, "Preparing Your Local Working Copy of the Ceph
99 Repository", must be followed only when you are first setting up your local
100 environment. If this is your first time working with the Ceph project, then
101 these commands are necessary and are the first commands that you should run.
105 Creating a Fork of the Ceph Repository
106 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
108 See the `GitHub documentation
109 <https://help.github.com/articles/fork-a-repo/#platform-linux>`_ for
110 detailed instructions on forking. In short, if your GitHub username is
111 "mygithubaccount", your fork of the upstream repo will appear at
112 ``https://github.com/mygithubaccount/ceph``.
119 After you have created your fork, clone it by running the following command:
123 git clone https://github.com/mygithubaccount/ceph
125 You must fork the Ceph repository before you clone it. If you fail to fork,
126 you cannot open a `GitHub pull request
127 <https://docs.github.com/en/free-pro-team@latest/github/collaborating-with-issues-and-pull-requests/creating-a-pull-request>`_.
129 For more information on using GitHub, refer to `GitHub Help
130 <https://help.github.com/>`_.
132 Configuring Your Local Environment
133 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
135 The commands in this section configure your local git environment so that it
136 generates "Signed-off-by:" tags. They also set up your local environment so
137 that it can stay synchronized with the upstream repository.
139 These commands are necessary only during the initial setup of your local
140 working copy. Another way to say that is "These commands are necessary
141 only the first time that you are working with the Ceph repository. They are,
142 however, unavoidable, and if you fail to run them then you will not be able
143 to work on the Ceph repository.".
145 1. Configure your local git environment with your name and email address.
148 These commands will work only from within the ``ceph/`` directory
149 that was created when you cloned your fork.
153 git config user.name "FIRST_NAME LAST_NAME"
154 git config user.email "MY_NAME@example.com"
156 2. Add the upstream repo as a "remote" and fetch it:
160 git remote add ceph https://github.com/ceph/ceph.git
163 These commands fetch all the branches and commits from ``ceph/ceph.git`` to
164 the local git repo as ``remotes/ceph/$BRANCH_NAME`` and can be referenced as
165 ``ceph/$BRANCH_NAME`` in local git commands.
172 Synchronizing Local Main with Upstream Main
173 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
175 In your local git environment, there is a copy of the ``main`` branch in
176 ``remotes/origin/main``. This is called "local main". This copy of the
177 main branch (https://github.com/your_github_id/ceph.git) is "frozen in time"
178 at the moment that you cloned it, but the upstream repo
179 (https://github.com/ceph/ceph.git, typically abbreviated to ``ceph/ceph.git``)
180 that it was forked from is not frozen in time: the upstream repo is still being
181 updated by other contributors.
183 Because upstream main is continually receiving updates from other
184 contributors, your fork will drift farther and farther from the state of the
185 upstream repo when you cloned it.
187 You must keep your fork's main branch synchronized with upstream main in
188 order to reduce drift between your fork's main branch and the upstream main
191 Here are the commands for keeping your fork synchronized with the
198 git reset --hard ceph/main
199 git push -u origin main
201 This procedure should be followed often, in order to keep your local ``main``
202 in sync with upstream ``main``.
206 Creating a Bugfix branch
207 ^^^^^^^^^^^^^^^^^^^^^^^^
209 Create a branch for your bugfix:
214 git checkout -b fix_1
215 git push -u origin fix_1
217 The first command (git checkout main) makes sure that the bugfix branch
218 "fix_1" is created from the most recent state of the main branch of the
221 The second command (git checkout -b fix_1) creates a "bugfix branch" called
222 "fix_1" in your local working copy of the repository. The changes that you make
223 in order to fix the bug will be commited to this branch.
225 The third command (git push -u origin fix_1) pushes the bugfix branch from
226 your local working repository to your fork of the upstream repository.
228 .. _fixing_bug_locally:
230 Fixing the bug in the local working copy
231 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
233 #. Updating the tracker
235 In the `Ceph issue tracker <https://tracker.ceph.com>`_, change the status
236 of the tracker issue to "In progress". This communicates to other Ceph
237 contributors that you have begun working on a fix, which helps to avoid
238 duplication of effort. If you don't have permission to change that field,
239 your comment that you are working on the issue is sufficient.
241 #. Fixing the bug itself
243 This guide cannot tell you how to fix the bug that you have chosen to fix.
244 This guide assumes that you know what required improvement, and that you
245 know what to do to provide that improvement.
247 It might be that your fix is simple and requires only minimal testing. But
248 that's unlikely. It is more likely that the process of fixing your bug will
249 be iterative and will involve trial, error, skill, and patience.
251 For a detailed discussion of the tools available for validating bugfixes,
252 see the chapters on testing.
254 Pushing the Fix to Your Fork
255 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
257 You have finished work on the bugfix. You have tested the bugfix, and you
258 believe that it works.
260 #. Commit the changes to your local working copy.
262 Commit the changes to the `fix_1` branch of your local working copy by using
263 the ``--signoff`` option (here represented as the `s` portion of the `-as`
272 #. Push the changes to your fork:
274 Push the changes from the `fix_1` branch of your local working copy to the
275 `fix_1` branch of your fork of the upstream repository:
279 git push origin fix_1
283 In the command `git push origin fix_1`, `origin` is the name of your fork
284 of the upstream Ceph repository, and can be thought of as a nickname for
285 `git@github.com:username/ceph.git`, where `username` is your GitHub
288 It is possible that `origin` is not the name of your fork. Discover the
289 name of your fork by running `git remote -v`, as shown here:
294 ceph https://github.com/ceph/ceph.git (fetch)
295 ceph https://github.com/ceph/ceph.git (push)
296 origin git@github.com:username/ceph.git (fetch)
297 origin git@github.com:username/ceph.git (push)
299 The line "origin git@github.com:username/ceph.git (fetch)" and the line
300 "origin git@github.com:username/ceph.git (push)" provide the information
301 that "origin" is the name of your fork of the Ceph repository.
304 Opening a GitHub pull request
305 -----------------------------
307 After you have pushed the bugfix to your fork, open a GitHub pull request
308 (PR). This makes your bugfix visible to the community of Ceph contributors.
309 They will review it. They may perform additional testing on your bugfix, and
310 they might request changes to the bugfix.
312 Be prepared to receive suggestions and constructive criticism in the form of
313 comments within the PR.
315 If you don't know how to create and manage pull requests, read `this GitHub
316 pull request tutorial`_.
318 .. _`this GitHub pull request tutorial`:
319 https://help.github.com/articles/using-pull-requests/
321 To learn what constitutes a "good" pull request, see
322 the `Git Commit Good Practice`_ article at the `OpenStack Project Wiki`_.
324 .. _`Git Commit Good Practice`: https://wiki.openstack.org/wiki/GitCommitMessages
325 .. _`OpenStack Project Wiki`: https://wiki.openstack.org/wiki/Main_Page
327 See also our own `Submitting Patches
328 <https://github.com/ceph/ceph/blob/main/SubmittingPatches.rst>`_ document.
330 After your pull request (PR) has been opened, update the :ref:`issue-tracker`
331 by adding a comment directing other contributors to your PR. The comment can be
334 *PR*: https://github.com/ceph/ceph/pull/$NUMBER_OF_YOUR_PULL_REQUEST
336 Understanding Automated PR validation
337 -------------------------------------
339 When you create or update your PR, the Ceph project's `Continuous Integration
340 (CI) <https://en.wikipedia.org/wiki/Continuous_integration>`_ infrastructure
341 automatically tests it. At the time of this writing (May 2022), the automated
342 CI testing included many tests. These five are among them:
344 #. a test to check that the commits are properly signed (see :ref:`submitting-patches`):
345 #. a test to check that the documentation builds
346 #. a test to check that the submodules are unmodified
347 #. a test to check that the API is in order
348 #. a :ref:`make check<make-check>` test
350 Additional tests may be run depending on which files your PR modifies.
352 The :ref:`make check<make-check>` test builds the PR and runs it through a
353 battery of tests. These tests run on servers that are operated by the Ceph
354 Continuous Integration (CI) team. When the tests have completed their run, the
355 result is shown on GitHub in the pull request itself.
357 Test your modifications before you open a PR. Refer to the chapters
358 on testing for details.
360 Notes on PR make check test
361 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
363 The GitHub :ref:`make check<make-check>` test is driven by a Jenkins instance.
365 Jenkins merges your PR branch into the latest version of the base branch before
366 it starts any tests. This means that you don't have to rebase the PR in order
367 to pick up any fixes.
369 You can trigger PR tests at any time by adding a comment to the PR - the
370 comment should contain the string "test this please". Since a human who is
371 subscribed to the PR might interpret that as a request for him or her to test
372 the PR, you must address Jenkins directly. For example, write "jenkins retest
373 this please". If you need to run only one of the tests, you can request it with
374 a command like "jenkins test signed". A list of these requests is automatically
375 added to the end of each new PR's description, so check there to find the
376 single test you need.
378 If there is a build failure and you aren't sure what caused it, check the
379 :ref:`make check<make-check>` log. To access the make check log, click the
380 "details" (next to the :ref:`make check<make-check>` test in the PR) link to
381 enter the Jenkins web GUI. Then click "Console Output" (on the left).
383 Jenkins is configured to search logs for strings that are known to have been
384 associated with :ref:`make check<make-check>` failures in the past. However,
385 there is no guarantee that these known strings are associated with any given
386 :ref:`make check<make-check>` failure. You'll have to read through the log to
387 determine the cause of your specific failure.
389 Integration tests AKA ceph-qa-suite
390 -----------------------------------
392 Since Ceph is complex, it may be necessary to test your fix to
393 see how it behaves on real clusters running on physical or virtual
394 hardware. Tests designed for this purpose live in the `ceph/qa
395 sub-directory`_ and are run via the `teuthology framework`_.
397 .. _`ceph/qa sub-directory`: https://github.com/ceph/ceph/tree/main/qa/
398 .. _`teuthology repository`: https://github.com/ceph/teuthology
399 .. _`teuthology framework`: https://github.com/ceph/teuthology
401 The Ceph community has access to the `Sepia lab
402 <https://wiki.sepia.ceph.com/doku.php>`_ where `integration tests`_ can be run
403 on physical hardware.
404 Other developers may add tags like "needs-qa" to your PR. This allows PRs that
405 need testing to be merged into a single branch and tested all at the same time.
406 Since teuthology suites can take hours (even days in some cases) to run, this
407 can save a lot of time.
409 To request access to the Sepia lab, start `here
410 <https://wiki.sepia.ceph.com/doku.php?id=vpnaccess>`_.
412 Integration testing is discussed in more detail in the `integration
415 .. _integration tests: ../testing_integration_tests/tests-integration-testing-teuthology-intro
420 Once your bugfix has been thoroughly tested, or even during this process,
421 it will be subjected to code review by other developers. This typically
422 takes the form of comments in the PR itself, but can be supplemented
423 by discussions on :ref:`irc` and the :ref:`mailing-list`.
428 While your PR is going through testing and `Code Review`_, you can
429 modify it at any time by editing files in your local branch.
431 After updates are committed locally (to the ``fix_1`` branch in our
432 example), they need to be pushed to GitHub so they appear in the PR.
434 Modifying the PR is done by adding commits to the ``fix_1`` branch upon
435 which it is based, often followed by rebasing to modify the branch's git
436 history. See `this tutorial
437 <https://www.atlassian.com/git/tutorials/rewriting-history>`_ for a good
438 introduction to rebasing. When you are done with your modifications, you
439 will need to force push your branch with:
443 git push --force origin fix_1
445 Why do we take these extra steps instead of simply adding additional commits
446 the PR? It is best practice for a PR to consist of a single commit; this
447 makes for clean history, eases peer review of your changes, and facilitates
448 merges. In rare circumstances it also makes it easier to cleanly revert
454 The bugfix process completes when a project lead merges your PR.
456 When this happens, it is a signal for you (or the lead who merged the PR)
457 to change the :ref:`issue-tracker` status to "Resolved". Some issues may be
458 flagged for backporting, in which case the status should be changed to
459 "Pending Backport" (see the :ref:`backporting` chapter for details).
461 See also :ref:`merging` for more information on merging.
463 Proper Merge Commit Format
464 ^^^^^^^^^^^^^^^^^^^^^^^^^^
466 This is the most basic form of a merge commit::
468 doc/component: title of the commit
470 Reviewed-by: Reviewer Name <rname@example.com>
472 This consists of two parts:
474 #. The title of the commit / PR to be merged.
475 #. The name and email address of the reviewer. Enclose the reviewer's email
476 address in angle brackets.
478 Using .githubmap to Find a Reviewer's Email Address
479 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
480 If you cannot find the email address of the reviewer on his or her GitHub
481 page, you can look it up in the **.githubmap** file, which can be found in
482 the repository at **/ceph/.githubmap**.
484 Using "git log" to find a Reviewer's Email Address
485 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
486 If you cannot find a reviewer's email address by using the above methods, you
487 can search the git log for their email address. Reviewers are likely to have
488 committed something before. If they have made previous contributions, the git
489 log will probably contain their email address.
491 Use the following command
493 .. prompt:: bash [branch-under-review]$
497 Using ptl-tool to Generate Merge Commits
498 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
500 Another method of generating merge commits involves using Patrick Donnelly's
501 **ptl-tool** pull commits. This tool can be found at
502 **/ceph/src/script/ptl-tool.py**. Merge commits that have been generated by
503 the **ptl-tool** have the following form::
505 Merge PR #36257 into main
506 * refs/pull/36257/head:
507 client: move client_lock to _unmount()
508 client: add timer_lock support
509 Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>