1 .. _tests-integration-testing-teuthology-workflow:
3 Integration Tests using Teuthology Workflow
4 ===========================================
12 Ceph binaries must be built for your branch before you can use teuthology to run integration tests on them. Follow these steps to build the Ceph binaries:
14 #. Push the branch to the `ceph-ci`_ repository. This triggers the process of
15 building the binaries on the Jenkins CI.
17 #. To ensure that the build process has been initiated, confirm that the branch
18 name has appeared in the list of "Latest Builds Available" at `Shaman`_.
19 Soon after you start the build process, the testing infrastructure adds
20 other, similarly-named builds to the list of "Latest Builds Available".
21 The names of these new builds will contain the names of various Linux
22 distributions of Linux and will be used to test your build against those
25 #. Wait for the packages to be built and uploaded to `Chacra`_, and wait for
26 the repositories offering the packages to be created. The entries for the
27 branch names in the list of "Latest Builds Available" on `Shaman`_ will turn
28 green to indicate that the packages have been uploaded to `Chacra`_ and to
29 indicate that their repositories have been created. Wait until each entry
30 is coloured green. This usually takes between two and three hours depending
31 on the availability of the machines.
33 The Chacra URL for a particular build can be queried from `the Chacra site`_.
35 .. note:: The branch to be pushed on ceph-ci can be any branch. The branch does
36 not have to be a PR branch.
38 .. note:: If you intend to push master or any other standard branch, check
39 `Shaman`_ beforehand since it might already have completed builds for it.
41 .. _the Chacra site: https://shaman.ceph.com/api/search/?status=ready&project=ceph
47 After you have built Ceph binaries for your branch, you can run tests using
48 teuthology. This procedure explains how to run tests using teuthology.
50 #. Log in to the teuthology machine:
54 ssh <username>@teuthology.front.sepia.ceph.com
56 This requires Sepia lab access. To request access to the Sepia lab, see:
57 https://ceph.github.io/sepia/adding_users/
59 #. Run the ``teuthology-suite`` command:
65 -c wip-devname-feature-x \
68 --filter "cephfs-shell" \
71 The options in the above command are defined here:
73 ============= =========================================================
75 ============= =========================================================
78 -c the name of the branch that was pushed on ceph-ci
80 -p the higher the number, the lower the priority of
82 --filter filter tests in a given suite. The argument
83 passed to this filter specifies which test you
85 -e <email> When tests finish or time out, send an email to the
86 specified address. Can also be specified in
87 ~/.teuthology.yaml as 'results_email'
88 ============= =========================================================
90 .. note:: The priority number present in the command above is a placeholder.
91 Do not use it in your own tests. See `Testing Priority`_ for information
92 about recommended values.
94 .. note:: Do not issue a command without a priority number. The default
95 value is 1000, a value so large that your job is unlikely ever to run.
97 Run ``teuthology-suite --help`` to read descriptions of these and other
100 #. Wait for the tests to run. ``teuthology-suite`` prints a link to
101 `Pulpito`_ where the test results can be viewed.
105 Other frequently used/useful options are ``-d`` (or ``--distro``),
106 ``--distroversion``, ``--filter-out``, ``--timeout``, ``flavor``, ``-rerun``,
107 ``-l`` (for limiting number of jobs) , ``-N`` (for how many times the job will
108 run), and ``--subset`` (used to reduce the number of tests that are triggered). Run
109 ``teuthology-suite --help`` to read descriptions of these and other options.
111 .. _teuthology_testing_qa_changes:
113 Testing QA changes (without re-building binaries)
114 *************************************************
116 If you are making changes only in the ``qa/`` directory, you do not have to
117 rebuild the binaries before you re-run tests. If you make changes only in
118 ``qa/``, you can use the binaries built for the ceph-ci branch to re-run tests.
119 You just have to make sure to tell the ``teuthology-suite`` command to use a
120 separate branch for running the tests.
122 If you made changes only in ``qa/``
123 (https://github.com/ceph/ceph/tree/master/qa), you do not need to rebuild the
124 binaries. You can use existing binaries that are built periodically for master and other stable branches and run your test changes against them.
125 Your branch with the qa changes can be tested by passing two extra arguments to the ``teuthology-suite`` command: (1) ``--suite-repo``, specifying your ceph repo, and (2) ``--suite-branch``, specifying your branch name.
127 For example, if you want to make changes in ``qa/`` after testing ``branch-x``
128 (for which the ceph-ci branch is ``wip-username-branch-x``), run the following
133 teuthology-suite -v \
135 -c wip-username-branch-x \
138 --filter cephfs-shell
140 Then make modifications locally, update the PR branch, and trigger tests from
141 your PR branch as follows:
145 teuthology-suite -v \
147 -c wip-username-branch-x \
149 --filter cephfs-shell \
150 --suite-repo https://github.com/$username/ceph \
151 --suite-branch branch-x
153 You can verify that the tests were run using this branch by looking at the
154 values for the keys ``suite_branch``, ``suite_repo`` and ``suite_sha1`` in the
155 job config printed at the beginning of the teuthology job.
157 .. note:: If you are making changes that are not in the ``qa/`` directory,
158 you must follow the standard process of triggering builds, waiting
159 for the builds to finish, then triggering tests and waiting for
162 About Suites and Filters
163 ************************
165 See `Suites Inventory`_ for a list of available suites of integration tests.
166 Each directory under ``qa/suites`` in the Ceph repository is an integration
167 test suite, and arguments appropriate to follow ``-s`` can be found there.
169 Keywords for filtering tests can be found in
170 ``qa/suites/<suite-name>/<subsuite-name>/tasks`` and can be used as arguments
171 for ``--filter``. Each YAML file in that directory can trigger tests; using the
172 name of the file without its filename extension as an argument to the
173 ``--filter`` triggers those tests.
175 For example, in the command above in the :ref:`Testing QA Changes
176 <teuthology_testing_qa_changes>` section, ``cephfs-shell`` is specified.
177 This works because there is a file named ``cephfs-shell.yaml`` in
178 ``qa/suites/fs/basic_functional/tasks/``.
180 If the filename doesn't suggest what kind of tests it triggers, search the
181 contents of the file for the ``modules`` attribute. For ``cephfs-shell.yaml``
182 the ``modules`` attribute is ``tasks.cephfs.test_cephfs_shell``. This means
183 that it triggers all tests in ``qa/tasks/cephfs/test_cephfs_shell.py``.
186 ---------------------
191 After the teuthology job is scheduled, the status and results of the test run
192 can be checked at https://pulpito.ceph.com/.
197 After the tests have finished running, the log for the job can be obtained by
198 clicking on the job ID at the Pulpito page associated with your tests. It's
199 more convenient to download the log and then view it rather than viewing it in
200 an internet browser since these logs can easily be up to 1 GB in size. It is
201 easier to ssh into the teuthology machine (``teuthology.front.sepia.ceph.com``)
202 and access the following path::
204 /ceph/teuthology-archive/<test-id>/<job-id>/teuthology.log
206 For example: for the above test ID, the path is::
208 /ceph/teuthology-archive/teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi/4588482/teuthology.log
210 This method can be used to view the log more quickly than would be possible through a browser.
212 .. note:: To access archives more conveniently, ``/a/`` has been symbolically
213 linked to ``/ceph/teuthology-archive/``. For instance, to access the previous
214 example, we can use something like::
216 /a/teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi/4588482/teuthology.log
220 ``teuthology-kill`` can be used to kill jobs that have been running
221 unexpectedly for several hours, or when developers want to terminate tests
222 before they complete.
224 Here is the command that terminates jobs:
228 teuthology-kill -r teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi
230 Let's call the argument passed to ``-r`` as test ID. It can be found
231 easily in the link to the Pulpito page for the tests you triggered. For
232 example, for the above test ID, the link is - http://pulpito.front.sepia.ceph.com/teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi/
237 The ``teuthology-suite`` command has a ``-r`` (or ``--rerun``) option, which
238 allows you to re-run tests. This is handy when your tests have failed or end
239 up dead. The ``--rerun`` option takes the name of a teuthology run as an
240 argument. Option ``-R`` (or ``--rerun-statuses``) can be passed along with
241 ``-r`` to choose which kind of tests should be picked from the run. For
242 example, you can re-run only those tests from previous run which had ended up
243 as dead. Following is a practical example:
247 teuthology-suite -v \
249 -c wip-rishabh-fs-test_cephfs_shell-fix \
251 --r teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi \
252 -R fail,dead,queued \
255 Following's the definition of new options introduced in this section:
257 ======================= ===============================================
259 ======================= ===============================================
260 -r, --rerun Attempt to reschedule a run, selecting only
261 those jobs whose status are mentioned by
263 -R, --rerun-statuses A comma-separated list of statuses to be used
264 with --rerun. Supported statuses: 'dead',
265 'fail', 'pass', 'queued', 'running' and
266 'waiting'. Default value: 'fail,dead'
267 ======================= ===============================================
269 Naming the ceph-ci branch
270 -------------------------
271 Prepend your branch with your name before you push it to ceph-ci. For example,
272 a branch named ``feature-x`` should be named ``wip-$yourname-feature-x``, where
273 ``$yourname`` is replaced with your name. Identifying your branch with your
274 name makes your branch easily findable on Shaman and Pulpito.
276 If you are using one of the stable branches (`quincy`, `pacific`, etc.), include
277 the name of that stable branch in your ceph-ci branch name.
278 For example, the ``feature-x`` PR branch should be named
279 ``wip-feature-x-nautilus``. *This is not just a convention. This ensures that your branch is built in the correct environment.*
281 Delete the branch from ceph-ci when you no longer need it. If you are
282 logged in to GitHub, all your branches on ceph-ci can be found here:
283 https://github.com/ceph/ceph-ci/branches.
285 .. _ceph-ci: https://github.com/ceph/ceph-ci
286 .. _Chacra: https://github.com/ceph/chacra/blob/master/README.rst
287 .. _Pulpito: http://pulpito.front.sepia.ceph.com/
288 .. _Running Your First Test: ../../running-tests-locally/#running-your-first-test
289 .. _Shaman: https://shaman.ceph.com/builds/ceph/
290 .. _Suites Inventory: ../tests-integration-testing-teuthology-intro/#suites-inventory
291 .. _Testing Priority: ../tests-integration-testing-teuthology-intro/#testing-priority
292 .. _Triggering Tests: ../tests-integration-testing-teuthology-workflow/#triggering-tests
293 .. _tests-sentry-developers-guide: ../tests-sentry-developers-guide/