]>
Commit | Line | Data |
---|---|---|
9f95a23c TL |
1 | Running Integration Tests using Teuthology |
2 | ========================================== | |
3 | ||
4 | Getting binaries | |
5 | ---------------- | |
6 | To run integration tests using teuthology, you need to have Ceph binaries | |
7 | built for your branch. Follow these steps to initiate the build process - | |
8 | ||
9 | #. Push the branch to `ceph-ci`_ repository. This triggers the process of | |
10 | building the binaries. | |
11 | ||
12 | #. To confirm that the build process has been initiated, spot the branch name | |
13 | at `Shaman`_. Little after the build process has been initiated, the single | |
14 | entry with your branch name would multiply, each new entry for a different | |
15 | combination of distro and flavour. | |
16 | ||
17 | #. Wait until the packages are built and uploaded, and the repository offering | |
18 | them are created. This is marked by colouring the entries for the branch | |
19 | name green. Preferably, wait until each entry is coloured green. Usually, | |
20 | it takes around 2-3 hours depending on the availability of the machines. | |
21 | ||
22 | .. note:: Branch to be pushed on ceph-ci can be any branch, it shouldn't | |
23 | necessarily be a PR branch. | |
24 | ||
25 | .. note:: In case you are pushing master or any other standard branch, check | |
26 | `Shaman`_ beforehand since it already might have builds ready for it. | |
27 | ||
28 | Triggering Tests | |
29 | ---------------- | |
30 | After building is complete, proceed to trigger tests - | |
31 | ||
32 | #. Log in to the teuthology machine:: | |
33 | ||
34 | ssh <username>@teuthology.front.sepia.ceph.com | |
35 | ||
36 | This would require Sepia lab access. To know how to request it, see: https://ceph.github.io/sepia/adding_users/ | |
37 | ||
38 | #. Next, get teuthology installed. Run the first set of commands in | |
39 | `Running Your First Test`_ for that. After that, activate the virtual | |
40 | environment in which teuthology is installed. | |
41 | ||
42 | #. Run the ``teuthology-suite`` command:: | |
43 | ||
44 | teuthology-suite -v -m smithi -c wip-devname-feature-x -s fs -p 110 --filter "cephfs-shell" | |
45 | ||
46 | Following are the options used in above command with their meanings - | |
47 | -v verbose | |
48 | -m machine name | |
49 | -c branch name, the branch that was pushed on ceph-ci | |
50 | -s test-suite name | |
51 | -p higher the number, lower the priority of the job | |
52 | --filter filter tests in given suite that needs to run, the arg to | |
53 | filter should be the test you want to run | |
54 | ||
55 | .. note:: The priority number present in the command above is just a | |
56 | placeholder. It might be highly inappropriate for the jobs you may want to | |
57 | trigger. See `Testing Priority`_ section to pick a priority number. | |
58 | ||
59 | .. note:: Don't skip passing a priority number, the default value is 1000 | |
60 | which way too high; the job probably might never run. | |
61 | ||
62 | #. Wait for the tests to run. ``teuthology-suite`` prints a link to the | |
63 | `Pulpito`_ page created for the tests triggered. | |
64 | ||
65 | Other frequently used/useful options are ``-d`` (or ``--distro``), | |
f67539c2 TL |
66 | ``--distroversion``, ``--filter-out``, ``--timeout``, ``flavor``, ``-rerun``, |
67 | ``-l`` (for limiting number of jobs) , ``-n`` (for how many times job would | |
68 | run) and ``-e`` (for email notifications). Run ``teuthology-suite --help`` | |
69 | to read description of these and every other options available. | |
70 | ||
71 | Testing QA changes (without re-building binaires) | |
72 | ------------------------------------------------- | |
73 | While writing a PR you might need to test your PR repeatedly using teuthology. | |
74 | If you are making non-QA changes, you need to follow the standard process of | |
75 | triggering builds, waiting for it to finish and then triggering tests and | |
76 | wait for the result. But if changes you made are purely changes in qa/, | |
77 | you don't need rebuild the binaries. Instead you can test binaries built for | |
78 | the ceph-ci branch and instruct ``teuthology-suite`` command to use a separate | |
79 | branch for running tests. The separate branch can be passed to the command | |
80 | by using ``--suite-repo`` and ``--suite-branch``. Pass the link to the GitHub | |
81 | fork where your PR branch exists to the first option and pass the PR branch | |
82 | name to the second option. | |
83 | ||
84 | For example, if you want to make changes in ``qa/`` after testing ``branch-x`` | |
85 | (of which has ceph-ci branch is ``wip-username-branch-x``) by running | |
86 | following command:: | |
87 | ||
88 | teuthology-suite -v -m smithi -c wip-username-branch-x -s fs -p 50 --filter cephfs-shell | |
89 | ||
90 | You can make the modifications locally, update the PR branch and then | |
91 | trigger tests from your PR branch as follows:: | |
92 | ||
93 | teuthology-suite -v -m smithi -c wip-username-branch-x -s fs -p 50 --filter cephfs-shell --suite-repo https://github.com/username/ceph --suite-branch branch-x | |
94 | ||
95 | You can verify if the tests were run using this branch by looking at values | |
96 | for the keys ``suite_branch``, ``suite_repo`` and ``suite_sha1`` in the job | |
97 | config printed at the very beginning of the teuthology job. | |
9f95a23c TL |
98 | |
99 | About Suites and Filters | |
100 | ------------------------ | |
101 | See `Suites Inventory`_ for a list of suites of integration tests present | |
102 | right now. Alternatively, each directory under ``qa/suites`` in Ceph | |
103 | repository is an integration test suite, so looking within that directory | |
104 | to decide an appropriate argument for ``-s`` also works. | |
105 | ||
106 | For picking an argument for ``--filter``, look within | |
107 | ``qa/suites/<suite-name>/<subsuite-name>/tasks`` to get keywords for filtering | |
108 | tests. Each YAML file in there can trigger a bunch of tests; using the name of | |
109 | the file, without the extension part of the file name, as an argument to the | |
110 | ``--filter`` will trigger those tests. For example, the sample command above | |
111 | uses ``cephfs-shell`` since there's a file named ``cephfs-shell.yaml`` in | |
112 | ``qa/suites/fs/basic_functional/tasks/``. In case, the file name doesn't hint | |
113 | what bunch of tests it would trigger, look at the contents of the file for | |
114 | ``modules`` attribute. For ``cephfs-shell.yaml`` the ``modules`` attribute | |
115 | is ``tasks.cephfs.test_cephfs_shell`` which means it'll trigger all tests in | |
116 | ``qa/tasks/cephfs/test_cephfs_shell.py``. | |
117 | ||
118 | Killing Tests | |
119 | ------------- | |
120 | Sometimes a teuthology job might not complete running for several minutes or | |
121 | even hours after tests that were trigged have completed running and other | |
122 | times wrong set of tests can be triggered is filter wasn't chosen carefully. | |
123 | To save resource it's better to termniate such a job. Following is the command | |
124 | to terminate a job:: | |
125 | ||
126 | teuthology-kill -r teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi | |
127 | ||
f67539c2 | 128 | Let's call the argument passed to ``-r`` as test ID. It can be found |
9f95a23c TL |
129 | easily in the link to the Pulpito page for the tests you triggered. For |
130 | example, for the above test ID, the link is - http://pulpito.front.sepia.ceph.com/teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi/ | |
131 | ||
132 | Re-running Tests | |
133 | ---------------- | |
134 | Pass ``--rerun`` option, with test ID as an argument to it, to | |
135 | ``teuthology-suite`` command:: | |
136 | ||
137 | teuthology-suite -v -m smithi -c wip-rishabh-fs-test_cephfs_shell-fix -p 50 --rerun teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi | |
138 | ||
139 | The meaning of rest of the options is already covered in `Triggering Tests` | |
140 | section. | |
141 | ||
142 | Teuthology Archives | |
143 | ------------------- | |
144 | Once the tests have finished running, the log for the job can be obtained by | |
145 | clicking on job ID at the Pulpito page for your tests. It's more convenient to | |
146 | download the log and then view it rather than viewing it in an internet | |
147 | browser since these logs can easily be upto size of 1 GB. What's much more | |
148 | easier is to log in to the teuthology machine again | |
149 | (``teuthology.front.sepia.ceph.com``), and access the following path:: | |
150 | ||
151 | /ceph/teuthology-archive/<test-id>/<job-id>/teuthology.log | |
152 | ||
153 | For example, for above test ID path is:: | |
154 | ||
155 | /ceph/teuthology-archive/teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi/4588482/teuthology.log | |
156 | ||
157 | This way the log remotely can be viewed remotely without having to wait too | |
158 | much. | |
159 | ||
160 | Naming the ceph-ci branch | |
161 | ------------------------- | |
162 | There are no hard conventions (except for the case of stable branch; see | |
163 | next paragraph) for how the branch pushed on ceph-ci is named. But, to make | |
164 | builds and tests easily identitifiable on Shaman and Pulpito respectively, | |
165 | prepend it with your name. For example branch ``feature-x`` can be named | |
166 | ``wip-yourname-feature-x`` while pushing on ceph-ci. | |
167 | ||
168 | In case you are using one of the stable branches (e.g. nautilis, mimic, | |
169 | etc.), include the name of that stable branch in your ceph-ci branch name. | |
170 | For example, ``feature-x`` PR branch should be named as | |
171 | ``wip-feature-x-nautilus``. *This is not just a matter of convention but this, | |
172 | more essentially, builds your branch in the correct environment.* | |
173 | ||
174 | Delete the branch from ceph-ci, once it's not required anymore. If you are | |
175 | logged in at GitHub, all your branches on ceph-ci can be easily found here - | |
176 | https://github.com/ceph/ceph-ci/branches. | |
177 | ||
178 | .. _ceph-ci: https://github.com/ceph/ceph-ci | |
179 | .. _Pulpito: http://pulpito.front.sepia.ceph.com/ | |
180 | .. _Running Your First Test: ../running-tests-locally/#running-your-first-test | |
181 | .. _Shaman: https://shaman.ceph.com/builds/ceph/ | |
182 | .. _Suites Inventory: ../tests-integration-tests/#suites-inventory | |
183 | .. _Testing Priority: ../tests-integration-tests/#testing-priority |