ceph/doc/dev/developer_guide/running-tests-using-teuth.rst

   1 Running Integration Tests using Teuthology
   2 ==========================================
   3
   4 Getting binaries
   5 ----------------
   6 To run integration tests using teuthology, you need to have Ceph binaries
   7 built for your branch. Follow these steps to initiate the build process -
   8
   9 #. Push the branch to `ceph-ci`_ repository. This triggers the process of
  10    building the binaries.
  11
  12 #. To confirm that the build process has been initiated, spot the branch name
  13    at `Shaman`_. Little after the build process has been initiated, the single
  14    entry with your branch name would multiply, each new entry for a different
  15    combination of distro and flavour.
  16
  17 #. Wait until the packages are built and uploaded, and the repository offering
  18    them are created. This is marked by colouring the entries for the branch
  19    name green. Preferably, wait until each entry is coloured green. Usually,
  20    it takes around 2-3 hours depending on the availability of the machines.
  21
  22 .. note:: Branch to be pushed on ceph-ci can be any branch, it shouldn't
  23    necessarily be a PR branch.
  24
  25 .. note:: In case you are pushing master or any other standard branch, check
  26    `Shaman`_ beforehand since it already might have builds ready for it.
  27
  28 Triggering Tests
  29 ----------------
  30 After building is complete, proceed to trigger tests -
  31
  32 #. Log in to the teuthology machine::
  33
  34        ssh <username>@teuthology.front.sepia.ceph.com
  35
  36    This would require Sepia lab access. To know how to request it, see: https://ceph.github.io/sepia/adding_users/
  37
  38 #. Next, get teuthology installed. Run the first set of commands in
  39    `Running Your First Test`_ for that. After that, activate the virtual
  40    environment in which teuthology is installed.
  41
  42 #. Run the ``teuthology-suite`` command::
  43
  44         teuthology-suite -v -m smithi -c wip-devname-feature-x -s fs -p 110 --filter "cephfs-shell"
  45
  46    Following are the options used in above command with their meanings -
  47         -v          verbose
  48         -m          machine name
  49         -c          branch name, the branch that was pushed on ceph-ci
  50         -s          test-suite name
  51         -p          higher the number, lower the priority of the job
  52         --filter    filter tests in given suite that needs to run, the arg to
  53                     filter should be the test you want to run
  54
  55 .. note:: The priority number present in the command above is just a
  56    placeholder. It might be highly inappropriate for the jobs you may want to
  57    trigger. See `Testing Priority`_ section to pick a priority number.
  58
  59 .. note:: Don't skip passing a priority number, the default value is 1000
  60    which way too high; the job probably might never run.
  61
  62 #. Wait for the tests to run. ``teuthology-suite`` prints a link to the
  63    `Pulpito`_ page created for the tests triggered.
  64
  65 Other frequently used/useful options are ``-d`` (or ``--distro``),
  66 ``--distroversion``, ``--filter-out``, ``--timeout``, ``flavor``, ``-rerun``.
  67 Run ``teuthology-suite --help`` to read description of these and every other
  68 options available.
  69
  70 About Suites and Filters
  71 ------------------------
  72 See `Suites Inventory`_ for a list of suites of integration tests present
  73 right now. Alternatively, each directory under ``qa/suites`` in Ceph
  74 repository is an integration test suite, so looking within that directory
  75 to decide an appropriate argument for ``-s`` also works.
  76
  77 For picking an argument for ``--filter``, look within
  78 ``qa/suites/<suite-name>/<subsuite-name>/tasks`` to get keywords for filtering
  79 tests. Each YAML file in there can trigger a bunch of tests; using the name of
  80 the file, without the extension part of the file name, as an argument to the
  81 ``--filter`` will trigger those tests. For example, the sample command above
  82 uses ``cephfs-shell`` since there's a file named ``cephfs-shell.yaml`` in
  83 ``qa/suites/fs/basic_functional/tasks/``. In case, the file name doesn't hint
  84 what bunch of tests it would trigger, look at the contents of the file for
  85 ``modules`` attribute. For ``cephfs-shell.yaml`` the ``modules`` attribute
  86 is ``tasks.cephfs.test_cephfs_shell`` which means it'll trigger all tests in
  87 ``qa/tasks/cephfs/test_cephfs_shell.py``.
  88
  89 Killing Tests
  90 -------------
  91 Sometimes a teuthology job might not complete running for several minutes or
  92 even hours after tests that were trigged have completed running and other
  93 times wrong set of tests can be triggered is filter wasn't chosen carefully.
  94 To save resource it's better to termniate such a job. Following is the command
  95 to terminate a job::
  96
  97         teuthology-kill -r teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi
  98
  99 Let's call the the argument passed to ``-r`` as test ID. It can be found
 100 easily in the link to the Pulpito page for the tests you triggered. For
 101 example, for the above test ID, the link is - http://pulpito.front.sepia.ceph.com/teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi/
 102
 103 Re-running Tests
 104 ----------------
 105 Pass ``--rerun`` option, with test ID as an argument to it, to
 106 ``teuthology-suite`` command::
 107
 108     teuthology-suite -v -m smithi -c wip-rishabh-fs-test_cephfs_shell-fix -p 50 --rerun teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi
 109
 110 The meaning of rest of the options is already covered in `Triggering Tests`
 111 section.
 112
 113 Teuthology Archives
 114 -------------------
 115 Once the tests have finished running, the log for the job can be obtained by
 116 clicking on job ID at the Pulpito page for your tests. It's more convenient to
 117 download the log and then view it rather than viewing it in an internet
 118 browser since these logs can easily be upto size of 1 GB. What's much more
 119 easier is to log in to the teuthology machine again
 120 (``teuthology.front.sepia.ceph.com``), and access the following path::
 121
 122     /ceph/teuthology-archive/<test-id>/<job-id>/teuthology.log
 123
 124 For example, for above test ID path is::
 125
 126     /ceph/teuthology-archive/teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi/4588482/teuthology.log
 127
 128 This way the log remotely can be viewed remotely without having to wait too
 129 much.
 130
 131 Naming the ceph-ci branch
 132 -------------------------
 133 There are no hard conventions (except for the case of stable branch; see
 134 next paragraph) for how the branch pushed on ceph-ci is named. But, to make
 135 builds and tests easily identitifiable on Shaman and Pulpito respectively,
 136 prepend it with your name. For example branch ``feature-x`` can be named
 137 ``wip-yourname-feature-x`` while pushing on ceph-ci.
 138
 139 In case you are using one of the stable branches (e.g.  nautilis, mimic,
 140 etc.), include the name of that stable branch in your ceph-ci branch name.
 141 For example, ``feature-x`` PR branch should be named as
 142 ``wip-feature-x-nautilus``. *This is not just a matter of convention but this,
 143 more essentially, builds your branch in the correct environment.*
 144
 145 Delete the branch from ceph-ci, once it's not required anymore. If you are
 146 logged in at GitHub, all your branches on ceph-ci can be easily found here -
 147 https://github.com/ceph/ceph-ci/branches.
 148
 149 .. _ceph-ci: https://github.com/ceph/ceph-ci
 150 .. _Pulpito: http://pulpito.front.sepia.ceph.com/
 151 .. _Running Your First Test: ../running-tests-locally/#running-your-first-test
 152 .. _Shaman: https://shaman.ceph.com/builds/ceph/
 153 .. _Suites Inventory: ../tests-integration-tests/#suites-inventory
 154 .. _Testing Priority: ../tests-integration-tests/#testing-priority