ceph/doc/dev/developer_guide/tests-unit-tests.rst

   1 Testing - unit tests
   2 ====================
   3
   4 The Ceph GitHub repository has two types of tests: unit tests (also called
   5 ``make check`` tests) and integration tests. Strictly speaking, the
   6 ``make check`` tests are not "unit tests", but rather tests that can be run
   7 easily on a single build machine after compiling Ceph from source, whereas
   8 integration tests require package installation and multi-machine clusters to
   9 run.
  10
  11 .. _make-check:
  12
  13 What does "make check" mean?
  14 ----------------------------
  15
  16 After compiling Ceph, the code can be run through a battery of tests. For
  17 historical reasons, this is often referred to as ``make check`` even though
  18 the actual command used to run the tests is now ``ctest``. To be included in
  19 this group of tests, a test must:
  20
  21 * bind ports that do not conflict with other tests
  22 * not require root access
  23 * not require more than one machine to run
  24 * complete within a few minutes
  25
  26 For the sake of simplicity, this class of tests is referred to as "make
  27 check tests" or "unit tests". This is meant to distinguish these tests from
  28 the more complex "integration tests" that are run via the `teuthology
  29 framework`_.
  30
  31 While it is possible to run ``ctest`` directly, it can be tricky to correctly
  32 set up your environment for it. Fortunately, there is a script that makes it
  33 easy to run the unit tests on your code. This script can be run from the
  34 top-level directory of the Ceph source tree by invoking:
  35
  36   .. prompt:: bash $
  37
  38      ./run-make-check.sh
  39
  40 You will need a minimum of 8GB of RAM and 32GB of free drive space for this
  41 command to complete successfully on x86_64 architectures; other architectures
  42 may have different requirements. Depending on your hardware, it can take from
  43 twenty minutes to three hours to complete.
  44
  45
  46 How unit tests are declared
  47 ---------------------------
  48
  49 Unit tests are declared in the ``CMakeLists.txt`` file, which is found in the
  50 ``./src`` directory. The ``add_ceph_test`` and ``add_ceph_unittest`` CMake
  51 functions are used to declare unit tests.  ``add_ceph_test`` and
  52 ``add_ceph_unittest`` are themselves defined in
  53 ``./cmake/modules/AddCephTest.cmake``.
  54
  55 Some unit tests are scripts and other unit tests are binaries that are
  56 compiled during the build process.
  57
  58 * ``add_ceph_test`` function - used to declare unit test scripts
  59 * ``add_ceph_unittest`` function - used for unit test binaries
  60
  61 Unit testing of CLI tools
  62 -------------------------
  63 Some of the CLI tools are tested using special files ending with the extension
  64 ``.t`` and stored under ``./src/test/cli``. These tests are run using a tool
  65 called `cram`_ via a shell script called ``./src/test/run-cli-tests``.
  66 `cram`_ tests that are not suitable for ``make check`` can also be run by
  67 teuthology using the `cram task`_.
  68
  69 .. _`cram`: https://bitheap.org/cram/
  70 .. _`cram task`: https://github.com/ceph/ceph/blob/master/qa/tasks/cram.py
  71
  72 Tox-based testing of Python modules
  73 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  74 Some of the Python modules in Ceph use `tox <https://tox.readthedocs.io/en/latest/>`_
  75 to run their unit tests.
  76
  77 Most of these Python modules can be found in the directory ``./src/pybind/``.
  78
  79 Currently (December 2020) the following modules use **tox**:
  80
  81 * Cephadm (``./src/cephadm/tox.ini``)
  82 * Ceph Manager Python API (``./src/pybind/mgr``)
  83
  84   * ``./src/pybind/mgr/tox.ini``
  85
  86   * ``./src/pybind/mgr/dashboard/tox.ini``
  87
  88   * ``./src/pybind/tox.ini``
  89
  90 * Dashboard (``./src/pybind/mgr/dashboard``)
  91 * Python common (``./src/python-common/tox.ini``)
  92 * CephFS (``./src/tools/cephfs/tox.ini``)
  93 * ceph-volume
  94
  95   * ``./src/ceph-volume/tox.ini``
  96
  97   * ``./src/ceph-volume/plugin/zfs/tox.ini``
  98
  99   * ``./src/ceph-volume/ceph_volume/tests/functional/batch/tox.ini``
 100
 101   * ``./src/ceph-volume/ceph_volume/tests/functional/simple/tox.ini``
 102
 103   * ``./src/ceph-volume/ceph_volume/tests/functional/lvm/tox.ini``
 104
 105 Configuring Tox environments and tasks
 106 """"""""""""""""""""""""""""""""""""""
 107 Most tox configurations support multiple environments and tasks.
 108
 109 The list of environments and tasks that are supported is in the ``tox.ini``
 110 file, under ``envlist``. For example, here are the first three lines of
 111 ``./src/cephadm/tox.ini``::
 112
 113    [tox]
 114    envlist = py3, mypy
 115    skipsdist=true
 116
 117 In this example, the ``Python 3`` and ``mypy`` environments are specified.
 118
 119 The list of environments can be retrieved with the following command:
 120
 121   .. prompt:: bash $
 122
 123      tox --list
 124
 125 Or:
 126
 127   .. prompt:: bash $
 128
 129      tox -l
 130
 131 Running Tox
 132 """""""""""
 133 To run **tox**, just execute ``tox`` in the directory containing
 134 ``tox.ini``.  If you do not specify any environments (for example, ``-e
 135 $env1,$env2``), then ``tox`` will run all environments. Jenkins will run
 136 ``tox`` by executing ``./src/script/run_tox.sh``.
 137
 138 Here are some examples from Ceph Dashboard that show how to specify different
 139 environments and run options::
 140
 141   ## Run Python 2+3 tests+lint commands:
 142   $ tox -e py27,py3,lint,check
 143
 144   ## Run Python 3 tests+lint commands:
 145   $ tox -e py3,lint,check
 146
 147   ## To run it as Jenkins would:
 148   $ ../../../script/run_tox.sh --tox-env py3,lint,check
 149
 150 Manager core unit tests
 151 """""""""""""""""""""""
 152
 153 Currently only doctests_ inside ``mgr_util.py`` are run.
 154
 155 To add more files to be tested inside the core of the manager, open the
 156 ``tox.ini`` file and add the files to be tested  at the end of the line that
 157 includes ``mgr_util.py``.
 158
 159 .. _doctests: https://docs.python.org/3/library/doctest.html
 160
 161 Unit test caveats
 162 -----------------
 163
 164 #. Unlike the various Ceph daemons and ``ceph-fuse``, the unit tests are
 165    linked against the default memory allocator (glibc) unless they are
 166    explicitly linked against something else. This enables tools such as
 167    **valgrind** to be used in the tests.
 168
 169 #. Google Test unit testing library hides the client output from the shell.
 170    In order to debug the client after setting the desired debug level
 171    (e.g ``ceph config set client debug_rbd 20``), the debug log file can
 172    be found at ``build/out/client.admin.<pid>.log``.
 173    This can also be handy when examining teuthology failed unit test
 174    jobs, the job's debug level can be set at the relevant yaml file.
 175
 176 .. _make check:
 177 .. _teuthology framework: https://github.com/ceph/teuthology