ceph/src/boost/libs/test/doc/runtime_configuration/test_unit_filtering.qbk

   1 [/
   2  / Copyright (c) 2003 Boost.Test contributors
   3  /
   4  / Distributed under the Boost Software License, Version 1.0. (See accompanying
   5  / file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
   6  /]
   7
   8
   9 [section:test_unit_filtering Test unit filtering]
  10
  11 The __UTF__ offers a number of ways to run only a subset of all test cases registered in the test tree.
  12
  13 [#ref_default_run_status][h3 Default run status]
  14
  15 Each test unit (either test case or test suite) has an associated ['default run status]. It can assume one of the three values:
  16
  17 # ['true] -- this means that, unless some runtime parameters specify otherwise, the test unit is designated to be run.
  18 # ['false] -- this means that, unless some runtime parameters specify otherwise, the test unit is designated ['not] to be run.
  19 # ['inherit] -- this means that the test unit's default run status is the same as that of its immediate parent test unit. This is applied recursively.
  20
  21 Initially, the master test suite has default run status set to ['true]. All other test units have default run status set to ['inherit].
  22 This implies that, unless any additional configuration is applied, all tests are designated to be run.
  23
  24 You can set a different default run status in any test unit by using [link boost_test.tests_organization.decorators decorators]:
  25 __decorator_disabled__, __decorator_enabled__ and __decorator_enable_if__. The default run status is set once, upon testing program
  26 initialization, and cannot be changed. The disabled tests are not executed by default, but are still present in the test tree,
  27 and are listed along with other tests when you use command-line argument
  28 [link boost_test.utf_reference.rt_param_reference.list_content `list_content`].
  29
  30 [bt_example decorator_20..default run status..run-fail]
  31
  32 [#ref_dynamic_test_dependency][h3 Dynamic test dependencies]
  33
  34 Additionally, it is possible to statically associate a test unit with a condition. This associated condition is evaluated immediately
  35 before executing the test unit. If the condition is met, the test unit is executed. Otherwise, the test unit is ['skipped].
  36 It is possible to add two dependencies:
  37
  38 # Upon another test unit. In this case the decorated test case is skipped if the test unit specified in the dependency is either
  39   failed or skipped or disabled. This can be declared with decorator __decorator_depends_on__.
  40 # Upon an arbitrary predicate. This can be declared with decorator __decorator_precondition__.
  41
  42 [#ref_command_line_control][h3 Command-line control]
  43
  44 Static configuration of the test-case filtering is used by default, unless command-line filtering is applied. With command-line argument
  45 __param_run_test__ it is possible to alter the static pre-set in a number of ways:
  46
  47 # Ignore the static configuration and manually specify test cases to be run.
  48 # Augment the statically defined set by enabling the disabled test cases.
  49 # Shrink the statically defined set by disabling some of the enabled test cases.
  50
  51 [#ref_command_line_control_absolute][h4 Absolute specification]
  52
  53 Term 'absolute' in this context means that the default run status of the test units, which has been statically set up,
  54 is completely ignored and the tests to be run are specified manually from scratch. First, in order to learn what test
  55 units are registered in the test tree the user needs to use command-line argument
  56 [link boost_test.utf_reference.rt_param_reference.list_content `list_content`].
  57 Next, in order to specify a set of test cases, the user needs to use command-line argument
  58 __param_run_test__ with absolute value:
  59
  60 ```
  61 > test_program --__param_run_test__=<test_set>
  62 ```
  63
  64 The format of `<test_set>` value can assume a number of forms. Given the following program:
  65
  66 ```
  67 #define __BOOST_TEST_MODULE__ example
  68 #include <boost/test/included/unit_test.hpp>
  69
  70 using boost::unit_test::__decorator_label__;
  71
  72 __BOOST_AUTO_TEST_CASE__(test_1, *label("L1")) {}
  73 BOOST_AUTO_TEST_CASE(test_2, *label("L1")) {}
  74
  75 __BOOST_AUTO_TEST_SUITE__(suite_1)
  76
  77   BOOST_AUTO_TEST_SUITE(suite_1)
  78     BOOST_AUTO_TEST_CASE(test_1) {}
  79     BOOST_AUTO_TEST_CASE(test_2) {}
  80   BOOST_AUTO_TEST_SUITE_END()
  81
  82   BOOST_AUTO_TEST_SUITE(suite_2)
  83     BOOST_AUTO_TEST_CASE(test_1, *label("L2")) {}
  84     BOOST_AUTO_TEST_CASE(test_2, *label("L2")) {}
  85   BOOST_AUTO_TEST_SUITE_END()
  86
  87   BOOST_AUTO_TEST_CASE(test_1, *label("L1")) {}
  88   BOOST_AUTO_TEST_CASE(test_2) {}
  89   BOOST_AUTO_TEST_CASE(test_2A) {}
  90
  91 BOOST_AUTO_TEST_SUITE_END()
  92 ```
  93
  94 The following table illustrates how different values of `<test_set>` control which test cases ware run.
  95
  96 [table
  97   [
  98     [Description]
  99     [Parameter value]
 100     [Test cases run]
 101   ]
 102
 103   [
 104     [Run single top-level test case by name]
 105     [[pre __param_run_test__=test_1]]
 106     [
 107 [pre
 108 test_1
 109 ]
 110     ]
 111   ]
 112
 113   [
 114     [Run single nested test case by name]
 115     [[pre __param_run_test__=suite_1/suite_1/test_1]]
 116     [
 117 [pre
 118 suite_1/suite_1/test_1
 119 ]
 120     ]
 121   ]
 122
 123   [
 124     [Run single test suite by name]
 125     [
 126 [pre __param_run_test__=suite_1/suite_2
 127 __param_run_test__=suite_1/suite_2/*
 128 ]
 129     ]
 130     [
 131 [pre
 132 suite_1/suite_2/test_1
 133 suite_1/suite_2/test_2
 134 ]
 135     ]
 136   ]
 137
 138   [
 139     [Run multiple test units that are *siblings* of the same test suite]
 140     [[pre __param_run_test__=suite_1/test_1,suite_2]]
 141     [
 142 [pre
 143 suite_1/suite_2/test_1
 144 suite_1/suite_2/test_2
 145 suite_1/test_1
 146 ]
 147     ]
 148   ]
 149
 150   [
 151     [Run multiple test units that are not necessarily siblings]
 152     [[pre __param_run_test__=suite_1/test_1:test_1]]
 153     [
 154 [pre
 155 suite_1/test_1
 156 test_1
 157 ]
 158     ]
 159   ]
 160
 161   [
 162     [Run all tests matching to a given label]
 163     [[pre __param_run_test__=@L1]]
 164     [
 165 [pre
 166 test_1
 167 test_2
 168 suite_1/test_1
 169 ]
 170     ]
 171   ]
 172
 173   [
 174     [Run every test case in the test tree]
 175     [[pre __param_run_test__=*]]
 176     [
 177 [pre
 178 test_1
 179 test_2
 180 suite_1/suite_1/test_1
 181 suite_1/suite_1/test_2
 182 suite_1/suite_2/test_1
 183 suite_1/suite_2/test_2
 184 suite_1/test_1
 185 suite_1/test_2
 186 suite_1/test_2A
 187 ]
 188     ]
 189   ]
 190
 191
 192   [
 193     [Run every test unit in a given suite with a given prefix]
 194     [[pre __param_run_test__=suite_1/test*]]
 195     [
 196 [pre
 197 suite_1/test_1
 198 suite_1/test_2
 199 suite_1/test_2A
 200 ]
 201     ]
 202   ]
 203
 204   [
 205     [Run every test unit in a given suite with a given suffix]
 206     [[pre __param_run_test__=suite_1/*_1]]
 207     [
 208 [pre
 209 suite_1/suite_1/test_1
 210 suite_1/suite_1/test_2
 211 suite_1/test_1
 212 ]
 213     ]
 214   ]
 215
 216
 217   [
 218     [Run every test unit in a given suite with a given infix]
 219     [[pre __param_run_test__=suite_1/\*_2\*]]
 220     [
 221 [pre
 222 suite_1/suite_2/test_1
 223 suite_1/suite_2/test_2
 224 suite_1/test_2
 225 suite_1/test_2A
 226 ]
 227     ]
 228   ]
 229
 230
 231   [
 232     [Run test(s) with given name in any N-level suite]
 233     [[pre __param_run_test__=\*/\*/test_2]]
 234     [
 235 [pre
 236 suite_1/suite_1/test_2
 237 suite_1/suite_2/test_2
 238 ]
 239     ]
 240   ]
 241 ]
 242
 243 For the syntax productions describing the structure of `<test_set>` value see [link boost_test.utf_reference.rt_param_reference.run_test here].
 244
 245 While using manual absolute test case specification ignores the default run status, it does not ignore the dynamic test dependencies.
 246 If test unit `B` depends on test unit `A` and test `B` is specified to be run by __param_run_test__, `A` is also run, even
 247 if it is not specified, and its failure may cause the execution of `B` to be skipped. Similarly, the failed check of
 248 the __decorator_precondition__ may cause the test selected test to be skipped.
 249
 250 [bt_example decorator_21..run_test and dynamic dependencies..run-fail]
 251
 252
 253 [#ref_command_line_control_enablers][h4 Relative specification]
 254
 255 Term 'relative' in this context means that the configuration is based on either the default run status of the test units or
 256 by the command-line override specified by the ['absolute specification]; and atop of this, we additionally either enable
 257 some disabled test units or disable some enabled tests units. The relative specification is controlled by command-line
 258 argument __param_run_test__, with the value using similar syntax as in the absolute specification, but preceded with
 259 either character `'!'` for disabling enabled test units or with character `'+'` for enabling the disabled test units.
 260 This can be summarized with the following table:
 261
 262 [table
 263  [ [command][specification type][semantics] ]
 264  [ [[pre > test_program --__param_run_test__=!<absolute_spec>]][disabler][Enabled test units that match `<absolute_spec>` become disabled.] ]
 265  [ [[pre > test_program --__param_run_test__=+<absolute_spec>]][enabler][Disabled test units that match `<absolute_spec>` as well as their upstream dependencies become enabled.] ]
 266 ]
 267
 268 The ['enabler] specification is used to enable a set of test units which are initially disabled.
 269 [bt_example decorator_22..command-line enabler..run]
 270
 271 Conversely, the ['disabler] specification is used to disable a set of test units which are initially enabled.
 272 [bt_example decorator_23..command-line disabler..run]
 273
 274 If there are both an enabler and disabler on one command line that specify the same test, the test becomes disabled. I.e.,
 275 the disabler takes the precedence over the enabler.
 276
 277 [note While enabler additionally enables the upstream dependencies (introduced with decorator __decorator_depends_on__),
 278  disabler does not disable them. Therefore when you enable and then disable the same test, you do not disable its upstream dependencies.]
 279
 280 [/----------------------------------------------]
 281
 282 [endsect] [/ section:test_unit_filtering]