]>
Commit | Line | Data |
---|---|---|
7c673cae FG |
1 | [/ |
2 | / Copyright (c) 2003 Boost.Test contributors | |
3 | / | |
4 | / Distributed under the Boost Software License, Version 1.0. (See accompanying | |
5 | / file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt) | |
6 | /] | |
7 | ||
8 | ||
9 | [section:test_unit_filtering Test unit filtering] | |
10 | ||
11 | The __UTF__ offers a number of ways to run only a subset of all test cases registered in the test tree. | |
12 | ||
13 | [#ref_default_run_status][h3 Default run status] | |
14 | ||
15 | Each test unit (either test case or test suite) has an associated ['default run status]. It can assume one of the three values: | |
16 | ||
17 | # ['true] -- this means that, unless some runtime parameters specify otherwise, the test unit is designated to be run. | |
18 | # ['false] -- this means that, unless some runtime parameters specify otherwise, the test unit is designated ['not] to be run. | |
19 | # ['inherit] -- this means that the test unit's default run status is the same as that of its immediate parent test unit. This is applied recursively. | |
20 | ||
21 | Initially, the master test suite has default run status set to ['true]. All other test units have default run status set to ['inherit]. | |
22 | This implies that, unless any additional configuration is applied, all tests are designated to be run. | |
23 | ||
24 | You can set a different default run status in any test unit by using [link boost_test.tests_organization.decorators decorators]: | |
25 | __decorator_disabled__, __decorator_enabled__ and __decorator_enable_if__. The default run status is set once, upon testing program | |
26 | initialization, and cannot be changed. The disabled tests are not executed by default, but are still present in the test tree, | |
27 | and are listed along with other tests when you use command-line argument | |
28 | [link boost_test.utf_reference.rt_param_reference.list_content `list_content`]. | |
29 | ||
30 | [bt_example decorator_20..default run status..run-fail] | |
31 | ||
32 | [#ref_dynamic_test_dependency][h3 Dynamic test dependencies] | |
33 | ||
34 | Additionally, it is possible to statically associate a test unit with a condition. This associated condition is evaluated immediately | |
35 | before executing the test unit. If the condition is met, the test unit is executed. Otherwise, the test unit is ['skipped]. | |
36 | It is possible to add two dependencies: | |
37 | ||
38 | # Upon another test unit. In this case the decorated test case is skipped if the test unit specified in the dependency is either | |
39 | failed or skipped or disabled. This can be declared with decorator __decorator_depends_on__. | |
40 | # Upon an arbitrary predicate. This can be declared with decorator __decorator_precondition__. | |
41 | ||
42 | [#ref_command_line_control][h3 Command-line control] | |
43 | ||
44 | Static configuration of the test-case filtering is used by default, unless command-line filtering is applied. With command-line argument | |
45 | __param_run_test__ it is possible to alter the static pre-set in a number of ways: | |
46 | ||
47 | # Ignore the static configuration and manually specify test cases to be run. | |
48 | # Augment the statically defined set by enabling the disabled test cases. | |
49 | # Shrink the statically defined set by disabling some of the enabled test cases. | |
50 | ||
51 | [#ref_command_line_control_absolute][h4 Absolute specification] | |
52 | ||
53 | Term 'absolute' in this context means that the default run status of the test units, which has been statically set up, | |
54 | is completely ignored and the tests to be run are specified manually from scratch. First, in order to learn what test | |
55 | units are registered in the test tree the user needs to use command-line argument | |
56 | [link boost_test.utf_reference.rt_param_reference.list_content `list_content`]. | |
57 | Next, in order to specify a set of test cases, the user needs to use command-line argument | |
58 | __param_run_test__ with absolute value: | |
59 | ||
60 | ``` | |
61 | > test_program --__param_run_test__=<test_set> | |
62 | ``` | |
63 | ||
64 | The format of `<test_set>` value can assume a number of forms. Given the following program: | |
65 | ||
66 | ``` | |
67 | #define __BOOST_TEST_MODULE__ example | |
68 | #include <boost/test/included/unit_test.hpp> | |
69 | ||
70 | using boost::unit_test::__decorator_label__; | |
71 | ||
72 | __BOOST_AUTO_TEST_CASE__(test_1, *label("L1")) {} | |
73 | BOOST_AUTO_TEST_CASE(test_2, *label("L1")) {} | |
74 | ||
75 | __BOOST_AUTO_TEST_SUITE__(suite_1) | |
76 | ||
77 | BOOST_AUTO_TEST_SUITE(suite_1) | |
78 | BOOST_AUTO_TEST_CASE(test_1) {} | |
79 | BOOST_AUTO_TEST_CASE(test_2) {} | |
80 | BOOST_AUTO_TEST_SUITE_END() | |
81 | ||
82 | BOOST_AUTO_TEST_SUITE(suite_2) | |
83 | BOOST_AUTO_TEST_CASE(test_1, *label("L2")) {} | |
84 | BOOST_AUTO_TEST_CASE(test_2, *label("L2")) {} | |
85 | BOOST_AUTO_TEST_SUITE_END() | |
86 | ||
87 | BOOST_AUTO_TEST_CASE(test_1, *label("L1")) {} | |
88 | BOOST_AUTO_TEST_CASE(test_2) {} | |
89 | BOOST_AUTO_TEST_CASE(test_2A) {} | |
90 | ||
91 | BOOST_AUTO_TEST_SUITE_END() | |
92 | ``` | |
93 | ||
94 | The following table illustrates how different values of `<test_set>` control which test cases ware run. | |
95 | ||
96 | [table | |
97 | [ | |
98 | [Description] | |
99 | [Parameter value] | |
100 | [Test cases run] | |
101 | ] | |
102 | ||
103 | [ | |
104 | [Run single top-level test case by name] | |
105 | [[pre __param_run_test__=test_1]] | |
106 | [ | |
107 | [pre | |
108 | test_1 | |
109 | ] | |
110 | ] | |
111 | ] | |
112 | ||
113 | [ | |
114 | [Run single nested test case by name] | |
115 | [[pre __param_run_test__=suite_1/suite_1/test_1]] | |
116 | [ | |
117 | [pre | |
118 | suite_1/suite_1/test_1 | |
119 | ] | |
120 | ] | |
121 | ] | |
122 | ||
123 | [ | |
124 | [Run single test suite by name] | |
125 | [ | |
126 | [pre __param_run_test__=suite_1/suite_2 | |
127 | __param_run_test__=suite_1/suite_2/* | |
128 | ] | |
129 | ] | |
130 | [ | |
131 | [pre | |
132 | suite_1/suite_2/test_1 | |
133 | suite_1/suite_2/test_2 | |
134 | ] | |
135 | ] | |
136 | ] | |
137 | ||
138 | [ | |
139 | [Run multiple test units that are *siblings* of the same test suite] | |
140 | [[pre __param_run_test__=suite_1/test_1,suite_2]] | |
141 | [ | |
142 | [pre | |
143 | suite_1/suite_2/test_1 | |
144 | suite_1/suite_2/test_2 | |
145 | suite_1/test_1 | |
146 | ] | |
147 | ] | |
148 | ] | |
149 | ||
150 | [ | |
151 | [Run multiple test units that are not necessarily siblings] | |
152 | [[pre __param_run_test__=suite_1/test_1:test_1]] | |
153 | [ | |
154 | [pre | |
155 | suite_1/test_1 | |
156 | test_1 | |
157 | ] | |
158 | ] | |
159 | ] | |
160 | ||
161 | [ | |
162 | [Run all tests matching to a given label] | |
163 | [[pre __param_run_test__=@L1]] | |
164 | [ | |
165 | [pre | |
166 | test_1 | |
167 | test_2 | |
168 | suite_1/test_1 | |
169 | ] | |
170 | ] | |
171 | ] | |
172 | ||
173 | [ | |
174 | [Run every test case in the test tree] | |
175 | [[pre __param_run_test__=*]] | |
176 | [ | |
177 | [pre | |
178 | test_1 | |
179 | test_2 | |
180 | suite_1/suite_1/test_1 | |
181 | suite_1/suite_1/test_2 | |
182 | suite_1/suite_2/test_1 | |
183 | suite_1/suite_2/test_2 | |
184 | suite_1/test_1 | |
185 | suite_1/test_2 | |
186 | suite_1/test_2A | |
187 | ] | |
188 | ] | |
189 | ] | |
190 | ||
191 | ||
192 | [ | |
193 | [Run every test unit in a given suite with a given prefix] | |
194 | [[pre __param_run_test__=suite_1/test*]] | |
195 | [ | |
196 | [pre | |
197 | suite_1/test_1 | |
198 | suite_1/test_2 | |
199 | suite_1/test_2A | |
200 | ] | |
201 | ] | |
202 | ] | |
203 | ||
204 | [ | |
205 | [Run every test unit in a given suite with a given suffix] | |
206 | [[pre __param_run_test__=suite_1/*_1]] | |
207 | [ | |
208 | [pre | |
209 | suite_1/suite_1/test_1 | |
210 | suite_1/suite_1/test_2 | |
211 | suite_1/test_1 | |
212 | ] | |
213 | ] | |
214 | ] | |
215 | ||
216 | ||
217 | [ | |
218 | [Run every test unit in a given suite with a given infix] | |
219 | [[pre __param_run_test__=suite_1/\*_2\*]] | |
220 | [ | |
221 | [pre | |
222 | suite_1/suite_2/test_1 | |
223 | suite_1/suite_2/test_2 | |
224 | suite_1/test_2 | |
225 | suite_1/test_2A | |
226 | ] | |
227 | ] | |
228 | ] | |
229 | ||
230 | ||
231 | [ | |
232 | [Run test(s) with given name in any N-level suite] | |
233 | [[pre __param_run_test__=\*/\*/test_2]] | |
234 | [ | |
235 | [pre | |
236 | suite_1/suite_1/test_2 | |
237 | suite_1/suite_2/test_2 | |
238 | ] | |
239 | ] | |
240 | ] | |
241 | ] | |
242 | ||
243 | For the syntax productions describing the structure of `<test_set>` value see [link boost_test.utf_reference.rt_param_reference.run_test here]. | |
244 | ||
245 | While using manual absolute test case specification ignores the default run status, it does not ignore the dynamic test dependencies. | |
246 | If test unit `B` depends on test unit `A` and test `B` is specified to be run by __param_run_test__, `A` is also run, even | |
247 | if it is not specified, and its failure may cause the execution of `B` to be skipped. Similarly, the failed check of | |
248 | the __decorator_precondition__ may cause the test selected test to be skipped. | |
249 | ||
250 | [bt_example decorator_21..run_test and dynamic dependencies..run-fail] | |
251 | ||
252 | ||
253 | [#ref_command_line_control_enablers][h4 Relative specification] | |
254 | ||
255 | Term 'relative' in this context means that the configuration is based on either the default run status of the test units or | |
256 | by the command-line override specified by the ['absolute specification]; and atop of this, we additionally either enable | |
257 | some disabled test units or disable some enabled tests units. The relative specification is controlled by command-line | |
258 | argument __param_run_test__, with the value using similar syntax as in the absolute specification, but preceded with | |
259 | either character `'!'` for disabling enabled test units or with character `'+'` for enabling the disabled test units. | |
260 | This can be summarized with the following table: | |
261 | ||
262 | [table | |
263 | [ [command][specification type][semantics] ] | |
264 | [ [[pre > test_program --__param_run_test__=!<absolute_spec>]][disabler][Enabled test units that match `<absolute_spec>` become disabled.] ] | |
265 | [ [[pre > test_program --__param_run_test__=+<absolute_spec>]][enabler][Disabled test units that match `<absolute_spec>` as well as their upstream dependencies become enabled.] ] | |
266 | ] | |
267 | ||
268 | The ['enabler] specification is used to enable a set of test units which are initially disabled. | |
269 | [bt_example decorator_22..command-line enabler..run] | |
270 | ||
271 | Conversely, the ['disabler] specification is used to disable a set of test units which are initially enabled. | |
272 | [bt_example decorator_23..command-line disabler..run] | |
273 | ||
274 | If there are both an enabler and disabler on one command line that specify the same test, the test becomes disabled. I.e., | |
275 | the disabler takes the precedence over the enabler. | |
276 | ||
277 | [note While enabler additionally enables the upstream dependencies (introduced with decorator __decorator_depends_on__), | |
278 | disabler does not disable them. Therefore when you enable and then disable the same test, you do not disable its upstream dependencies.] | |
279 | ||
280 | [/----------------------------------------------] | |
281 | ||
282 | [endsect] [/ section:test_unit_filtering] |