]>
Commit | Line | Data |
---|---|---|
7c673cae FG |
1 | [/ |
2 | / Copyright (c) 2003 Boost.Test contributors | |
3 | / | |
4 | / Distributed under the Boost Software License, Version 1.0. (See accompanying | |
5 | / file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt) | |
6 | /] | |
7 | ||
8 | [section:expected_failures Expected failures specification] | |
9 | ||
10 | While in a perfect world all test assertions should pass in order for a test module to pass, in some situations | |
11 | it is desirable to temporarily allow particular tests to fail. For example, where a particular feature is not | |
12 | implemented yet and one needs to prepare a library for the release or when particular test fails on some | |
13 | platforms. To avoid a nagging red box in regression tests table, you can use the expected failures feature. | |
14 | ||
15 | ||
16 | This feature allows specifying an expected number of failed assertions per test unit. The value is specified | |
17 | during test tree construction, and can't be updated during test execution. | |
18 | ||
19 | The feature is not intended to be used to check for expected functionality failures. To check that a particular | |
20 | input is causing an exception to be thrown use __BOOST_LEVEL_THROW__ family of testing | |
21 | tools. | |
22 | ||
23 | The usage of this feature should be limited and employed only after careful consideration. In general you should | |
24 | only use this feature when it is necessary to force a test module to pass without actually fixing the problem. | |
25 | Obviously, an excessive usage of expected failures defeats the purpose of the unit test. In most cases it only | |
26 | needs be applied temporarily. | |
27 | ||
28 | You also need to remember that the expected failure specification is per test case. This means that any failed | |
29 | assertion within that test case can satisfy the expected failures quota. Meaning it is possible for an | |
30 | unexpected failure to occur to satisfy this quota. | |
31 | ||
32 | ||
33 | [note If an assertion at fault is fixed and passed while an expected failures specification still present, | |
34 | the number of failures becomes smaller than expected. The test is going to be reported as passed; instead, | |
35 | a warning message will be issued. | |
36 | ] | |
37 | ||
38 | ||
39 | [/-----------------------------------------------------------------] | |
40 | [#l_expected_failure][h3 Expected failure specification] | |
41 | ||
42 | The decorator __decorator_expected_failures__ defines the number of assertions that are expected to fail within the corresponding test | |
43 | unit. It is reported as failure when the number of failed assertions is greater than the declared expected number of | |
44 | failures. If the number of failed assertions is less than the number of expected failures a message is reported. The | |
45 | total number of expected failures for a given test suite `S` is the sum of the declared expected failures in `S` and the | |
46 | sum of expected failures in all nested test units: | |
47 | ||
48 | [bt_example decorator_10..decorator expected_failures..run-fail] | |
49 | ||
50 | In the above example, we first run all test cases with four failed assertions. The total number of expected failures | |
51 | is 3: 1 (for test `suite1`) + 2 (for `test1`). Because the expected failure count is exceeded, the error is reported. | |
52 | In the second case, we only run test case `suite1/test1`: two failures occur, two failures are expected, therefore no | |
53 | error is reported. | |
54 | ||
55 | ||
56 | ||
57 | ||
58 | [/-----------------------------------------------------------------] | |
59 | [h3 Usage with automatically registered test cases] | |
60 | ||
61 | [caution this usage is considered as deprecated. Please consider using the [link l_expected_failure `expected_failures`] | |
62 | decorator instead.] | |
63 | ||
64 | For backwards compatibility, it is possible to indicate the expected failures with | |
65 | __BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES__ [footnote deprecated] before the test case definition. | |
66 | ||
67 | `` | |
68 | BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES(test_case_name, number_of_expected_failures); | |
69 | `` | |
70 | ||
71 | You can use this macro both on a file scope and inside a test suite. Moreover you can use it even if name of test | |
72 | units coincide in different test suites. Expected failures specification applies to the test unit belonging to the same | |
73 | test suite where __BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES__ resides. | |
74 | ||
75 | [bt_example example17..Expected failures specification for automatically registered test case..run] | |
76 | ||
77 | ||
78 | [/-----------------------------------------------------------------] | |
79 | [h3 Usage with manually registered test cases] | |
80 | ||
81 | [caution this usage is considered as deprecated. Please consider using the [link l_expected_failure `expected_failures`] | |
82 | decorator instead.] | |
83 | ||
84 | To set the value of expected failures for the manually registered test unit pass it as a second argument for the | |
85 | [link ref_test_case_registration `test_suite::add`] call during test unit registration. | |
86 | ||
87 | [bt_example example16..Expected failures specification for manually registered test case..run] | |
88 | ||
89 | [endsect] [/ expected failures] | |
90 | ||
91 | [/EOF] |