src/doc/book/src/ch11-02-running-tests.md

   1 ## Controlling How Tests Are Run
   2
   3 Just as `cargo run` compiles your code and then runs the resulting binary,
   4 `cargo test` compiles your code in test mode and runs the resulting test
   5 binary. The default behavior of the binary produced by `cargo test` is to run
   6 all the tests in parallel and capture output generated during test runs,
   7 preventing the output from being displayed and making it easier to read the
   8 output related to the test results. You can, however, specify command line
   9 options to change this default behavior.
  10
  11 Some command line options go to `cargo test`, and some go to the resulting test
  12 binary. To separate these two types of arguments, you list the arguments that
  13 go to `cargo test` followed by the separator `--` and then the ones that go to
  14 the test binary. Running `cargo test --help` displays the options you can use
  15 with `cargo test`, and running `cargo test -- --help` displays the options you
  16 can use after the separator.
  17
  18 ### Running Tests in Parallel or Consecutively
  19
  20 When you run multiple tests, by default they run in parallel using threads,
  21 meaning they finish running faster and you get feedback quicker. Because the
  22 tests are running at the same time, you must make sure your tests don’t depend
  23 on each other or on any shared state, including a shared environment, such as
  24 the current working directory or environment variables.
  25
  26 For example, say each of your tests runs some code that creates a file on disk
  27 named *test-output.txt* and writes some data to that file. Then each test reads
  28 the data in that file and asserts that the file contains a particular value,
  29 which is different in each test. Because the tests run at the same time, one
  30 test might overwrite the file in the time between another test writing and
  31 reading the file. The second test will then fail, not because the code is
  32 incorrect but because the tests have interfered with each other while running
  33 in parallel. One solution is to make sure each test writes to a different file;
  34 another solution is to run the tests one at a time.
  35
  36 If you don’t want to run the tests in parallel or if you want more fine-grained
  37 control over the number of threads used, you can send the `--test-threads` flag
  38 and the number of threads you want to use to the test binary. Take a look at
  39 the following example:
  40
  41 ```console
  42 $ cargo test -- --test-threads=1
  43 ```
  44
  45 We set the number of test threads to `1`, telling the program not to use any
  46 parallelism. Running the tests using one thread will take longer than running
  47 them in parallel, but the tests won’t interfere with each other if they share
  48 state.
  49
  50 ### Showing Function Output
  51
  52 By default, if a test passes, Rust’s test library captures anything printed to
  53 standard output. For example, if we call `println!` in a test and the test
  54 passes, we won’t see the `println!` output in the terminal; we’ll see only the
  55 line that indicates the test passed. If a test fails, we’ll see whatever was
  56 printed to standard output with the rest of the failure message.
  57
  58 As an example, Listing 11-10 has a silly function that prints the value of its
  59 parameter and returns 10, as well as a test that passes and a test that fails.
  60
  61 <span class="filename">Filename: src/lib.rs</span>
  62
  63 ```rust,panics,noplayground
  64 {{#rustdoc_include ../listings/ch11-writing-automated-tests/listing-11-10/src/lib.rs}}
  65 ```
  66
  67 <span class="caption">Listing 11-10: Tests for a function that calls
  68 `println!`</span>
  69
  70 When we run these tests with `cargo test`, we’ll see the following output:
  71
  72 ```console
  73 {{#include ../listings/ch11-writing-automated-tests/listing-11-10/output.txt}}
  74 ```
  75
  76 Note that nowhere in this output do we see `I got the value 4`, which is what
  77 is printed when the test that passes runs. That output has been captured. The
  78 output from the test that failed, `I got the value 8`, appears in the section
  79 of the test summary output, which also shows the cause of the test failure.
  80
  81 If we want to see printed values for passing tests as well, we can tell Rust
  82 to also show the output of successful tests with `--show-output`.
  83
  84 ```console
  85 $ cargo test -- --show-output
  86 ```
  87
  88 When we run the tests in Listing 11-10 again with the `--show-output` flag, we
  89 see the following output:
  90
  91 ```console
  92 {{#include ../listings/ch11-writing-automated-tests/output-only-01-show-output/output.txt}}
  93 ```
  94
  95 ### Running a Subset of Tests by Name
  96
  97 Sometimes, running a full test suite can take a long time. If you’re working on
  98 code in a particular area, you might want to run only the tests pertaining to
  99 that code. You can choose which tests to run by passing `cargo test` the name
 100 or names of the test(s) you want to run as an argument.
 101
 102 To demonstrate how to run a subset of tests, we’ll first create three tests for
 103 our `add_two` function, as shown in Listing 11-11, and choose which ones to run.
 104
 105 <span class="filename">Filename: src/lib.rs</span>
 106
 107 ```rust,noplayground
 108 {{#rustdoc_include ../listings/ch11-writing-automated-tests/listing-11-11/src/lib.rs}}
 109 ```
 110
 111 <span class="caption">Listing 11-11: Three tests with three different
 112 names</span>
 113
 114 If we run the tests without passing any arguments, as we saw earlier, all the
 115 tests will run in parallel:
 116
 117 ```console
 118 {{#include ../listings/ch11-writing-automated-tests/listing-11-11/output.txt}}
 119 ```
 120
 121 #### Running Single Tests
 122
 123 We can pass the name of any test function to `cargo test` to run only that test:
 124
 125 ```console
 126 {{#include ../listings/ch11-writing-automated-tests/output-only-02-single-test/output.txt}}
 127 ```
 128
 129 Only the test with the name `one_hundred` ran; the other two tests didn’t match
 130 that name. The test output lets us know we had more tests that didn’t run by
 131 displaying `2 filtered out` at the end.
 132
 133 We can’t specify the names of multiple tests in this way; only the first value
 134 given to `cargo test` will be used. But there is a way to run multiple tests.
 135
 136 #### Filtering to Run Multiple Tests
 137
 138 We can specify part of a test name, and any test whose name matches that value
 139 will be run. For example, because two of our tests’ names contain `add`, we can
 140 run those two by running `cargo test add`:
 141
 142 ```console
 143 {{#include ../listings/ch11-writing-automated-tests/output-only-03-multiple-tests/output.txt}}
 144 ```
 145
 146 This command ran all tests with `add` in the name and filtered out the test
 147 named `one_hundred`. Also note that the module in which a test appears becomes
 148 part of the test’s name, so we can run all the tests in a module by filtering
 149 on the module’s name.
 150
 151 ### Ignoring Some Tests Unless Specifically Requested
 152
 153 Sometimes a few specific tests can be very time-consuming to execute, so you
 154 might want to exclude them during most runs of `cargo test`. Rather than
 155 listing as arguments all tests you do want to run, you can instead annotate the
 156 time-consuming tests using the `ignore` attribute to exclude them, as shown
 157 here:
 158
 159 <span class="filename">Filename: src/lib.rs</span>
 160
 161 ```rust,noplayground
 162 {{#rustdoc_include ../listings/ch11-writing-automated-tests/no-listing-11-ignore-a-test/src/lib.rs}}
 163 ```
 164
 165 After `#[test]` we add the `#[ignore]` line to the test we want to exclude. Now
 166 when we run our tests, `it_works` runs, but `expensive_test` doesn’t:
 167
 168 ```console
 169 {{#include ../listings/ch11-writing-automated-tests/no-listing-11-ignore-a-test/output.txt}}
 170 ```
 171
 172 The `expensive_test` function is listed as `ignored`. If we want to run only
 173 the ignored tests, we can use `cargo test -- --ignored`:
 174
 175 ```console
 176 {{#include ../listings/ch11-writing-automated-tests/output-only-04-running-ignored/output.txt}}
 177 ```
 178
 179 By controlling which tests run, you can make sure your `cargo test` results
 180 will be fast. When you’re at a point where it makes sense to check the results
 181 of the `ignored` tests and you have time to wait for the results, you can run
 182 `cargo test -- --ignored` instead. If you want to run all tests whether they’re
 183 ignored or not, you can run `cargo test -- --include-ignored`.