src/doc/book/testing.md

   1 % Testing
   2
   3 > Program testing can be a very effective way to show the presence of bugs, but
   4 > it is hopelessly inadequate for showing their absence.
   5 >
   6 > Edsger W. Dijkstra, "The Humble Programmer" (1972)
   7
   8 Let's talk about how to test Rust code. What we will not be talking about is
   9 the right way to test Rust code. There are many schools of thought regarding
  10 the right and wrong way to write tests. All of these approaches use the same
  11 basic tools, and so we'll show you the syntax for using them.
  12
  13 # The `test` attribute
  14
  15 At its simplest, a test in Rust is a function that's annotated with the `test`
  16 attribute. Let's make a new project with Cargo called `adder`:
  17
  18 ```bash
  19 $ cargo new adder
  20 $ cd adder
  21 ```
  22
  23 Cargo will automatically generate a simple test when you make a new project.
  24 Here's the contents of `src/lib.rs`:
  25
  26 ```rust
  27 #[test]
  28 fn it_works() {
  29 }
  30 ```
  31
  32 Note the `#[test]`. This attribute indicates that this is a test function. It
  33 currently has no body. That's good enough to pass! We can run the tests with
  34 `cargo test`:
  35
  36 ```bash
  37 $ cargo test
  38    Compiling adder v0.0.1 (file:///home/you/projects/adder)
  39      Running target/adder-91b3e234d4ed382a
  40
  41 running 1 test
  42 test it_works ... ok
  43
  44 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
  45
  46    Doc-tests adder
  47
  48 running 0 tests
  49
  50 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
  51 ```
  52
  53 Cargo compiled and ran our tests. There are two sets of output here: one
  54 for the test we wrote, and another for documentation tests. We'll talk about
  55 those later. For now, see this line:
  56
  57 ```text
  58 test it_works ... ok
  59 ```
  60
  61 Note the `it_works`. This comes from the name of our function:
  62
  63 ```rust
  64 fn it_works() {
  65 # }
  66 ```
  67
  68 We also get a summary line:
  69
  70 ```text
  71 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
  72 ```
  73
  74 So why does our do-nothing test pass? Any test which doesn't `panic!` passes,
  75 and any test that does `panic!` fails. Let's make our test fail:
  76
  77 ```rust
  78 #[test]
  79 fn it_works() {
  80     assert!(false);
  81 }
  82 ```
  83
  84 `assert!` is a macro provided by Rust which takes one argument: if the argument
  85 is `true`, nothing happens. If the argument is `false`, it `panic!`s. Let's run
  86 our tests again:
  87
  88 ```bash
  89 $ cargo test
  90    Compiling adder v0.0.1 (file:///home/you/projects/adder)
  91      Running target/adder-91b3e234d4ed382a
  92
  93 running 1 test
  94 test it_works ... FAILED
  95
  96 failures:
  97
  98 ---- it_works stdout ----
  99         thread 'it_works' panicked at 'assertion failed: false', /home/steve/tmp/adder/src/lib.rs:3
 100
 101
 102
 103 failures:
 104     it_works
 105
 106 test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured
 107
 108 thread '<main>' panicked at 'Some tests failed', /home/steve/src/rust/src/libtest/lib.rs:247
 109 ```
 110
 111 Rust indicates that our test failed:
 112
 113 ```text
 114 test it_works ... FAILED
 115 ```
 116
 117 And that's reflected in the summary line:
 118
 119 ```text
 120 test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured
 121 ```
 122
 123 We also get a non-zero status code. We can use `$?` on OS X and Linux:
 124
 125 ```bash
 126 $ echo $?
 127 101
 128 ```
 129
 130 On Windows, if you’re using `cmd`:
 131
 132 ```bash
 133 > echo %ERRORLEVEL%
 134 ```
 135
 136 And if you’re using PowerShell:
 137
 138 ```bash
 139 > echo $LASTEXITCODE # the code itself
 140 > echo $? # a boolean, fail or succeed
 141 ```
 142
 143 This is useful if you want to integrate `cargo test` into other tooling.
 144
 145 We can invert our test's failure with another attribute: `should_panic`:
 146
 147 ```rust
 148 #[test]
 149 #[should_panic]
 150 fn it_works() {
 151     assert!(false);
 152 }
 153 ```
 154
 155 This test will now succeed if we `panic!` and fail if we complete. Let's try it:
 156
 157 ```bash
 158 $ cargo test
 159    Compiling adder v0.0.1 (file:///home/you/projects/adder)
 160      Running target/adder-91b3e234d4ed382a
 161
 162 running 1 test
 163 test it_works ... ok
 164
 165 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
 166
 167    Doc-tests adder
 168
 169 running 0 tests
 170
 171 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
 172 ```
 173
 174 Rust provides another macro, `assert_eq!`, that compares two arguments for
 175 equality:
 176
 177 ```rust
 178 #[test]
 179 #[should_panic]
 180 fn it_works() {
 181     assert_eq!("Hello", "world");
 182 }
 183 ```
 184
 185 Does this test pass or fail? Because of the `should_panic` attribute, it
 186 passes:
 187
 188 ```bash
 189 $ cargo test
 190    Compiling adder v0.0.1 (file:///home/you/projects/adder)
 191      Running target/adder-91b3e234d4ed382a
 192
 193 running 1 test
 194 test it_works ... ok
 195
 196 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
 197
 198    Doc-tests adder
 199
 200 running 0 tests
 201
 202 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
 203 ```
 204
 205 `should_panic` tests can be fragile, as it's hard to guarantee that the test
 206 didn't fail for an unexpected reason. To help with this, an optional `expected`
 207 parameter can be added to the `should_panic` attribute. The test harness will
 208 make sure that the failure message contains the provided text. A safer version
 209 of the example above would be:
 210
 211 ```rust
 212 #[test]
 213 #[should_panic(expected = "assertion failed")]
 214 fn it_works() {
 215     assert_eq!("Hello", "world");
 216 }
 217 ```
 218
 219 That's all there is to the basics! Let's write one 'real' test:
 220
 221 ```rust,ignore
 222 pub fn add_two(a: i32) -> i32 {
 223     a + 2
 224 }
 225
 226 #[test]
 227 fn it_works() {
 228     assert_eq!(4, add_two(2));
 229 }
 230 ```
 231
 232 This is a very common use of `assert_eq!`: call some function with
 233 some known arguments and compare it to the expected output.
 234
 235 # The `ignore` attribute
 236
 237 Sometimes a few specific tests can be very time-consuming to execute. These
 238 can be disabled by default by using the `ignore` attribute:
 239
 240 ```rust
 241 #[test]
 242 fn it_works() {
 243     assert_eq!(4, add_two(2));
 244 }
 245
 246 #[test]
 247 #[ignore]
 248 fn expensive_test() {
 249     // code that takes an hour to run
 250 }
 251 ```
 252
 253 Now we run our tests and see that `it_works` is run, but `expensive_test` is
 254 not:
 255
 256 ```bash
 257 $ cargo test
 258    Compiling adder v0.0.1 (file:///home/you/projects/adder)
 259      Running target/adder-91b3e234d4ed382a
 260
 261 running 2 tests
 262 test expensive_test ... ignored
 263 test it_works ... ok
 264
 265 test result: ok. 1 passed; 0 failed; 1 ignored; 0 measured
 266
 267    Doc-tests adder
 268
 269 running 0 tests
 270
 271 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
 272 ```
 273
 274 The expensive tests can be run explicitly using `cargo test -- --ignored`:
 275
 276 ```bash
 277 $ cargo test -- --ignored
 278      Running target/adder-91b3e234d4ed382a
 279
 280 running 1 test
 281 test expensive_test ... ok
 282
 283 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
 284
 285    Doc-tests adder
 286
 287 running 0 tests
 288
 289 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
 290 ```
 291
 292 The `--ignored` argument is an argument to the test binary, and not to Cargo,
 293 which is why the command is `cargo test -- --ignored`.
 294
 295 # The `tests` module
 296
 297 There is one way in which our existing example is not idiomatic: it's
 298 missing the `tests` module. The idiomatic way of writing our example
 299 looks like this:
 300
 301 ```rust,ignore
 302 pub fn add_two(a: i32) -> i32 {
 303     a + 2
 304 }
 305
 306 #[cfg(test)]
 307 mod tests {
 308     use super::add_two;
 309
 310     #[test]
 311     fn it_works() {
 312         assert_eq!(4, add_two(2));
 313     }
 314 }
 315 ```
 316
 317 There's a few changes here. The first is the introduction of a `mod tests` with
 318 a `cfg` attribute. The module allows us to group all of our tests together, and
 319 to also define helper functions if needed, that don't become a part of the rest
 320 of our crate. The `cfg` attribute only compiles our test code if we're
 321 currently trying to run the tests. This can save compile time, and also ensures
 322 that our tests are entirely left out of a normal build.
 323
 324 The second change is the `use` declaration. Because we're in an inner module,
 325 we need to bring our test function into scope. This can be annoying if you have
 326 a large module, and so this is a common use of globs. Let's change our
 327 `src/lib.rs` to make use of it:
 328
 329 ```rust,ignore
 330 pub fn add_two(a: i32) -> i32 {
 331     a + 2
 332 }
 333
 334 #[cfg(test)]
 335 mod tests {
 336     use super::*;
 337
 338     #[test]
 339     fn it_works() {
 340         assert_eq!(4, add_two(2));
 341     }
 342 }
 343 ```
 344
 345 Note the different `use` line. Now we run our tests:
 346
 347 ```bash
 348 $ cargo test
 349     Updating registry `https://github.com/rust-lang/crates.io-index`
 350    Compiling adder v0.0.1 (file:///home/you/projects/adder)
 351      Running target/adder-91b3e234d4ed382a
 352
 353 running 1 test
 354 test tests::it_works ... ok
 355
 356 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
 357
 358    Doc-tests adder
 359
 360 running 0 tests
 361
 362 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
 363 ```
 364
 365 It works!
 366
 367 The current convention is to use the `tests` module to hold your "unit-style"
 368 tests. Anything that just tests one small bit of functionality makes sense to
 369 go here. But what about "integration-style" tests instead? For that, we have
 370 the `tests` directory.
 371
 372 # The `tests` directory
 373
 374 To write an integration test, let's make a `tests` directory, and
 375 put a `tests/lib.rs` file inside, with this as its contents:
 376
 377 ```rust,ignore
 378 extern crate adder;
 379
 380 #[test]
 381 fn it_works() {
 382     assert_eq!(4, adder::add_two(2));
 383 }
 384 ```
 385
 386 This looks similar to our previous tests, but slightly different. We now have
 387 an `extern crate adder` at the top. This is because the tests in the `tests`
 388 directory are an entirely separate crate, and so we need to import our library.
 389 This is also why `tests` is a suitable place to write integration-style tests:
 390 they use the library like any other consumer of it would.
 391
 392 Let's run them:
 393
 394 ```bash
 395 $ cargo test
 396    Compiling adder v0.0.1 (file:///home/you/projects/adder)
 397      Running target/adder-91b3e234d4ed382a
 398
 399 running 1 test
 400 test tests::it_works ... ok
 401
 402 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
 403
 404      Running target/lib-c18e7d3494509e74
 405
 406 running 1 test
 407 test it_works ... ok
 408
 409 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
 410
 411    Doc-tests adder
 412
 413 running 0 tests
 414
 415 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
 416 ```
 417
 418 Now we have three sections: our previous test is also run, as well as our new
 419 one.
 420
 421 That's all there is to the `tests` directory. The `tests` module isn't needed
 422 here, since the whole thing is focused on tests.
 423
 424 Let's finally check out that third section: documentation tests.
 425
 426 # Documentation tests
 427
 428 Nothing is better than documentation with examples. Nothing is worse than
 429 examples that don't actually work, because the code has changed since the
 430 documentation has been written. To this end, Rust supports automatically
 431 running examples in your documentation (**note:** this only works in library
 432 crates, not binary crates). Here's a fleshed-out `src/lib.rs` with examples:
 433
 434 ```rust,ignore
 435 //! The `adder` crate provides functions that add numbers to other numbers.
 436 //!
 437 //! # Examples
 438 //!
 439 //! ```
 440 //! assert_eq!(4, adder::add_two(2));
 441 //! ```
 442
 443 /// This function adds two to its argument.
 444 ///
 445 /// # Examples
 446 ///
 447 /// ```
 448 /// use adder::add_two;
 449 ///
 450 /// assert_eq!(4, add_two(2));
 451 /// ```
 452 pub fn add_two(a: i32) -> i32 {
 453     a + 2
 454 }
 455
 456 #[cfg(test)]
 457 mod tests {
 458     use super::*;
 459
 460     #[test]
 461     fn it_works() {
 462         assert_eq!(4, add_two(2));
 463     }
 464 }
 465 ```
 466
 467 Note the module-level documentation with `//!` and the function-level
 468 documentation with `///`. Rust's documentation supports Markdown in comments,
 469 and so triple graves mark code blocks. It is conventional to include the
 470 `# Examples` section, exactly like that, with examples following.
 471
 472 Let's run the tests again:
 473
 474 ```bash
 475 $ cargo test
 476    Compiling adder v0.0.1 (file:///home/steve/tmp/adder)
 477      Running target/adder-91b3e234d4ed382a
 478
 479 running 1 test
 480 test tests::it_works ... ok
 481
 482 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
 483
 484      Running target/lib-c18e7d3494509e74
 485
 486 running 1 test
 487 test it_works ... ok
 488
 489 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
 490
 491    Doc-tests adder
 492
 493 running 2 tests
 494 test add_two_0 ... ok
 495 test _0 ... ok
 496
 497 test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured
 498 ```
 499
 500 Now we have all three kinds of tests running! Note the names of the
 501 documentation tests: the `_0` is generated for the module test, and `add_two_0`
 502 for the function test. These will auto increment with names like `add_two_1` as
 503 you add more examples.
 504
 505 We haven’t covered all of the details with writing documentation tests. For more,
 506 please see the [Documentation chapter](documentation.html)
 507
 508 One final note: documentation tests *cannot* be run on binary crates.
 509 To see more on file arrangement see the [Crates and
 510 Modules](crates-and-modules.html) section.