src/doc/book/testing.md

   1 % Testing
   2
   3 > Program testing can be a very effective way to show the presence of bugs, but
   4 > it is hopelessly inadequate for showing their absence.
   5 >
   6 > Edsger W. Dijkstra, "The Humble Programmer" (1972)
   7
   8 Let's talk about how to test Rust code. What we will not be talking about is
   9 the right way to test Rust code. There are many schools of thought regarding
  10 the right and wrong way to write tests. All of these approaches use the same
  11 basic tools, and so we'll show you the syntax for using them.
  12
  13 # The `test` attribute
  14
  15 At its simplest, a test in Rust is a function that's annotated with the `test`
  16 attribute. Let's make a new project with Cargo called `adder`:
  17
  18 ```bash
  19 $ cargo new adder
  20 $ cd adder
  21 ```
  22
  23 Cargo will automatically generate a simple test when you make a new project.
  24 Here's the contents of `src/lib.rs`:
  25
  26 ```rust
  27 # fn main() {}
  28 #[test]
  29 fn it_works() {
  30 }
  31 ```
  32
  33 Note the `#[test]`. This attribute indicates that this is a test function. It
  34 currently has no body. That's good enough to pass! We can run the tests with
  35 `cargo test`:
  36
  37 ```bash
  38 $ cargo test
  39    Compiling adder v0.0.1 (file:///home/you/projects/adder)
  40      Running target/adder-91b3e234d4ed382a
  41
  42 running 1 test
  43 test it_works ... ok
  44
  45 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
  46
  47    Doc-tests adder
  48
  49 running 0 tests
  50
  51 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
  52 ```
  53
  54 Cargo compiled and ran our tests. There are two sets of output here: one
  55 for the test we wrote, and another for documentation tests. We'll talk about
  56 those later. For now, see this line:
  57
  58 ```text
  59 test it_works ... ok
  60 ```
  61
  62 Note the `it_works`. This comes from the name of our function:
  63
  64 ```rust
  65 fn it_works() {
  66 # }
  67 ```
  68
  69 We also get a summary line:
  70
  71 ```text
  72 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
  73 ```
  74
  75 So why does our do-nothing test pass? Any test which doesn't `panic!` passes,
  76 and any test that does `panic!` fails. Let's make our test fail:
  77
  78 ```rust
  79 # fn main() {}
  80 #[test]
  81 fn it_works() {
  82     assert!(false);
  83 }
  84 ```
  85
  86 `assert!` is a macro provided by Rust which takes one argument: if the argument
  87 is `true`, nothing happens. If the argument is `false`, it `panic!`s. Let's run
  88 our tests again:
  89
  90 ```bash
  91 $ cargo test
  92    Compiling adder v0.0.1 (file:///home/you/projects/adder)
  93      Running target/adder-91b3e234d4ed382a
  94
  95 running 1 test
  96 test it_works ... FAILED
  97
  98 failures:
  99
 100 ---- it_works stdout ----
 101         thread 'it_works' panicked at 'assertion failed: false', /home/steve/tmp/adder/src/lib.rs:3
 102
 103
 104
 105 failures:
 106     it_works
 107
 108 test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured
 109
 110 thread '<main>' panicked at 'Some tests failed', /home/steve/src/rust/src/libtest/lib.rs:247
 111 ```
 112
 113 Rust indicates that our test failed:
 114
 115 ```text
 116 test it_works ... FAILED
 117 ```
 118
 119 And that's reflected in the summary line:
 120
 121 ```text
 122 test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured
 123 ```
 124
 125 We also get a non-zero status code. We can use `$?` on OS X and Linux:
 126
 127 ```bash
 128 $ echo $?
 129 101
 130 ```
 131
 132 On Windows, if you’re using `cmd`:
 133
 134 ```bash
 135 > echo %ERRORLEVEL%
 136 ```
 137
 138 And if you’re using PowerShell:
 139
 140 ```bash
 141 > echo $LASTEXITCODE # the code itself
 142 > echo $? # a boolean, fail or succeed
 143 ```
 144
 145 This is useful if you want to integrate `cargo test` into other tooling.
 146
 147 We can invert our test's failure with another attribute: `should_panic`:
 148
 149 ```rust
 150 # fn main() {}
 151 #[test]
 152 #[should_panic]
 153 fn it_works() {
 154     assert!(false);
 155 }
 156 ```
 157
 158 This test will now succeed if we `panic!` and fail if we complete. Let's try it:
 159
 160 ```bash
 161 $ cargo test
 162    Compiling adder v0.0.1 (file:///home/you/projects/adder)
 163      Running target/adder-91b3e234d4ed382a
 164
 165 running 1 test
 166 test it_works ... ok
 167
 168 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
 169
 170    Doc-tests adder
 171
 172 running 0 tests
 173
 174 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
 175 ```
 176
 177 Rust provides another macro, `assert_eq!`, that compares two arguments for
 178 equality:
 179
 180 ```rust
 181 # fn main() {}
 182 #[test]
 183 #[should_panic]
 184 fn it_works() {
 185     assert_eq!("Hello", "world");
 186 }
 187 ```
 188
 189 Does this test pass or fail? Because of the `should_panic` attribute, it
 190 passes:
 191
 192 ```bash
 193 $ cargo test
 194    Compiling adder v0.0.1 (file:///home/you/projects/adder)
 195      Running target/adder-91b3e234d4ed382a
 196
 197 running 1 test
 198 test it_works ... ok
 199
 200 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
 201
 202    Doc-tests adder
 203
 204 running 0 tests
 205
 206 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
 207 ```
 208
 209 `should_panic` tests can be fragile, as it's hard to guarantee that the test
 210 didn't fail for an unexpected reason. To help with this, an optional `expected`
 211 parameter can be added to the `should_panic` attribute. The test harness will
 212 make sure that the failure message contains the provided text. A safer version
 213 of the example above would be:
 214
 215 ```rust
 216 # fn main() {}
 217 #[test]
 218 #[should_panic(expected = "assertion failed")]
 219 fn it_works() {
 220     assert_eq!("Hello", "world");
 221 }
 222 ```
 223
 224 That's all there is to the basics! Let's write one 'real' test:
 225
 226 ```rust,ignore
 227 # fn main() {}
 228 pub fn add_two(a: i32) -> i32 {
 229     a + 2
 230 }
 231
 232 #[test]
 233 fn it_works() {
 234     assert_eq!(4, add_two(2));
 235 }
 236 ```
 237
 238 This is a very common use of `assert_eq!`: call some function with
 239 some known arguments and compare it to the expected output.
 240
 241 # The `ignore` attribute
 242
 243 Sometimes a few specific tests can be very time-consuming to execute. These
 244 can be disabled by default by using the `ignore` attribute:
 245
 246 ```rust
 247 # fn main() {}
 248 #[test]
 249 fn it_works() {
 250     assert_eq!(4, add_two(2));
 251 }
 252
 253 #[test]
 254 #[ignore]
 255 fn expensive_test() {
 256     // code that takes an hour to run
 257 }
 258 ```
 259
 260 Now we run our tests and see that `it_works` is run, but `expensive_test` is
 261 not:
 262
 263 ```bash
 264 $ cargo test
 265    Compiling adder v0.0.1 (file:///home/you/projects/adder)
 266      Running target/adder-91b3e234d4ed382a
 267
 268 running 2 tests
 269 test expensive_test ... ignored
 270 test it_works ... ok
 271
 272 test result: ok. 1 passed; 0 failed; 1 ignored; 0 measured
 273
 274    Doc-tests adder
 275
 276 running 0 tests
 277
 278 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
 279 ```
 280
 281 The expensive tests can be run explicitly using `cargo test -- --ignored`:
 282
 283 ```bash
 284 $ cargo test -- --ignored
 285      Running target/adder-91b3e234d4ed382a
 286
 287 running 1 test
 288 test expensive_test ... ok
 289
 290 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
 291
 292    Doc-tests adder
 293
 294 running 0 tests
 295
 296 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
 297 ```
 298
 299 The `--ignored` argument is an argument to the test binary, and not to Cargo,
 300 which is why the command is `cargo test -- --ignored`.
 301
 302 # The `tests` module
 303
 304 There is one way in which our existing example is not idiomatic: it's
 305 missing the `tests` module. The idiomatic way of writing our example
 306 looks like this:
 307
 308 ```rust,ignore
 309 # fn main() {}
 310 pub fn add_two(a: i32) -> i32 {
 311     a + 2
 312 }
 313
 314 #[cfg(test)]
 315 mod tests {
 316     use super::add_two;
 317
 318     #[test]
 319     fn it_works() {
 320         assert_eq!(4, add_two(2));
 321     }
 322 }
 323 ```
 324
 325 There's a few changes here. The first is the introduction of a `mod tests` with
 326 a `cfg` attribute. The module allows us to group all of our tests together, and
 327 to also define helper functions if needed, that don't become a part of the rest
 328 of our crate. The `cfg` attribute only compiles our test code if we're
 329 currently trying to run the tests. This can save compile time, and also ensures
 330 that our tests are entirely left out of a normal build.
 331
 332 The second change is the `use` declaration. Because we're in an inner module,
 333 we need to bring our test function into scope. This can be annoying if you have
 334 a large module, and so this is a common use of globs. Let's change our
 335 `src/lib.rs` to make use of it:
 336
 337 ```rust,ignore
 338 # fn main() {}
 339 pub fn add_two(a: i32) -> i32 {
 340     a + 2
 341 }
 342
 343 #[cfg(test)]
 344 mod tests {
 345     use super::*;
 346
 347     #[test]
 348     fn it_works() {
 349         assert_eq!(4, add_two(2));
 350     }
 351 }
 352 ```
 353
 354 Note the different `use` line. Now we run our tests:
 355
 356 ```bash
 357 $ cargo test
 358     Updating registry `https://github.com/rust-lang/crates.io-index`
 359    Compiling adder v0.0.1 (file:///home/you/projects/adder)
 360      Running target/adder-91b3e234d4ed382a
 361
 362 running 1 test
 363 test tests::it_works ... ok
 364
 365 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
 366
 367    Doc-tests adder
 368
 369 running 0 tests
 370
 371 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
 372 ```
 373
 374 It works!
 375
 376 The current convention is to use the `tests` module to hold your "unit-style"
 377 tests. Anything that tests one small bit of functionality makes sense to
 378 go here. But what about "integration-style" tests instead? For that, we have
 379 the `tests` directory.
 380
 381 # The `tests` directory
 382
 383 To write an integration test, let's make a `tests` directory, and
 384 put a `tests/lib.rs` file inside, with this as its contents:
 385
 386 ```rust,ignore
 387 extern crate adder;
 388
 389 # fn main() {}
 390 #[test]
 391 fn it_works() {
 392     assert_eq!(4, adder::add_two(2));
 393 }
 394 ```
 395
 396 This looks similar to our previous tests, but slightly different. We now have
 397 an `extern crate adder` at the top. This is because the tests in the `tests`
 398 directory are an entirely separate crate, and so we need to import our library.
 399 This is also why `tests` is a suitable place to write integration-style tests:
 400 they use the library like any other consumer of it would.
 401
 402 Let's run them:
 403
 404 ```bash
 405 $ cargo test
 406    Compiling adder v0.0.1 (file:///home/you/projects/adder)
 407      Running target/adder-91b3e234d4ed382a
 408
 409 running 1 test
 410 test tests::it_works ... ok
 411
 412 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
 413
 414      Running target/lib-c18e7d3494509e74
 415
 416 running 1 test
 417 test it_works ... ok
 418
 419 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
 420
 421    Doc-tests adder
 422
 423 running 0 tests
 424
 425 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
 426 ```
 427
 428 Now we have three sections: our previous test is also run, as well as our new
 429 one.
 430
 431 That's all there is to the `tests` directory. The `tests` module isn't needed
 432 here, since the whole thing is focused on tests.
 433
 434 Let's finally check out that third section: documentation tests.
 435
 436 # Documentation tests
 437
 438 Nothing is better than documentation with examples. Nothing is worse than
 439 examples that don't actually work, because the code has changed since the
 440 documentation has been written. To this end, Rust supports automatically
 441 running examples in your documentation (**note:** this only works in library
 442 crates, not binary crates). Here's a fleshed-out `src/lib.rs` with examples:
 443
 444 ```rust,ignore
 445 # fn main() {}
 446 //! The `adder` crate provides functions that add numbers to other numbers.
 447 //!
 448 //! # Examples
 449 //!
 450 //! ```
 451 //! assert_eq!(4, adder::add_two(2));
 452 //! ```
 453
 454 /// This function adds two to its argument.
 455 ///
 456 /// # Examples
 457 ///
 458 /// ```
 459 /// use adder::add_two;
 460 ///
 461 /// assert_eq!(4, add_two(2));
 462 /// ```
 463 pub fn add_two(a: i32) -> i32 {
 464     a + 2
 465 }
 466
 467 #[cfg(test)]
 468 mod tests {
 469     use super::*;
 470
 471     #[test]
 472     fn it_works() {
 473         assert_eq!(4, add_two(2));
 474     }
 475 }
 476 ```
 477
 478 Note the module-level documentation with `//!` and the function-level
 479 documentation with `///`. Rust's documentation supports Markdown in comments,
 480 and so triple graves mark code blocks. It is conventional to include the
 481 `# Examples` section, exactly like that, with examples following.
 482
 483 Let's run the tests again:
 484
 485 ```bash
 486 $ cargo test
 487    Compiling adder v0.0.1 (file:///home/steve/tmp/adder)
 488      Running target/adder-91b3e234d4ed382a
 489
 490 running 1 test
 491 test tests::it_works ... ok
 492
 493 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
 494
 495      Running target/lib-c18e7d3494509e74
 496
 497 running 1 test
 498 test it_works ... ok
 499
 500 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
 501
 502    Doc-tests adder
 503
 504 running 2 tests
 505 test add_two_0 ... ok
 506 test _0 ... ok
 507
 508 test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured
 509 ```
 510
 511 Now we have all three kinds of tests running! Note the names of the
 512 documentation tests: the `_0` is generated for the module test, and `add_two_0`
 513 for the function test. These will auto increment with names like `add_two_1` as
 514 you add more examples.
 515
 516 We haven’t covered all of the details with writing documentation tests. For more,
 517 please see the [Documentation chapter](documentation.html).