src/doc/trpl/testing.md

   1 % Testing
   2
   3 > Program testing can be a very effective way to show the presence of bugs, but
   4 > it is hopelessly inadequate for showing their absence.
   5 >
   6 > Edsger W. Dijkstra, "The Humble Programmer" (1972)
   7
   8 Let's talk about how to test Rust code. What we will not be talking about is
   9 the right way to test Rust code. There are many schools of thought regarding
  10 the right and wrong way to write tests. All of these approaches use the same
  11 basic tools, and so we'll show you the syntax for using them.
  12
  13 # The `test` attribute
  14
  15 At its simplest, a test in Rust is a function that's annotated with the `test`
  16 attribute. Let's make a new project with Cargo called `adder`:
  17
  18 ```bash
  19 $ cargo new adder
  20 $ cd adder
  21 ```
  22
  23 Cargo will automatically generate a simple test when you make a new project.
  24 Here's the contents of `src/lib.rs`:
  25
  26 ```rust
  27 #[test]
  28 fn it_works() {
  29 }
  30 ```
  31
  32 Note the `#[test]`. This attribute indicates that this is a test function. It
  33 currently has no body. That's good enough to pass! We can run the tests with
  34 `cargo test`:
  35
  36 ```bash
  37 $ cargo test
  38    Compiling adder v0.0.1 (file:///home/you/projects/adder)
  39      Running target/adder-91b3e234d4ed382a
  40
  41 running 1 test
  42 test it_works ... ok
  43
  44 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
  45
  46    Doc-tests adder
  47
  48 running 0 tests
  49
  50 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
  51 ```
  52
  53 Cargo compiled and ran our tests. There are two sets of output here: one
  54 for the test we wrote, and another for documentation tests. We'll talk about
  55 those later. For now, see this line:
  56
  57 ```text
  58 test it_works ... ok
  59 ```
  60
  61 Note the `it_works`. This comes from the name of our function:
  62
  63 ```rust
  64 fn it_works() {
  65 # }
  66 ```
  67
  68 We also get a summary line:
  69
  70 ```text
  71 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
  72 ```
  73
  74 So why does our do-nothing test pass? Any test which doesn't `panic!` passes,
  75 and any test that does `panic!` fails. Let's make our test fail:
  76
  77 ```rust
  78 #[test]
  79 fn it_works() {
  80     assert!(false);
  81 }
  82 ```
  83
  84 `assert!` is a macro provided by Rust which takes one argument: if the argument
  85 is `true`, nothing happens. If the argument is false, it `panic!`s. Let's run
  86 our tests again:
  87
  88 ```bash
  89 $ cargo test
  90    Compiling adder v0.0.1 (file:///home/you/projects/adder)
  91      Running target/adder-91b3e234d4ed382a
  92
  93 running 1 test
  94 test it_works ... FAILED
  95
  96 failures:
  97
  98 ---- it_works stdout ----
  99         thread 'it_works' panicked at 'assertion failed: false', /home/steve/tmp/adder/src/lib.rs:3
 100
 101
 102
 103 failures:
 104     it_works
 105
 106 test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured
 107
 108 thread '<main>' panicked at 'Some tests failed', /home/steve/src/rust/src/libtest/lib.rs:247
 109 ```
 110
 111 Rust indicates that our test failed:
 112
 113 ```text
 114 test it_works ... FAILED
 115 ```
 116
 117 And that's reflected in the summary line:
 118
 119 ```text
 120 test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured
 121 ```
 122
 123 We also get a non-zero status code:
 124
 125 ```bash
 126 $ echo $?
 127 101
 128 ```
 129
 130 This is useful if you want to integrate `cargo test` into other tooling.
 131
 132 We can invert our test's failure with another attribute: `should_panic`:
 133
 134 ```rust
 135 #[test]
 136 #[should_panic]
 137 fn it_works() {
 138     assert!(false);
 139 }
 140 ```
 141
 142 This test will now succeed if we `panic!` and fail if we complete. Let's try it:
 143
 144 ```bash
 145 $ cargo test
 146    Compiling adder v0.0.1 (file:///home/you/projects/adder)
 147      Running target/adder-91b3e234d4ed382a
 148
 149 running 1 test
 150 test it_works ... ok
 151
 152 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
 153
 154    Doc-tests adder
 155
 156 running 0 tests
 157
 158 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
 159 ```
 160
 161 Rust provides another macro, `assert_eq!`, that compares two arguments for
 162 equality:
 163
 164 ```rust
 165 #[test]
 166 #[should_panic]
 167 fn it_works() {
 168     assert_eq!("Hello", "world");
 169 }
 170 ```
 171
 172 Does this test pass or fail? Because of the `should_panic` attribute, it
 173 passes:
 174
 175 ```bash
 176 $ cargo test
 177    Compiling adder v0.0.1 (file:///home/you/projects/adder)
 178      Running target/adder-91b3e234d4ed382a
 179
 180 running 1 test
 181 test it_works ... ok
 182
 183 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
 184
 185    Doc-tests adder
 186
 187 running 0 tests
 188
 189 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
 190 ```
 191
 192 `should_panic` tests can be fragile, as it's hard to guarantee that the test
 193 didn't fail for an unexpected reason. To help with this, an optional `expected`
 194 parameter can be added to the `should_panic` attribute. The test harness will
 195 make sure that the failure message contains the provided text. A safer version
 196 of the example above would be:
 197
 198 ```rust
 199 #[test]
 200 #[should_panic(expected = "assertion failed")]
 201 fn it_works() {
 202     assert_eq!("Hello", "world");
 203 }
 204 ```
 205
 206 That's all there is to the basics! Let's write one 'real' test:
 207
 208 ```rust,ignore
 209 pub fn add_two(a: i32) -> i32 {
 210     a + 2
 211 }
 212
 213 #[test]
 214 fn it_works() {
 215     assert_eq!(4, add_two(2));
 216 }
 217 ```
 218
 219 This is a very common use of `assert_eq!`: call some function with
 220 some known arguments and compare it to the expected output.
 221
 222 # The `tests` module
 223
 224 There is one way in which our existing example is not idiomatic: it's
 225 missing the `tests` module. The idiomatic way of writing our example
 226 looks like this:
 227
 228 ```rust,ignore
 229 pub fn add_two(a: i32) -> i32 {
 230     a + 2
 231 }
 232
 233 #[cfg(test)]
 234 mod tests {
 235     use super::add_two;
 236
 237     #[test]
 238     fn it_works() {
 239         assert_eq!(4, add_two(2));
 240     }
 241 }
 242 ```
 243
 244 There's a few changes here. The first is the introduction of a `mod tests` with
 245 a `cfg` attribute. The module allows us to group all of our tests together, and
 246 to also define helper functions if needed, that don't become a part of the rest
 247 of our crate. The `cfg` attribute only compiles our test code if we're
 248 currently trying to run the tests. This can save compile time, and also ensures
 249 that our tests are entirely left out of a normal build.
 250
 251 The second change is the `use` declaration. Because we're in an inner module,
 252 we need to bring our test function into scope. This can be annoying if you have
 253 a large module, and so this is a common use of the `glob` feature. Let's change
 254 our `src/lib.rs` to make use of it:
 255
 256 ```rust,ignore
 257
 258 pub fn add_two(a: i32) -> i32 {
 259     a + 2
 260 }
 261
 262 #[cfg(test)]
 263 mod tests {
 264     use super::*;
 265
 266     #[test]
 267     fn it_works() {
 268         assert_eq!(4, add_two(2));
 269     }
 270 }
 271 ```
 272
 273 Note the different `use` line. Now we run our tests:
 274
 275 ```bash
 276 $ cargo test
 277     Updating registry `https://github.com/rust-lang/crates.io-index`
 278    Compiling adder v0.0.1 (file:///home/you/projects/adder)
 279      Running target/adder-91b3e234d4ed382a
 280
 281 running 1 test
 282 test tests::it_works ... ok
 283
 284 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
 285
 286    Doc-tests adder
 287
 288 running 0 tests
 289
 290 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
 291 ```
 292
 293 It works!
 294
 295 The current convention is to use the `tests` module to hold your "unit-style"
 296 tests. Anything that just tests one small bit of functionality makes sense to
 297 go here. But what about "integration-style" tests instead? For that, we have
 298 the `tests` directory
 299
 300 # The `tests` directory
 301
 302 To write an integration test, let's make a `tests` directory, and
 303 put a `tests/lib.rs` file inside, with this as its contents:
 304
 305 ```rust,ignore
 306 extern crate adder;
 307
 308 #[test]
 309 fn it_works() {
 310     assert_eq!(4, adder::add_two(2));
 311 }
 312 ```
 313
 314 This looks similar to our previous tests, but slightly different. We now have
 315 an `extern crate adder` at the top. This is because the tests in the `tests`
 316 directory are an entirely separate crate, and so we need to import our library.
 317 This is also why `tests` is a suitable place to write integration-style tests:
 318 they use the library like any other consumer of it would.
 319
 320 Let's run them:
 321
 322 ```bash
 323 $ cargo test
 324    Compiling adder v0.0.1 (file:///home/you/projects/adder)
 325      Running target/adder-91b3e234d4ed382a
 326
 327 running 1 test
 328 test tests::it_works ... ok
 329
 330 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
 331
 332      Running target/lib-c18e7d3494509e74
 333
 334 running 1 test
 335 test it_works ... ok
 336
 337 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
 338
 339    Doc-tests adder
 340
 341 running 0 tests
 342
 343 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
 344 ```
 345
 346 Now we have three sections: our previous test is also run, as well as our new
 347 one.
 348
 349 That's all there is to the `tests` directory. The `tests` module isn't needed
 350 here, since the whole thing is focused on tests.
 351
 352 Let's finally check out that third section: documentation tests.
 353
 354 # Documentation tests
 355
 356 Nothing is better than documentation with examples. Nothing is worse than
 357 examples that don't actually work, because the code has changed since the
 358 documentation has been written. To this end, Rust supports automatically
 359 running examples in your documentation. Here's a fleshed-out `src/lib.rs`
 360 with examples:
 361
 362 ```rust,ignore
 363 //! The `adder` crate provides functions that add numbers to other numbers.
 364 //!
 365 //! # Examples
 366 //!
 367 //! ```
 368 //! assert_eq!(4, adder::add_two(2));
 369 //! ```
 370
 371 /// This function adds two to its argument.
 372 ///
 373 /// # Examples
 374 ///
 375 /// ```
 376 /// use adder::add_two;
 377 ///
 378 /// assert_eq!(4, add_two(2));
 379 /// ```
 380 pub fn add_two(a: i32) -> i32 {
 381     a + 2
 382 }
 383
 384 #[cfg(test)]
 385 mod tests {
 386     use super::*;
 387
 388     #[test]
 389     fn it_works() {
 390         assert_eq!(4, add_two(2));
 391     }
 392 }
 393 ```
 394
 395 Note the module-level documentation with `//!` and the function-level
 396 documentation with `///`. Rust's documentation supports Markdown in comments,
 397 and so triple graves mark code blocks. It is conventional to include the
 398 `# Examples` section, exactly like that, with examples following.
 399
 400 Let's run the tests again:
 401
 402 ```bash
 403 $ cargo test
 404    Compiling adder v0.0.1 (file:///home/steve/tmp/adder)
 405      Running target/adder-91b3e234d4ed382a
 406
 407 running 1 test
 408 test tests::it_works ... ok
 409
 410 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
 411
 412      Running target/lib-c18e7d3494509e74
 413
 414 running 1 test
 415 test it_works ... ok
 416
 417 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
 418
 419    Doc-tests adder
 420
 421 running 2 tests
 422 test add_two_0 ... ok
 423 test _0 ... ok
 424
 425 test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured
 426 ```
 427
 428 Now we have all three kinds of tests running! Note the names of the
 429 documentation tests: the `_0` is generated for the module test, and `add_two_0`
 430 for the function test. These will auto increment with names like `add_two_1` as
 431 you add more examples.
 432