]> git.proxmox.com Git - rustc.git/blob - src/doc/book/testing.md
New upstream version 1.12.0+dfsg1
[rustc.git] / src / doc / book / testing.md
1 % Testing
2
3 > Program testing can be a very effective way to show the presence of bugs, but
4 > it is hopelessly inadequate for showing their absence.
5 >
6 > Edsger W. Dijkstra, "The Humble Programmer" (1972)
7
8 Let's talk about how to test Rust code. What we will not be talking about is
9 the right way to test Rust code. There are many schools of thought regarding
10 the right and wrong way to write tests. All of these approaches use the same
11 basic tools, and so we'll show you the syntax for using them.
12
13 # The `test` attribute
14
15 At its simplest, a test in Rust is a function that's annotated with the `test`
16 attribute. Let's make a new project with Cargo called `adder`:
17
18 ```bash
19 $ cargo new adder
20 $ cd adder
21 ```
22
23 Cargo will automatically generate a simple test when you make a new project.
24 Here's the contents of `src/lib.rs`:
25
26 ```rust
27 # fn main() {}
28 #[test]
29 fn it_works() {
30 }
31 ```
32
33 Note the `#[test]`. This attribute indicates that this is a test function. It
34 currently has no body. That's good enough to pass! We can run the tests with
35 `cargo test`:
36
37 ```bash
38 $ cargo test
39 Compiling adder v0.0.1 (file:///home/you/projects/adder)
40 Running target/adder-91b3e234d4ed382a
41
42 running 1 test
43 test it_works ... ok
44
45 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
46
47 Doc-tests adder
48
49 running 0 tests
50
51 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
52 ```
53
54 Cargo compiled and ran our tests. There are two sets of output here: one
55 for the test we wrote, and another for documentation tests. We'll talk about
56 those later. For now, see this line:
57
58 ```text
59 test it_works ... ok
60 ```
61
62 Note the `it_works`. This comes from the name of our function:
63
64 ```rust
65 fn it_works() {
66 # }
67 ```
68
69 We also get a summary line:
70
71 ```text
72 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
73 ```
74
75 So why does our do-nothing test pass? Any test which doesn't `panic!` passes,
76 and any test that does `panic!` fails. Let's make our test fail:
77
78 ```rust
79 # fn main() {}
80 #[test]
81 fn it_works() {
82 assert!(false);
83 }
84 ```
85
86 `assert!` is a macro provided by Rust which takes one argument: if the argument
87 is `true`, nothing happens. If the argument is `false`, it will `panic!`. Let's
88 run our tests again:
89
90 ```bash
91 $ cargo test
92 Compiling adder v0.0.1 (file:///home/you/projects/adder)
93 Running target/adder-91b3e234d4ed382a
94
95 running 1 test
96 test it_works ... FAILED
97
98 failures:
99
100 ---- it_works stdout ----
101 thread 'it_works' panicked at 'assertion failed: false', /home/steve/tmp/adder/src/lib.rs:3
102
103
104
105 failures:
106 it_works
107
108 test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured
109
110 thread 'main' panicked at 'Some tests failed', /home/steve/src/rust/src/libtest/lib.rs:247
111 ```
112
113 Rust indicates that our test failed:
114
115 ```text
116 test it_works ... FAILED
117 ```
118
119 And that's reflected in the summary line:
120
121 ```text
122 test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured
123 ```
124
125 We also get a non-zero status code. We can use `$?` on OS X and Linux:
126
127 ```bash
128 $ echo $?
129 101
130 ```
131
132 On Windows, if you’re using `cmd`:
133
134 ```bash
135 > echo %ERRORLEVEL%
136 ```
137
138 And if you’re using PowerShell:
139
140 ```bash
141 > echo $LASTEXITCODE # the code itself
142 > echo $? # a boolean, fail or succeed
143 ```
144
145 This is useful if you want to integrate `cargo test` into other tooling.
146
147 We can invert our test's failure with another attribute: `should_panic`:
148
149 ```rust
150 # fn main() {}
151 #[test]
152 #[should_panic]
153 fn it_works() {
154 assert!(false);
155 }
156 ```
157
158 This test will now succeed if we `panic!` and fail if we complete. Let's try it:
159
160 ```bash
161 $ cargo test
162 Compiling adder v0.0.1 (file:///home/you/projects/adder)
163 Running target/adder-91b3e234d4ed382a
164
165 running 1 test
166 test it_works ... ok
167
168 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
169
170 Doc-tests adder
171
172 running 0 tests
173
174 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
175 ```
176
177 Rust provides another macro, `assert_eq!`, that compares two arguments for
178 equality:
179
180 ```rust
181 # fn main() {}
182 #[test]
183 #[should_panic]
184 fn it_works() {
185 assert_eq!("Hello", "world");
186 }
187 ```
188
189 Does this test pass or fail? Because of the `should_panic` attribute, it
190 passes:
191
192 ```bash
193 $ cargo test
194 Compiling adder v0.0.1 (file:///home/you/projects/adder)
195 Running target/adder-91b3e234d4ed382a
196
197 running 1 test
198 test it_works ... ok
199
200 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
201
202 Doc-tests adder
203
204 running 0 tests
205
206 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
207 ```
208
209 `should_panic` tests can be fragile, as it's hard to guarantee that the test
210 didn't fail for an unexpected reason. To help with this, an optional `expected`
211 parameter can be added to the `should_panic` attribute. The test harness will
212 make sure that the failure message contains the provided text. A safer version
213 of the example above would be:
214
215 ```rust
216 # fn main() {}
217 #[test]
218 #[should_panic(expected = "assertion failed")]
219 fn it_works() {
220 assert_eq!("Hello", "world");
221 }
222 ```
223
224 That's all there is to the basics! Let's write one 'real' test:
225
226 ```rust,ignore
227 # fn main() {}
228 pub fn add_two(a: i32) -> i32 {
229 a + 2
230 }
231
232 #[test]
233 fn it_works() {
234 assert_eq!(4, add_two(2));
235 }
236 ```
237
238 This is a very common use of `assert_eq!`: call some function with
239 some known arguments and compare it to the expected output.
240
241 # The `ignore` attribute
242
243 Sometimes a few specific tests can be very time-consuming to execute. These
244 can be disabled by default by using the `ignore` attribute:
245
246 ```rust
247 # fn main() {}
248 #[test]
249 fn it_works() {
250 assert_eq!(4, add_two(2));
251 }
252
253 #[test]
254 #[ignore]
255 fn expensive_test() {
256 // code that takes an hour to run
257 }
258 ```
259
260 Now we run our tests and see that `it_works` is run, but `expensive_test` is
261 not:
262
263 ```bash
264 $ cargo test
265 Compiling adder v0.0.1 (file:///home/you/projects/adder)
266 Running target/adder-91b3e234d4ed382a
267
268 running 2 tests
269 test expensive_test ... ignored
270 test it_works ... ok
271
272 test result: ok. 1 passed; 0 failed; 1 ignored; 0 measured
273
274 Doc-tests adder
275
276 running 0 tests
277
278 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
279 ```
280
281 The expensive tests can be run explicitly using `cargo test -- --ignored`:
282
283 ```bash
284 $ cargo test -- --ignored
285 Running target/adder-91b3e234d4ed382a
286
287 running 1 test
288 test expensive_test ... ok
289
290 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
291
292 Doc-tests adder
293
294 running 0 tests
295
296 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
297 ```
298
299 The `--ignored` argument is an argument to the test binary, and not to Cargo,
300 which is why the command is `cargo test -- --ignored`.
301
302 # The `tests` module
303
304 There is one way in which our existing example is not idiomatic: it's
305 missing the `tests` module. The idiomatic way of writing our example
306 looks like this:
307
308 ```rust,ignore
309 # fn main() {}
310 pub fn add_two(a: i32) -> i32 {
311 a + 2
312 }
313
314 #[cfg(test)]
315 mod tests {
316 use super::add_two;
317
318 #[test]
319 fn it_works() {
320 assert_eq!(4, add_two(2));
321 }
322 }
323 ```
324
325 There's a few changes here. The first is the introduction of a `mod tests` with
326 a `cfg` attribute. The module allows us to group all of our tests together, and
327 to also define helper functions if needed, that don't become a part of the rest
328 of our crate. The `cfg` attribute only compiles our test code if we're
329 currently trying to run the tests. This can save compile time, and also ensures
330 that our tests are entirely left out of a normal build.
331
332 The second change is the `use` declaration. Because we're in an inner module,
333 we need to bring our test function into scope. This can be annoying if you have
334 a large module, and so this is a common use of globs. Let's change our
335 `src/lib.rs` to make use of it:
336
337 ```rust,ignore
338 # fn main() {}
339 pub fn add_two(a: i32) -> i32 {
340 a + 2
341 }
342
343 #[cfg(test)]
344 mod tests {
345 use super::*;
346
347 #[test]
348 fn it_works() {
349 assert_eq!(4, add_two(2));
350 }
351 }
352 ```
353
354 Note the different `use` line. Now we run our tests:
355
356 ```bash
357 $ cargo test
358 Updating registry `https://github.com/rust-lang/crates.io-index`
359 Compiling adder v0.0.1 (file:///home/you/projects/adder)
360 Running target/adder-91b3e234d4ed382a
361
362 running 1 test
363 test tests::it_works ... ok
364
365 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
366
367 Doc-tests adder
368
369 running 0 tests
370
371 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
372 ```
373
374 It works!
375
376 The current convention is to use the `tests` module to hold your "unit-style"
377 tests. Anything that tests one small bit of functionality makes sense to
378 go here. But what about "integration-style" tests instead? For that, we have
379 the `tests` directory.
380
381 # The `tests` directory
382
383 Each file in `tests/*.rs` directory is treated as individual crate.
384 So, to write an integration test, let's make a `tests` directory, and
385 put a `tests/integration_test.rs` file inside, with this as its contents:
386
387 ```rust,ignore
388 extern crate adder;
389
390 # fn main() {}
391 #[test]
392 fn it_works() {
393 assert_eq!(4, adder::add_two(2));
394 }
395 ```
396
397 This looks similar to our previous tests, but slightly different. We now have
398 an `extern crate adder` at the top. This is because each test in the `tests`
399 directory is an entirely separate crate, and so we need to import our library.
400 This is also why `tests` is a suitable place to write integration-style tests:
401 they use the library like any other consumer of it would.
402
403 Let's run them:
404
405 ```bash
406 $ cargo test
407 Compiling adder v0.0.1 (file:///home/you/projects/adder)
408 Running target/adder-91b3e234d4ed382a
409
410 running 1 test
411 test tests::it_works ... ok
412
413 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
414
415 Running target/lib-c18e7d3494509e74
416
417 running 1 test
418 test it_works ... ok
419
420 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
421
422 Doc-tests adder
423
424 running 0 tests
425
426 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
427 ```
428
429 Now we have three sections: our previous test is also run, as well as our new
430 one.
431
432 Cargo will ignore files in subdirectories of the `tests/` directory.
433 Therefore shared modules in integrations tests are possible.
434 For example `tests/common/mod.rs` is not separately compiled by cargo but can
435 be imported in every test with `mod common;`
436
437 That's all there is to the `tests` directory. The `tests` module isn't needed
438 here, since the whole thing is focused on tests.
439
440 Let's finally check out that third section: documentation tests.
441
442 # Documentation tests
443
444 Nothing is better than documentation with examples. Nothing is worse than
445 examples that don't actually work, because the code has changed since the
446 documentation has been written. To this end, Rust supports automatically
447 running examples in your documentation (**note:** this only works in library
448 crates, not binary crates). Here's a fleshed-out `src/lib.rs` with examples:
449
450 ```rust,ignore
451 # fn main() {}
452 //! The `adder` crate provides functions that add numbers to other numbers.
453 //!
454 //! # Examples
455 //!
456 //! ```
457 //! assert_eq!(4, adder::add_two(2));
458 //! ```
459
460 /// This function adds two to its argument.
461 ///
462 /// # Examples
463 ///
464 /// ```
465 /// use adder::add_two;
466 ///
467 /// assert_eq!(4, add_two(2));
468 /// ```
469 pub fn add_two(a: i32) -> i32 {
470 a + 2
471 }
472
473 #[cfg(test)]
474 mod tests {
475 use super::*;
476
477 #[test]
478 fn it_works() {
479 assert_eq!(4, add_two(2));
480 }
481 }
482 ```
483
484 Note the module-level documentation with `//!` and the function-level
485 documentation with `///`. Rust's documentation supports Markdown in comments,
486 and so triple graves mark code blocks. It is conventional to include the
487 `# Examples` section, exactly like that, with examples following.
488
489 Let's run the tests again:
490
491 ```bash
492 $ cargo test
493 Compiling adder v0.0.1 (file:///home/steve/tmp/adder)
494 Running target/adder-91b3e234d4ed382a
495
496 running 1 test
497 test tests::it_works ... ok
498
499 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
500
501 Running target/lib-c18e7d3494509e74
502
503 running 1 test
504 test it_works ... ok
505
506 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
507
508 Doc-tests adder
509
510 running 2 tests
511 test add_two_0 ... ok
512 test _0 ... ok
513
514 test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured
515 ```
516
517 Now we have all three kinds of tests running! Note the names of the
518 documentation tests: the `_0` is generated for the module test, and `add_two_0`
519 for the function test. These will auto increment with names like `add_two_1` as
520 you add more examples.
521
522 We haven’t covered all of the details with writing documentation tests. For more,
523 please see the [Documentation chapter](documentation.html).