7 `compiletest` is the main test harness of the Rust test suite.
8 It allows test authors to organize large numbers of tests
9 (the Rust compiler has many thousands),
10 efficient test execution (parallel execution is supported),
11 and allows the test author to configure behavior and expected results of both
12 individual and groups of tests.
14 `compiletest` may check test code for success, for runtime failure,
15 or for compile-time failure.
16 Tests are typically organized as a Rust source file with annotations in
17 comments before and/or within the test code.
18 These comments serve to direct `compiletest` on if or how to run the test,
19 what behavior to expect, and more.
20 See [header commands](headers.md) and the test suite documentation below
21 for more details on these annotations.
23 See the [Adding new tests](adding.md) chapter for a tutorial on creating a new
24 test, and the [Running tests](running.md) chapter on how to run the test
29 All of the tests are in the [`src/test`] directory.
30 The tests are organized into "suites", with each suite in a separate subdirectory.
31 Each test suite behaves a little differently, with different compiler behavior
32 and different checks for correctness.
33 For example, the [`src/test/incremental`] directory contains tests for
34 incremental compilation.
35 The various suites are defined in [`src/tools/compiletest/src/common.rs`] in
36 the `pub enum Mode` declaration.
38 The following test suites are available, with links for more information:
40 - [`ui`](ui.md) — tests that check the stdout/stderr from the compilation
41 and/or running the resulting executable
42 - `ui-fulldeps` — `ui` tests which require a linkable build of `rustc` (such
43 as using `extern crate rustc_span;` or used as a plugin)
44 - [`pretty`](#pretty-printer-tests) — tests for pretty printing
45 - [`incremental`](#incremental-tests) — tests incremental compilation behavior
46 - [`debuginfo`](#debuginfo-tests) — tests for debuginfo generation running debuggers
47 - [`codegen`](#codegen-tests) — tests for code generation
48 - [`codegen-units`](#codegen-units-tests) — tests for codegen unit partitioning
49 - [`assembly`](#assembly-tests) — verifies assembly output
50 - [`mir-opt`](#mir-opt-tests) — tests for MIR generation
51 - [`run-make`](#run-make-tests) — general purpose tests using a Makefile
52 - `run-make-fulldeps` — `run-make` tests which require a linkable build of `rustc`,
54 - [`run-pass-valgrind`](#valgrind-tests) — tests run with Valgrind
55 - [Rustdoc tests](../rustdoc.md#tests):
56 - `rustdoc` — tests for rustdoc, making sure that the generated files
57 contain the expected documentation.
58 - `rustdoc-gui` — tests for rustdoc's GUI using a web browser.
59 - `rustdoc-js` — tests to ensure the rustdoc search is working as expected.
60 - `rustdoc-js-std` — tests to ensure the rustdoc search is working as expected
61 (run specifically on the std docs).
62 - `rustdoc-json` — tests on the JSON output of rustdoc.
63 - `rustdoc-ui` — tests on the terminal output of rustdoc.
65 [`src/test`]: https://github.com/rust-lang/rust/blob/master/src/test
66 [`src/tools/compiletest/src/common.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs
68 ### Pretty-printer tests
70 The tests in [`src/test/pretty`] exercise the "pretty-printing" functionality of `rustc`.
71 The `-Z unpretty` CLI option for `rustc` causes it to translate the input source
72 into various different formats, such as the Rust source after macro expansion.
74 The pretty-printer tests have several [header commands](headers.md) described below.
75 These commands can significantly change the behavior of the test, but the
76 default behavior without any commands is to:
78 1. Run `rustc -Zunpretty=normal` on the source file
79 2. Run `rustc -Zunpretty=normal` on the output of the previous step
80 3. The output of the previous two steps should be the same.
81 4. Run `rustc -Zno-codegen` on the output to make sure that it can type check
82 (this is similar to running `cargo check`)
84 If any of the commands above fail, then the test fails.
86 The header commands for pretty-printing tests are:
88 * `pretty-mode` specifies the mode pretty-print tests should run in
89 (that is, the argument to `-Zunpretty`).
90 The default is `normal` if not specified.
91 * `pretty-compare-only` causes a pretty test to only compare the pretty-printed output
92 (stopping after step 3 from above).
93 It will not try to compile the expanded output to type check it.
94 This is needed for a pretty-mode that does not expand to valid
95 Rust, or for other situations where the expanded output cannot be compiled.
96 * `pretty-expanded` allows a pretty test to also check that the expanded
97 output can be type checked.
98 That is, after the steps above, it does two more steps:
100 > 5. Run `rustc -Zunpretty=expanded` on the original source
101 > 6. Run `rustc -Zno-codegen` on the expanded output to make sure that it can type check
103 This is needed because not all code can be compiled after being expanded.
104 Pretty tests should specify this if they can.
105 An example where this cannot be used is if the test includes `println!`.
106 That macro expands to reference private internal functions of the standard
107 library that cannot be called directly without the `fmt_internals` feature
110 More history about this may be found in
111 [#23616](https://github.com/rust-lang/rust/issues/23616#issuecomment-484999901).
112 * `pp-exact` is used to ensure a pretty-print test results in specific output.
113 If specified without a value, then it means the pretty-print output should
114 match the original source.
115 If specified with a value, as in `// pp-exact:foo.pp`,
116 it will ensure that the pretty-printed output matches the contents of the given file.
117 Otherwise, if `pp-exact` is not specified, then the pretty-printed output
118 will be pretty-printed one more time, and the output of the two
119 pretty-printing rounds will be compared to ensure that the pretty-printed
120 output converges to a steady state.
122 [`src/test/pretty`]: https://github.com/rust-lang/rust/tree/master/src/test/pretty
124 ### Incremental tests
126 The tests in [`src/test/incremental`] exercise incremental compilation.
127 They use [revision headers](#revisions) to tell compiletest to run the
128 compiler in a series of steps.
129 Compiletest starts with an empty directory with the `-C incremental` flag, and
130 then runs the compiler for each revision, reusing the incremental results from
132 The revisions should start with:
134 * `rpass` — the test should compile and run successfully
135 * `rfail` — the test should compile successfully, but the executable should fail to run
136 * `cfail` — the test should fail to compile
138 To make the revisions unique, you should add a suffix like `rpass1` and `rpass2`.
140 To simulate changing the source, compiletest also passes a `--cfg` flag with
141 the current revision name.
142 For example, this will run twice, simulating changing a function:
145 // revisions: rpass1 rpass2
160 `cfail` tests support the `forbid-output` header to specify that a certain
161 substring must not appear anywhere in the compiler output.
162 This can be useful to ensure certain errors do not appear, but this can be
163 fragile as error messages change over time, and a test may no longer be
164 checking the right thing but will still pass.
166 `cfail` tests support the `should-ice` header to specify that a test should
167 cause an Internal Compiler Error (ICE).
168 This is a highly specialized header to check that the incremental cache
169 continues to work after an ICE.
171 [`src/test/incremental`]: https://github.com/rust-lang/rust/tree/master/src/test/incremental
176 The tests in [`src/test/debuginfo`] test debuginfo generation.
177 They build a program, launch a debugger, and issue commands to the debugger.
178 A single test can work with cdb, gdb, and lldb.
180 Most tests should have the `// compile-flags: -g` header or something similar
181 to generate the appropriate debuginfo.
183 To set a breakpoint on a line, add a `// #break` comment on the line.
185 The debuginfo tests consist of a series of debugger commands along with
186 "check" lines which specify output that is expected from the debugger.
188 The commands are comments of the form `// $DEBUGGER-command:$COMMAND` where
189 `$DEBUGGER` is the debugger being used and `$COMMAND` is the debugger command
191 The debugger values can be:
195 * `gdbg` — GDB without Rust support (versions older than 7.11)
196 * `gdbr` — GDB with Rust support
198 * `lldbg` — LLDB without Rust support
199 * `lldbr` — LLDB with Rust support (this no longer exists)
201 The command to check the output are of the form `// $DEBUGGER-check:$OUTPUT`
202 where `$OUTPUT` is the output to expect.
204 For example, the following will build the test, start the debugger, set a
205 breakpoint, launch the program, inspect a value, and check what the debugger
212 // lldb-command: print foo
213 // lldb-check: $0 = 123
223 The following [header commands](headers.md) are available to disable a
224 test based on the debugger currently being used:
226 * `min-cdb-version: 10.0.18317.1001` — ignores the test if the version of cdb
227 is below the given version
228 * `min-gdb-version: 8.2` — ignores the test if the version of gdb is below the
230 * `ignore-gdb-version: 9.2` — ignores the test if the version of gdb is equal
232 * `ignore-gdb-version: 7.11.90 - 8.0.9` — ignores the test if the version of
233 gdb is in a range (inclusive)
234 * `min-lldb-version: 310` — ignores the test if the version of lldb is below
236 * `rust-lldb` — ignores the test if lldb is not contain the Rust plugin.
237 NOTE: The "Rust" version of LLDB doesn't exist anymore, so this will always be ignored.
238 This should probably be removed.
240 [`src/test/debuginfo`]: https://github.com/rust-lang/rust/tree/master/src/test/debuginfo
245 The tests in [`src/test/codegen`] test LLVM code generation.
246 They compile the test with the `--emit=llvm-ir` flag to emit LLVM IR.
247 They then run the LLVM [FileCheck] tool.
248 The test is annotated with various `// CHECK` comments to check the generated code.
249 See the FileCheck documentation for a tutorial and more information.
251 See also the [assembly tests](#assembly-tests) for a similar set of tests.
253 [`src/test/codegen`]: https://github.com/rust-lang/rust/tree/master/src/test/codegen
254 [FileCheck]: https://llvm.org/docs/CommandGuide/FileCheck.html
259 The tests in [`src/test/assembly`] test LLVM assembly output.
260 They compile the test with the `--emit=asm` flag to emit a `.s` file with the
262 They then run the LLVM [FileCheck] tool.
264 Each test should be annotated with the `// assembly-output:` header
265 with a value of either `emit-asm` or `ptx-linker` to indicate
266 the type of assembly output.
268 Then, they should be annotated with various `// CHECK` comments to check the
270 See the FileCheck documentation for a tutorial and more information.
272 See also the [codegen tests](#codegen-tests) for a similar set of tests.
274 [`src/test/assembly`]: https://github.com/rust-lang/rust/tree/master/src/test/assembly
277 ### Codegen-units tests
279 The tests in [`src/test/codegen-units`] test the
280 [monomorphization](../backend/monomorph.md) collector and CGU partitioning.
282 These tests work by running `rustc` with a flag to print the result of the
283 monomorphization collection pass, and then special annotations in the file are
284 used to compare against that.
286 Each test should be annotated with the `// compile-flags:-Zprint-mono-items=VAL`
287 header with the appropriate VAL to instruct `rustc` to print the
288 monomorphization information.
290 Then, the test should be annotated with comments of the form `//~ MONO_ITEM name`
291 where `name` is the monomorphized string printed by rustc like `fn <u32 as Trait>::foo`.
293 To check for CGU partitioning, a comment of the form `//~ MONO_ITEM name @@ cgu`
294 where `cgu` is a space separated list of the CGU names and the linkage
295 information in brackets.
296 For example: `//~ MONO_ITEM static function::FOO @@ statics[Internal]`
298 [`src/test/codegen-units`]: https://github.com/rust-lang/rust/tree/master/src/test/codegen-units
303 The tests in [`src/test/mir-opt`] check parts of the generated MIR to make
304 sure it is generated correctly and is doing the expected optimizations.
305 Check out the [MIR Optimizations](../mir/optimizations.md) chapter for more.
307 Compiletest will build the test with several flags to dump the MIR output and
308 set a baseline for optimizations:
312 * `-Zmir-opt-level=4`
314 * `-Zdump-mir-exclude-pass-number`
316 The test should be annotated with `// EMIT_MIR` comments that specify files that
317 will contain the expected MIR output.
318 You can use `x.py test --bless` to create the initial expected files.
320 There are several forms the `EMIT_MIR` comment can take:
322 * `// EMIT_MIR $MIR_PATH.mir` — This will check that the given filename
323 matches the exact output from the MIR dump.
324 For example, `my_test.main.SimplifyCfg-elaborate-drops.after.mir` will load
325 that file from the test directory, and compare it against the dump from
328 Checking the "after" file (which is after optimization) is useful if you are
329 interested in the final state after an optimization.
330 Some rare cases may want to use the "before" file for completeness.
332 * `// EMIT_MIR $MIR_PATH.diff` — where `$MIR_PATH` is the filename of the MIR
333 dump, such as `my_test_name.my_function.EarlyOtherwiseBranch`.
334 Compiletest will diff the `.before.mir` and `.after.mir` files, and compare
335 the diff output to the expected `.diff` file from the `EMIT_MIR` comment.
337 This is useful if you want to see how an optimization changes the MIR.
339 * `// EMIT_MIR $MIR_PATH.dot` or `$MIR_PATH.html` — These are special cases
340 for other MIR outputs (via `-Z dump-mir-graphviz` and `-Z dump-mir-spanview`)
341 that will check that the output matches the given file.
343 By default 32 bit and 64 bit targets use the same dump files, which can be
344 problematic in the presence of pointers in constants or other bit width
345 dependent things. In that case you can add `// EMIT_MIR_FOR_EACH_BIT_WIDTH` to
346 your test, causing separate files to be generated for 32bit and 64bit systems.
348 [`src/test/mir-opt`]: https://github.com/rust-lang/rust/tree/master/src/test/mir-opt
353 The tests in [`src/test/run-make`] are general-purpose tests using Makefiles
354 which provide the ultimate in flexibility.
355 These should be used as a last resort.
356 If possible, you should use one of the other test suites.
357 If there is some minor feature missing which you need for your test,
358 consider extending compiletest to add a header command for what you need.
359 However, sometimes just running a bunch of commands is really what you
360 need, `run-make` is here to the rescue!
362 Each test should be in a separate directory with a `Makefile` indicating the
364 There is a [`tools.mk`] Makefile which you can include which provides a bunch of
365 utilities to make it easier to run commands and compare outputs.
366 Take a look at some of the other tests for some examples on how to get started.
368 [`tools.mk`]: https://github.com/rust-lang/rust/blob/master/src/test/run-make-fulldeps/tools.mk
369 [`src/test/run-make`]: https://github.com/rust-lang/rust/tree/master/src/test/run-make
374 The tests in [`src/test/run-pass-valgrind`] are for use with [Valgrind].
375 These are currently vestigial, as Valgrind is no longer used in CI.
376 These may be removed in the future.
378 [Valgrind]: https://valgrind.org/
379 [`src/test/run-pass-valgrind`]: https://github.com/rust-lang/rust/tree/master/src/test/run-pass-valgrind
382 ## Building auxiliary crates
384 It is common that some tests require additional auxiliary crates to be compiled.
385 There are two [headers](headers.md) to assist with that:
390 `aux-build` will build a separate crate from the named source file.
391 The source file should be in a directory called `auxiliary` beside the test file.
394 // aux-build: my-helper.rs
396 extern crate my_helper;
397 // ... You can use my_helper.
400 The aux crate will be built as a dylib if possible (unless on a platform that
401 does not support them, or the `no-prefer-dynamic` header is specified in the
403 The `-L` flag is used to find the extern crates.
405 `aux-crate` is very similar to `aux-build`; however, it uses the `--extern`
406 flag to link to the extern crate.
407 That allows you to specify the additional syntax of the `--extern` flag, such
408 as renaming a dependency.
409 For example, `// aux-crate:foo=bar.rs` will compile `auxiliary/bar.rs` and
410 make it available under then name `foo` within the test.
411 This is similar to how Cargo does dependency renaming.
413 ### Auxiliary proc-macro
415 If you want a proc-macro dependency, then there currently is some ceremony
417 Place the proc-macro itself in a file like `auxiliary/my-proc-macro.rs`
418 with the following structure:
424 #![crate_type = "proc-macro"]
426 extern crate proc_macro;
427 use proc_macro::TokenStream;
430 pub fn foo(input: TokenStream) -> TokenStream {
435 The `force-host` is needed because proc-macros are loaded in the host
436 compiler, and `no-prefer-dynamic` is needed to tell compiletest to not use
437 `prefer-dynamic` which is not compatible with proc-macros.
438 The `#![crate_type]` attribute is needed to specify the correct crate-type.
440 Then in your test, you can build with with `aux-build`:
443 // aux-build: my-proc-macro.rs
445 extern crate my_proc_macro;
448 my_proc_macro::foo!();
455 Certain classes of tests support "revisions" (as of <!-- date: 2022-07 --> July 2022,
456 this includes UI, assembly, codegen, debuginfo, incremental, and rustdoc UI tests,
457 though incremental tests are somewhat different).
458 Revisions allow a single test file to be used for multiple tests.
459 This is done by adding a special header at the top of the file:
462 // revisions: foo bar baz
465 This will result in the test being compiled (and tested) three times,
466 once with `--cfg foo`, once with `--cfg bar`, and once with `--cfg
468 You can therefore use `#[cfg(foo)]` etc within the test to tweak
469 each of these results.
471 You can also customize headers and expected error messages to a particular
472 revision. To do this, add `[foo]` (or `bar`, `baz`, etc) after the `//`
476 // A flag to pass in only for cfg `foo`:
477 //[foo]compile-flags: -Z verbose
481 let x: usize = 32_u32; //[foo]~ ERROR mismatched types
485 Note that not all headers have meaning when customized to a revision.
486 For example, the `ignore-test` header (and all "ignore" headers)
487 currently only apply to the test as a whole, not to particular
488 revisions. The only headers that are intended to really work when
489 customized to a revision are error patterns and compiler flags.
494 Compiletest can be run in different modes, called *compare modes*, which can
495 be used to compare the behavior of all tests with different compiler flags
497 This can help highlight what differences might appear with certain flags, and
498 check for any problems that might arise.
500 To run the tests in a different mode, you need to pass the `--compare-mode`
504 ./x.py test src/test/ui --compare-mode=chalk
507 The possible compare modes are:
509 * `polonius` — Runs with Polonius with `-Zpolonius`.
510 * `chalk` — Runs with Chalk with `-Zchalk`.
511 * `split-dwarf` — Runs with unpacked split-DWARF with `-Csplit-debuginfo=unpacked`.
512 * `split-dwarf-single` — Runs with packed split-DWARF with `-Csplit-debuginfo=packed`.
514 See [UI compare modes](ui.md#compare-modes) for more information about how UI
515 tests support different output for different modes.
517 In CI, compare modes are only used in one Linux builder, and only with the
520 * `src/test/debuginfo`: Uses `split-dwarf` mode.
521 This helps ensure that none of the debuginfo tests are affected when
522 enabling split-DWARF.
524 Note that compare modes are separate to [revisions](#revisions).
525 All revisions are tested when running `./x.py test src/test/ui`, however
526 compare-modes must be manually run individually via the `--compare-mode` flag.