src/doc/rustc-guide/src/tests/intro.md

   1 # The compiler testing framework
   2
   3 The Rust project runs a wide variety of different tests, orchestrated
   4 by the build system (`x.py test`).  The main test harness for testing
   5 the compiler itself is a tool called compiletest (sources in the
   6 [`src/tools/compiletest`]). This section gives a brief overview of how
   7 the testing framework is setup, and then gets into some of the details
   8 on [how to run tests](./running.html) as well as
   9 [how to add new tests](./adding.html).
  10
  11 [`src/tools/compiletest`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest
  12
  13 ## Compiletest test suites
  14
  15 The compiletest tests are located in the tree in the [`src/test`]
  16 directory. Immediately within you will see a series of subdirectories
  17 (e.g. `ui`, `run-make`, and so forth). Each of those directories is
  18 called a **test suite** – they house a group of tests that are run in
  19 a distinct mode.
  20
  21 [`src/test`]: https://github.com/rust-lang/rust/tree/master/src/test
  22
  23 Here is a brief summary of the test suites as of this writing and what
  24 they mean. In some cases, the test suites are linked to parts of the manual
  25 that give more details.
  26
  27 - [`ui`](./adding.html#ui) – tests that check the exact
  28   stdout/stderr from compilation and/or running the test
  29 - `run-pass-valgrind` – tests that ought to run with valgrind
  30 - `run-fail` – tests that are expected to compile but then panic
  31   during execution
  32 - `compile-fail` – tests that are expected to fail compilation.
  33 - `parse-fail` – tests that are expected to fail to parse
  34 - `pretty` – tests targeting the Rust "pretty printer", which
  35   generates valid Rust code from the AST
  36 - `debuginfo` – tests that run in gdb or lldb and query the debug info
  37 - `codegen` – tests that compile and then test the generated LLVM
  38   code to make sure that the optimizations we want are taking effect.
  39 - `assembly` – similar to `codegen` tests, but verifies assembly output
  40   to make sure LLVM target backend can handle provided code.
  41 - `mir-opt` – tests that check parts of the generated MIR to make
  42   sure we are building things correctly or doing the optimizations we
  43   expect.
  44 - `incremental` – tests for incremental compilation, checking that
  45   when certain modifications are performed, we are able to reuse the
  46   results from previous compilations.
  47 - `run-make` – tests that basically just execute a `Makefile`; the
  48   ultimate in flexibility but quite annoying to write.
  49 - `rustdoc` – tests for rustdoc, making sure that the generated files
  50   contain the expected documentation.
  51 - `*-fulldeps` – same as above, but indicates that the test depends
  52   on things other than `libstd` (and hence those things must be built)
  53
  54 ## Other Tests
  55
  56 The Rust build system handles running tests for various other things,
  57 including:
  58
  59 - **Tidy** – This is a custom tool used for validating source code
  60   style and formatting conventions, such as rejecting long lines.
  61   There is more information in the
  62   [section on coding conventions](../conventions.html#formatting).
  63
  64   Example: `./x.py test src/tools/tidy`
  65
  66 - **Unit tests** – The Rust standard library and many of the Rust packages
  67   include typical Rust `#[test]` unittests.  Under the hood, `x.py` will run
  68   `cargo test` on each package to run all the tests.
  69
  70   Example: `./x.py test src/libstd`
  71
  72 - **Doc tests** – Example code embedded within Rust documentation is executed
  73   via `rustdoc --test`.  Examples:
  74
  75   `./x.py test src/doc` – Runs `rustdoc --test` for all documentation in
  76   `src/doc`.
  77
  78   `./x.py test --doc src/libstd` – Runs `rustdoc --test` on the standard
  79   library.
  80
  81 - **Link checker** – A small tool for verifying `href` links within
  82   documentation.
  83
  84   Example: `./x.py test src/tools/linkchecker`
  85
  86 - **Dist check** – This verifies that the source distribution tarball created
  87   by the build system will unpack, build, and run all tests.
  88
  89   Example: `./x.py test distcheck`
  90
  91 - **Tool tests** – Packages that are included with Rust have all of their
  92   tests run as well (typically by running `cargo test` within their
  93   directory).  This includes things such as cargo, clippy, rustfmt, rls, miri,
  94   bootstrap (testing the Rust build system itself), etc.
  95
  96 - **Cargo test** – This is a small tool which runs `cargo test` on a few
  97   significant projects (such as `servo`, `ripgrep`, `tokei`, etc.) just to
  98   ensure there aren't any significant regressions.
  99
 100   Example: `./x.py test src/tools/cargotest`
 101
 102 ## Testing infrastructure
 103
 104 When a Pull Request is opened on Github, [Azure Pipelines] will automatically
 105 launch a build that will run all tests on some configurations
 106 (x86_64-gnu-llvm-6.0 linux. x86_64-gnu-tools linux, mingw-check linux). In
 107 essence, it runs `./x.py test` after building for each of them.
 108
 109 The integration bot [bors] is used for coordinating merges to the master
 110 branch. When a PR is approved, it goes into a [queue] where merges are tested
 111 one at a time on a wide set of platforms using Azure Pipelines (currently over
 112 50 different configurations). Most platforms only run the build steps, some run
 113 a restricted set of tests, only a subset run the full suite of tests (see
 114 Rust's [platform tiers]).
 115
 116 [Azure Pipelines]: https://dev.azure.com/rust-lang/rust/
 117 [bors]: https://github.com/servo/homu
 118 [queue]: https://buildbot2.rust-lang.org/homu/queue/rust
 119 [platform tiers]: https://forge.rust-lang.org/platform-support.html
 120
 121 ## Testing with Docker images
 122
 123 The Rust tree includes [Docker] image definitions for the platforms used on
 124 Azure Pipelines in [src/ci/docker].  The script [src/ci/docker/run.sh] is used to build
 125 the Docker image, run it, build Rust within the image, and run the tests.
 126
 127 > TODO: What is a typical workflow for testing/debugging on a platform that
 128 > you don't have easy access to?  Do people build Docker images and enter them
 129 > to test things out?
 130
 131 [Docker]: https://www.docker.com/
 132 [src/ci/docker]: https://github.com/rust-lang/rust/tree/master/src/ci/docker
 133 [src/ci/docker/run.sh]: https://github.com/rust-lang/rust/blob/master/src/ci/docker/run.sh
 134
 135 ## Testing on emulators
 136
 137 Some platforms are tested via an emulator for architectures that aren't
 138 readily available.  There is a set of tools for orchestrating running the
 139 tests within the emulator.  Platforms such as `arm-android` and
 140 `arm-unknown-linux-gnueabihf` are set up to automatically run the tests under
 141 emulation on Travis.  The following will take a look at how a target's tests
 142 are run under emulation.
 143
 144 The Docker image for [armhf-gnu] includes [QEMU] to emulate the ARM CPU
 145 architecture.  Included in the Rust tree are the tools [remote-test-client]
 146 and [remote-test-server] which are programs for sending test programs and
 147 libraries to the emulator, and running the tests within the emulator, and
 148 reading the results.  The Docker image is set up to launch
 149 `remote-test-server` and the build tools use `remote-test-client` to
 150 communicate with the server to coordinate running tests (see
 151 [src/bootstrap/test.rs]).
 152
 153 > TODO: What are the steps for manually running tests within an emulator?
 154 > `./src/ci/docker/run.sh armhf-gnu` will do everything, but takes hours to
 155 > run and doesn't offer much help with interacting within the emulator.
 156 >
 157 > Is there any support for emulating other (non-Android) platforms, such as
 158 > running on an iOS emulator?
 159 >
 160 > Is there anything else interesting that can be said here about running tests
 161 > remotely on real hardware?
 162 >
 163 > It's also unclear to me how the wasm or asm.js tests are run.
 164
 165 [armhf-gnu]: https://github.com/rust-lang/rust/tree/master/src/ci/docker/armhf-gnu
 166 [QEMU]: https://www.qemu.org/
 167 [remote-test-client]: https://github.com/rust-lang/rust/tree/master/src/tools/remote-test-client
 168 [remote-test-server]: https://github.com/rust-lang/rust/tree/master/src/tools/remote-test-server
 169 [src/bootstrap/test.rs]: https://github.com/rust-lang/rust/tree/master/src/bootstrap/test.rs
 170
 171 ## Crater
 172
 173 [Crater](https://github.com/rust-lang-nursery/crater) is a tool for compiling
 174 and running tests for _every_ crate on [crates.io](https://crates.io) (and a
 175 few on GitHub). It is mainly used for checking for extent of breakage when
 176 implementing potentially breaking changes and ensuring lack of breakage by
 177 running beta vs stable compiler versions.
 178
 179 ### When to run Crater
 180
 181 You should request a crater run if your PR makes large changes to the compiler
 182 or could cause breakage. If you are unsure, feel free to ask your PR's reviewer.
 183
 184 ### Requesting Crater Runs
 185
 186 The rust team maintains a few machines that can be used for running crater runs
 187 on the changes introduced by a PR. If your PR needs a crater run, leave a
 188 comment for the triage team in the PR thread. Please inform the team whether
 189 you require a "check-only" crater run, a "build only" crater run, or a
 190 "build-and-test" crater run. The difference is primarily in time; the
 191 conservative (if you're not sure) option is to go for the build-and-test run.
 192 If making changes that will only have an effect at compile-time (e.g.,
 193 implementing a new trait) then you only need a check run.
 194
 195 Your PR will be enqueued by the triage team and the results will be posted when
 196 they are ready. Check runs will take around ~3-4 days, with the other two
 197 taking 5-6 days on average.
 198
 199 While crater is really useful, it is also important to be aware of a few
 200 caveats:
 201
 202 - Not all code is on crates.io! There is a lot of code in repos on GitHub and
 203   elsewhere. Also, companies may not wish to publish their code. Thus, a
 204   successful crater run is not a magically green light that there will be no
 205   breakage; you still need to be careful.
 206
 207 - Crater only runs Linux builds on x86_64. Thus, other architectures and
 208   platforms are not tested. Critically, this includes Windows.
 209
 210 - Many crates are not tested. This could be for a lot of reasons, including
 211   that the crate doesn't compile any more (e.g. used old nightly features),
 212   has broken or flaky tests, requires network access, or other reasons.
 213
 214 - Before crater can be run, `@bors try` needs to succeed in building artifacts.
 215   This means that if your code doesn't compile, you cannot run crater.
 216
 217 ## Perf runs
 218
 219 A lot of work is put into improving the performance of the compiler and
 220 preventing performance regressions. A "perf run" is used to compare the
 221 performance of the compiler in different configurations for a large collection
 222 of popular crates. Different configurations include "fresh builds", builds
 223 with incremental compilation, etc.
 224
 225 The result of a perf run is a comparison between two versions of the
 226 compiler (by their commit hashes).
 227
 228 You should request a perf run if your PR may affect performance, especially
 229 if it can affect performance adversely.
 230
 231 ## Further reading
 232
 233 The following blog posts may also be of interest:
 234
 235 - brson's classic ["How Rust is tested"][howtest]
 236
 237 [howtest]: https://brson.github.io/2017/07/10/how-rust-is-tested