src/doc/rustc-dev-guide/src/tests/running.md

   1 # Running tests
   2
   3 <!-- toc -->
   4
   5 You can run the tests using `x.py`. The most basic command – which
   6 you will almost never want to use! – is as follows:
   7
   8 ```bash
   9 ./x.py test
  10 ```
  11
  12 This will build the stage 1 compiler and then run the whole test
  13 suite. You probably don't want to do this very often, because it takes
  14 a very long time, and anyway bors / GitHub Actions will do it for you.
  15 (Often, I will run this command in the background after opening a PR that
  16 I think is done, but rarely otherwise. -nmatsakis)
  17
  18 The test results are cached and previously successful tests are
  19 `ignored` during testing. The stdout/stderr contents as well as a
  20 timestamp file for every test can be found under `build/ARCH/test/`.
  21 To force-rerun a test (e.g. in case the test runner fails to notice
  22 a change) you can simply remove the timestamp file.
  23
  24 Note that some tests require a Python-enabled gdb. You can test if
  25 your gdb install supports Python by using the `python` command from
  26 within gdb. Once invoked you can type some Python code (e.g.
  27 `print("hi")`) followed by return and then `CTRL+D` to execute it.
  28 If you are building gdb from source, you will need to configure with
  29 `--with-python=<path-to-python-binary>`.
  30
  31 ## Running a subset of the test suites
  32
  33 When working on a specific PR, you will usually want to run a smaller
  34 set of tests. For example, a good "smoke test" that can be used after
  35 modifying rustc to see if things are generally working correctly would be the
  36 following:
  37
  38 ```bash
  39 ./x.py test src/test/ui
  40 ```
  41
  42 This will run the `ui` test suite. Of course, the choice
  43 of test suites is somewhat arbitrary, and may not suit the task you are
  44 doing. For example, if you are hacking on debuginfo, you may be better off
  45 with the debuginfo test suite:
  46
  47 ```bash
  48 ./x.py test src/test/debuginfo
  49 ```
  50
  51 If you only need to test a specific subdirectory of tests for any
  52 given test suite, you can pass that directory to `x.py test`:
  53
  54 ```bash
  55 ./x.py test src/test/ui/const-generics
  56 ```
  57
  58 Likewise, you can test a single file by passing its path:
  59
  60 ```bash
  61 ./x.py test src/test/ui/const-generics/const-test.rs
  62 ```
  63
  64 ### Run only the tidy script
  65
  66 ```bash
  67 ./x.py test tidy
  68 ```
  69
  70 ### Run tests on the standard library
  71
  72 ```bash
  73 ./x.py test --stage 0 library/std
  74 ```
  75
  76 ### Run the tidy script and tests on the standard library
  77
  78 ```bash
  79 ./x.py test --stage 0 tidy library/std
  80 ```
  81
  82 ### Run tests on the standard library using a stage 1 compiler
  83
  84 ```bash
  85 ./x.py test library/std
  86 ```
  87
  88 By listing which test suites you want to run you avoid having to run
  89 tests for components you did not change at all.
  90
  91 **Warning:** Note that bors only runs the tests with the full stage 2
  92 build; therefore, while the tests **usually** work fine with stage 1,
  93 there are some limitations.
  94
  95 ## Run unit tests on the compiler/library
  96
  97 You may want to run unit tests on a specific file with following:
  98
  99 ```bash
 100 ./x.py test compiler/rustc_data_structures/src/thin_vec/tests.rs
 101 ```
 102
 103 But unfortunately, it's impossible. You should invoke following instead:
 104
 105 ```bash
 106 ./x.py test compiler/rustc_data_structures/ --test-args thin_vec
 107 ```
 108
 109 ## Running an individual test
 110
 111 Another common thing that people want to do is to run an **individual
 112 test**, often the test they are trying to fix. As mentioned earlier,
 113 you may pass the full file path to achieve this, or alternatively one
 114 may invoke `x.py` with the `--test-args` option:
 115
 116 ```bash
 117 ./x.py test src/test/ui --test-args issue-1234
 118 ```
 119
 120 Under the hood, the test runner invokes the standard rust test runner
 121 (the same one you get with `#[test]`), so this command would wind up
 122 filtering for tests that include "issue-1234" in the name. (Thus
 123 `--test-args` is a good way to run a collection of related tests.)
 124
 125 ## Editing and updating the reference files
 126
 127 If you have changed the compiler's output intentionally, or you are
 128 making a new test, you can pass `--bless` to the test subcommand. E.g.
 129 if some tests in `src/test/ui` are failing, you can run
 130
 131 ```text
 132 ./x.py test src/test/ui --bless
 133 ```
 134
 135 to automatically adjust the `.stderr`, `.stdout` or `.fixed` files of
 136 all tests. Of course you can also target just specific tests with the
 137 `--test-args your_test_name` flag, just like when running the tests.
 138
 139 ## Passing `--pass $mode`
 140
 141 Pass UI tests now have three modes, `check-pass`, `build-pass` and
 142 `run-pass`. When `--pass $mode` is passed, these tests will be forced
 143 to run under the given `$mode` unless the directive `// ignore-pass`
 144 exists in the test file. For example, you can run all the tests in
 145 `src/test/ui` as `check-pass`:
 146
 147 ```bash
 148 ./x.py test src/test/ui --pass check
 149 ```
 150
 151 By passing `--pass $mode`, you can reduce the testing time. For each
 152 mode, please see [here][mode].
 153
 154 [mode]: ./adding.md#tests-that-do-not-result-in-compile-errors
 155
 156 ## Using incremental compilation
 157
 158 You can further enable the `--incremental` flag to save additional
 159 time in subsequent rebuilds:
 160
 161 ```bash
 162 ./x.py test src/test/ui --incremental --test-args issue-1234
 163 ```
 164
 165 If you don't want to include the flag with every command, you can
 166 enable it in the `config.toml`:
 167
 168 ```toml
 169 [rust]
 170 incremental = true
 171 ```
 172
 173 Note that incremental compilation will use more disk space than usual.
 174 If disk space is a concern for you, you might want to check the size
 175 of the `build` directory from time to time.
 176
 177 ## Running tests with different "compare modes"
 178
 179 UI tests may have different output depending on certain "modes" that
 180 the compiler is in. For example, when in "non-lexical lifetimes" (NLL)
 181 mode a test `foo.rs` will first look for expected output in
 182 `foo.nll.stderr`, falling back to the usual `foo.stderr` if not found.
 183 To run the UI test suite in NLL mode, one would use the following:
 184
 185 ```bash
 186 ./x.py test src/test/ui --compare-mode=nll
 187 ```
 188
 189 Other examples of compare-modes are "noopt", "migrate", and
 190 [revisions](./adding.html#revisions).
 191
 192 ## Running tests manually
 193
 194 Sometimes it's easier and faster to just run the test by hand. Most tests are
 195 just `rs` files, so you can do something like
 196
 197 ```bash
 198 rustc +stage1 src/test/ui/issue-1234.rs
 199 ```
 200
 201 This is much faster, but doesn't always work. For example, some tests
 202 include directives that specify specific compiler flags, or which rely
 203 on other crates, and they may not run the same without those options.