src/bootstrap/README.md

   1 # rustbuild - Bootstrapping Rust
   2
   3 This README is aimed at helping to explain how Rust is bootstrapped,
   4 and some of the technical details of the build system.
   5
   6 Note that this README only covers internal information, not how to use the tool.
   7 Please check [bootstrapping dev guide][bootstrapping-dev-guide] for further information.
   8
   9 [bootstrapping-dev-guide]: https://rustc-dev-guide.rust-lang.org/building/bootstrapping/intro.html
  10
  11 ## Introduction
  12
  13 The build system defers most of the complicated logic of managing invocations
  14 of rustc and rustdoc to Cargo itself. However, moving through various stages
  15 and copying artifacts is still necessary for it to do. Each time rustbuild
  16 is invoked, it will iterate through the list of predefined steps and execute
  17 each serially in turn if it matches the paths passed or is a default rule.
  18 For each step, rustbuild relies on the step internally being incremental and
  19 parallel. Note, though, that the `-j` parameter to rustbuild gets forwarded
  20 to appropriate test harnesses and such.
  21
  22 ## Build phases
  23
  24 The rustbuild build system goes through a few phases to actually build the
  25 compiler. What actually happens when you invoke rustbuild is:
  26
  27 1. The entry point script (`x` for unix like systems, `x.ps1` for windows systems,
  28    `x.py` cross-platform) is run. This script is responsible for downloading the stage0
  29    compiler/Cargo binaries, and it then compiles the build system itself (this folder).
  30    Finally, it then invokes the actual `bootstrap` binary build system.
  31 2. In Rust, `bootstrap` will slurp up all configuration, perform a number of
  32    sanity checks (whether compilers exist, for example), and then start building the
  33    stage0 artifacts.
  34 3. The stage0 `cargo`, downloaded earlier, is used to build the standard library
  35    and the compiler, and then these binaries are then copied to the `stage1`
  36    directory. That compiler is then used to generate the stage1 artifacts which
  37    are then copied to the stage2 directory, and then finally, the stage2
  38    artifacts are generated using that compiler.
  39
  40 The goal of each stage is to (a) leverage Cargo as much as possible and failing
  41 that (b) leverage Rust as much as possible!
  42
  43 ## Directory Layout
  44
  45 This build system houses all output under the `build` directory, which looks
  46 like this:
  47
  48 ```sh
  49 # Root folder of all output. Everything is scoped underneath here
  50 build/
  51
  52   # Location where the stage0 compiler downloads are all cached. This directory
  53   # only contains the tarballs themselves, as they're extracted elsewhere.
  54   cache/
  55     2015-12-19/
  56     2016-01-15/
  57     2016-01-21/
  58     ...
  59
  60   # Output directory for building this build system itself. The stage0
  61   # cargo/rustc are used to build the build system into this location.
  62   bootstrap/
  63     debug/
  64     release/
  65
  66   # Output of the dist-related steps like dist-std, dist-rustc, and dist-docs
  67   dist/
  68
  69   # Temporary directory used for various input/output as part of various stages
  70   tmp/
  71
  72   # Each remaining directory is scoped by the "host" triple of compilation at
  73   # hand.
  74   x86_64-unknown-linux-gnu/
  75
  76     # The build artifacts for the `compiler-rt` library for the target that
  77     # this folder is under. The exact layout here will likely depend on the
  78     # platform, and this is also built with CMake, so the build system is
  79     # also likely different.
  80     compiler-rt/
  81       build/
  82
  83     # Output folder for LLVM if it is compiled for this target
  84     llvm/
  85
  86       # build folder (e.g. the platform-specific build system). Like with
  87       # compiler-rt, this is compiled with CMake
  88       build/
  89
  90       # Installation of LLVM. Note that we run the equivalent of 'make install'
  91       # for LLVM, to setup these folders.
  92       bin/
  93       lib/
  94       include/
  95       share/
  96       ...
  97
  98     # Output folder for all documentation of this target. This is what's filled
  99     # in whenever the `doc` step is run.
 100     doc/
 101
 102     # Output for all compiletest-based test suites
 103     test/
 104       ui/
 105       debuginfo/
 106       ...
 107
 108     # Location where the stage0 Cargo and Rust compiler are unpacked. This
 109     # directory is purely an extracted and overlaid tarball of these two (done
 110     # by the bootstrap Python script). In theory, the build system does not
 111     # modify anything under this directory afterwards.
 112     stage0/
 113
 114     # These to-build directories are the cargo output directories for builds of
 115     # the standard library, the test system, the compiler, and various tools,
 116     # respectively. Internally, these may also
 117     # have other target directories, which represent artifacts being compiled
 118     # from the host to the specified target.
 119     #
 120     # Essentially, each of these directories is filled in by one `cargo`
 121     # invocation. The build system instruments calling Cargo in the right order
 122     # with the right variables to ensure that these are filled in correctly.
 123     stageN-std/
 124     stageN-test/
 125     stageN-rustc/
 126     stageN-tools/
 127
 128     # This is a special case of the above directories, **not** filled in via
 129     # Cargo but rather the build system itself. The stage0 compiler already has
 130     # a set of target libraries for its own host triple (in its own sysroot)
 131     # inside of stage0/. When we run the stage0 compiler to bootstrap more
 132     # things, however, we don't want to use any of these libraries (as those are
 133     # the ones that we're building). So essentially, when the stage1 compiler is
 134     # being compiled (e.g. after libstd has been built), *this* is used as the
 135     # sysroot for the stage0 compiler being run.
 136     #
 137     # Basically, this directory is just a temporary artifact used to configure the
 138     # stage0 compiler to ensure that the libstd that we just built is used to
 139     # compile the stage1 compiler.
 140     stage0-sysroot/lib/
 141
 142     # These output directories are intended to be standalone working
 143     # implementations of the compiler (corresponding to each stage). The build
 144     # system will link (using hard links) output from stageN-{std,rustc} into
 145     # each of these directories.
 146     #
 147     # In theory these are working rustc sysroot directories, meaning there is
 148     # no extra build output in these directories.
 149     stage1/
 150     stage2/
 151     stage3/
 152 ```
 153
 154 ## Extending rustbuild
 155
 156 When you use the bootstrap system, you'll call it through the entry point script
 157 (`x`, `x.ps1`, or `x.py`). However, most of the code lives in `src/bootstrap`.
 158 `bootstrap` has a difficult problem: it is written in Rust, but yet it is run
 159 before the Rust compiler is built! To work around this, there are two components
 160 of bootstrap: the main one written in rust, and `bootstrap.py`. `bootstrap.py`
 161 is what gets run by entry point script. It takes care of downloading the `stage0`
 162 compiler, which will then build the bootstrap binary written in Rust.
 163
 164 Because there are two separate codebases behind `x.py`, they need to
 165 be kept in sync. In particular, both `bootstrap.py` and the bootstrap binary
 166 parse `config.toml` and read the same command line arguments. `bootstrap.py`
 167 keeps these in sync by setting various environment variables, and the
 168 programs sometimes have to add arguments that are explicitly ignored, to be
 169 read by the other.
 170
 171 Some general areas that you may be interested in modifying are:
 172
 173 * Adding a new build tool? Take a look at `bootstrap/src/core/build_steps/tool.rs`
 174   for examples of other tools.
 175 * Adding a new compiler crate? Look no further! Adding crates can be done by
 176   adding a new directory with `Cargo.toml`, followed by configuring all
 177   `Cargo.toml` files accordingly.
 178 * Adding a new dependency from crates.io? This should just work inside the
 179   compiler artifacts stage (everything other than libtest and libstd).
 180 * Adding a new configuration option? You'll want to modify `bootstrap/src/core/config/flags.rs`
 181   for command line flags and then `bootstrap/src/core/config/config.rs` to copy the flags to the
 182   `Config` struct.
 183 * Adding a sanity check? Take a look at `bootstrap/src/core/sanity.rs`.
 184
 185 If you make a major change on bootstrap configuration, please remember to:
 186
 187 + Update `CONFIG_CHANGE_HISTORY` in `src/bootstrap/src/utils/change_tracker.rs`.
 188 * Update `change-id = {pull-request-id}` in `config.example.toml`.
 189
 190 A 'major change' includes
 191
 192 * A new option or
 193 * A change in the default options.
 194
 195 Changes that do not affect contributors to the compiler or users
 196 building rustc from source don't need an update to `CONFIG_CHANGE_HISTORY`.
 197
 198 If you have any questions, feel free to reach out on the `#t-infra/bootstrap` channel
 199 at [Rust Bootstrap Zulip server][rust-bootstrap-zulip]. When you encounter bugs,
 200 please file issues on the [Rust issue tracker][rust-issue-tracker].
 201
 202 [rust-bootstrap-zulip]: https://rust-lang.zulipchat.com/#narrow/stream/t-infra.2Fbootstrap
 203 [rust-issue-tracker]: https://github.com/rust-lang/rust/issues
 204
 205 ## Changelog
 206
 207 Because we do not release bootstrap with versions, we also do not maintain CHANGELOG files. To
 208 review the changes made to bootstrap, simply run `git log --no-merges --oneline -- src/bootstrap`.