src/bootstrap/README.md

   1 # rustbuild - Bootstrapping Rust
   2
   3 This is an in-progress README which is targeted at helping to explain how Rust
   4 is bootstrapped and in general some of the technical details of the build
   5 system.
   6
   7 > **Note**: This build system is currently under active development and is not
   8 > intended to be the primarily used one just yet. The makefiles are currently
   9 > the ones that are still "guaranteed to work" as much as possible at least.
  10
  11 ## Using rustbuild
  12
  13 The rustbuild build system has a primary entry point, a top level `x.py` script:
  14
  15 ```
  16 python ./x.py build
  17 ```
  18
  19 Note that if you're on Unix you should be able to execute the script directly:
  20
  21 ```
  22 ./x.py build
  23 ```
  24
  25 The script accepts commands, flags, and arguments to determine what to do:
  26
  27 * `build` - a general purpose command for compiling code. Alone `build` will
  28   bootstrap the entire compiler, and otherwise arguments passed indicate what to
  29   build. For example:
  30
  31   ```
  32   # build the whole compiler
  33   ./x.py build
  34
  35   # build the stage1 compiler
  36   ./x.py build --stage 1
  37
  38   # build stage0 libstd
  39   ./x.py build --stage 0 src/libstd
  40
  41   # build a particular crate in stage0
  42   ./x.py build --stage 0 src/libtest
  43   ```
  44
  45   If files are dirty that would normally be rebuilt from stage 0, that can be
  46   overidden using `--keep-stage 0`. Using `--keep-stage n` will skip all steps
  47   that belong to stage n or earlier:
  48
  49   ```
  50   # keep old build products for stage 0 and build stage 1
  51   ./x.py build --keep-stage 0 --stage 1
  52   ```
  53
  54 * `test` - a command for executing unit tests. Like the `build` command this
  55   will execute the entire test suite by default, and otherwise it can be used to
  56   select which test suite is run:
  57
  58   ```
  59   # run all unit tests
  60   ./x.py test
  61
  62   # execute the run-pass test suite
  63   ./x.py test src/test/run-pass
  64
  65   # execute only some tests in the run-pass test suite
  66   ./x.py test src/test/run-pass --test-args substring-of-test-name
  67
  68   # execute tests in the standard library in stage0
  69   ./x.py test --stage 0 src/libstd
  70
  71   # execute all doc tests
  72   ./x.py test src/doc
  73   ```
  74
  75 * `doc` - a command for building documentation. Like above can take arguments
  76   for what to document.
  77
  78 ## Configuring rustbuild
  79
  80 There are currently two primary methods for configuring the rustbuild build
  81 system. First, the `./configure` options serialized in `config.mk` will be
  82 parsed and read. That is, if any `./configure` options are passed, they'll be
  83 handled naturally.
  84
  85 Next, rustbuild offers a TOML-based configuration system with a `config.toml`
  86 file in the same location as `config.mk`. An example of this configuration can
  87 be found at `src/bootstrap/config.toml.example`, and the configuration file
  88 can also be passed as `--config path/to/config.toml` if the build system is
  89 being invoked manually (via the python script).
  90
  91 Finally, rustbuild makes use of the [gcc-rs crate] which has [its own
  92 method][env-vars] of configuring C compilers and C flags via environment
  93 variables.
  94
  95 [gcc-rs crate]: https://github.com/alexcrichton/gcc-rs
  96 [env-vars]: https://github.com/alexcrichton/gcc-rs#external-configuration-via-environment-variables
  97
  98 ## Build stages
  99
 100 The rustbuild build system goes through a few phases to actually build the
 101 compiler. What actually happens when you invoke rustbuild is:
 102
 103 1. The entry point script, `x.py` is run. This script is
 104    responsible for downloading the stage0 compiler/Cargo binaries, and it then
 105    compiles the build system itself (this folder). Finally, it then invokes the
 106    actual `bootstrap` binary build system.
 107 2. In Rust, `bootstrap` will slurp up all configuration, perform a number of
 108    sanity checks (compilers exist for example), and then start building the
 109    stage0 artifacts.
 110 3. The stage0 `cargo` downloaded earlier is used to build the standard library
 111    and the compiler, and then these binaries are then copied to the `stage1`
 112    directory. That compiler is then used to generate the stage1 artifacts which
 113    are then copied to the stage2 directory, and then finally the stage2
 114    artifacts are generated using that compiler.
 115
 116 The goal of each stage is to (a) leverage Cargo as much as possible and failing
 117 that (b) leverage Rust as much as possible!
 118
 119 ## Directory Layout
 120
 121 This build system houses all output under the `build` directory, which looks
 122 like this:
 123
 124 ```
 125 # Root folder of all output. Everything is scoped underneath here
 126 build/
 127
 128   # Location where the stage0 compiler downloads are all cached. This directory
 129   # only contains the tarballs themselves as they're extracted elsewhere.
 130   cache/
 131     2015-12-19/
 132     2016-01-15/
 133     2016-01-21/
 134     ...
 135
 136   # Output directory for building this build system itself. The stage0
 137   # cargo/rustc are used to build the build system into this location.
 138   bootstrap/
 139     debug/
 140     release/
 141
 142   # Output of the dist-related steps like dist-std, dist-rustc, and dist-docs
 143   dist/
 144
 145   # Temporary directory used for various input/output as part of various stages
 146   tmp/
 147
 148   # Each remaining directory is scoped by the "host" triple of compilation at
 149   # hand.
 150   x86_64-unknown-linux-gnu/
 151
 152     # The build artifacts for the `compiler-rt` library for the target this
 153     # folder is under. The exact layout here will likely depend on the platform,
 154     # and this is also built with CMake so the build system is also likely
 155     # different.
 156     compiler-rt/
 157       build/
 158
 159     # Output folder for LLVM if it is compiled for this target
 160     llvm/
 161
 162       # build folder (e.g. the platform-specific build system). Like with
 163       # compiler-rt this is compiled with CMake
 164       build/
 165
 166       # Installation of LLVM. Note that we run the equivalent of 'make install'
 167       # for LLVM to setup these folders.
 168       bin/
 169       lib/
 170       include/
 171       share/
 172       ...
 173
 174     # Output folder for all documentation of this target. This is what's filled
 175     # in whenever the `doc` step is run.
 176     doc/
 177
 178     # Output for all compiletest-based test suites
 179     test/
 180       run-pass/
 181       compile-fail/
 182       debuginfo/
 183       ...
 184
 185     # Location where the stage0 Cargo and Rust compiler are unpacked. This
 186     # directory is purely an extracted and overlaid tarball of these two (done
 187     # by the bootstrapy python script). In theory the build system does not
 188     # modify anything under this directory afterwards.
 189     stage0/
 190
 191     # These to build directories are the cargo output directories for builds of
 192     # the standard library and compiler, respectively. Internally these may also
 193     # have other target directories, which represent artifacts being compiled
 194     # from the host to the specified target.
 195     #
 196     # Essentially, each of these directories is filled in by one `cargo`
 197     # invocation. The build system instruments calling Cargo in the right order
 198     # with the right variables to ensure these are filled in correctly.
 199     stageN-std/
 200     stageN-test/
 201     stageN-rustc/
 202     stageN-tools/
 203
 204     # This is a special case of the above directories, **not** filled in via
 205     # Cargo but rather the build system itself. The stage0 compiler already has
 206     # a set of target libraries for its own host triple (in its own sysroot)
 207     # inside of stage0/. When we run the stage0 compiler to bootstrap more
 208     # things, however, we don't want to use any of these libraries (as those are
 209     # the ones that we're building). So essentially, when the stage1 compiler is
 210     # being compiled (e.g. after libstd has been built), *this* is used as the
 211     # sysroot for the stage0 compiler being run.
 212     #
 213     # Basically this directory is just a temporary artifact use to configure the
 214     # stage0 compiler to ensure that the libstd we just built is used to
 215     # compile the stage1 compiler.
 216     stage0-sysroot/lib/
 217
 218     # These output directories are intended to be standalone working
 219     # implementations of the compiler (corresponding to each stage). The build
 220     # system will link (using hard links) output from stageN-{std,rustc} into
 221     # each of these directories.
 222     #
 223     # In theory there is no extra build output in these directories.
 224     stage1/
 225     stage2/
 226     stage3/
 227 ```
 228
 229 ## Cargo projects
 230
 231 The current build is unfortunately not quite as simple as `cargo build` in a
 232 directory, but rather the compiler is split into three different Cargo projects:
 233
 234 * `src/rustc/std_shim` - a project which builds and compiles libstd
 235 * `src/rustc/test_shim` - a project which builds and compiles libtest
 236 * `src/rustc` - the actual compiler itself
 237
 238 Each "project" has a corresponding Cargo.lock file with all dependencies, and
 239 this means that building the compiler involves running Cargo three times. The
 240 structure here serves two goals:
 241
 242 1. Facilitating dependencies coming from crates.io. These dependencies don't
 243    depend on `std`, so libstd is a separate project compiled ahead of time
 244    before the actual compiler builds.
 245 2. Splitting "host artifacts" from "target artifacts". That is, when building
 246    code for an arbitrary target you don't need the entire compiler, but you'll
 247    end up needing libraries like libtest that depend on std but also want to use
 248    crates.io dependencies. Hence, libtest is split out as its own project that
 249    is sequenced after `std` but before `rustc`. This project is built for all
 250    targets.
 251
 252 There is some loss in build parallelism here because libtest can be compiled in
 253 parallel with a number of rustc artifacts, but in theory the loss isn't too bad!
 254
 255 ## Build tools
 256
 257 We've actually got quite a few tools that we use in the compiler's build system
 258 and for testing. To organize these, each tool is a project in `src/tools` with a
 259 corresponding `Cargo.toml`. All tools are compiled with Cargo (currently having
 260 independent `Cargo.lock` files) and do not currently explicitly depend on the
 261 compiler or standard library. Compiling each tool is sequenced after the
 262 appropriate libstd/libtest/librustc compile above.
 263
 264 ## Extending rustbuild
 265
 266 So you'd like to add a feature to the rustbuild build system or just fix a bug.
 267 Great! One of the major motivational factors for moving away from `make` is that
 268 Rust is in theory much easier to read, modify, and write. If you find anything
 269 excessively confusing, please open an issue on this and we'll try to get it
 270 documented or simplified pronto.
 271
 272 First up, you'll probably want to read over the documentation above as that'll
 273 give you a high level overview of what rustbuild is doing. You also probably
 274 want to play around a bit yourself by just getting it up and running before you
 275 dive too much into the actual build system itself.
 276
 277 After that, each module in rustbuild should have enough documentation to keep
 278 you up and running. Some general areas that you may be interested in modifying
 279 are:
 280
 281 * Adding a new build tool? Take a look at `bootstrap/step.rs` for examples of
 282   other tools.
 283 * Adding a new compiler crate? Look no further! Adding crates can be done by
 284   adding a new directory with `Cargo.toml` followed by configuring all
 285   `Cargo.toml` files accordingly.
 286 * Adding a new dependency from crates.io? We're still working on that, so hold
 287   off on that for now.
 288 * Adding a new configuration option? Take a look at `bootstrap/config.rs` or
 289   perhaps `bootstrap/flags.rs` and then modify the build elsewhere to read that
 290   option.
 291 * Adding a sanity check? Take a look at `bootstrap/sanity.rs`.
 292
 293 If you have any questions feel free to reach out on `#rust-internals` on IRC or
 294 open an issue in the bug tracker!